Rights organisations say the group includes some ethnic minorities which have suffered persecution in Myanmar.
They say sending them back to Myanmar, which is in the midst of a coup, could put them at even greater risk.
But Malaysia has said those being sent back committed immigration offences, and are not asylum seekers.
"All of those returned had agreed to be sent back voluntarily without being forced by any party," Khairul Dzaimee Daud, Malaysia's director-general of immigration, said in a statement.
Mr Daud added that those being repatriated did not include Rohingya refugees or asylum-seekers, echoing an earlier statement by Malaysia that it would not deport those registered with the UN High Commissioner for Refugees (UNHCR).