The three bodies were retrieved off the coast of Crete and 16 others were still missing Sunday morning. They were among 49 Bangladeshis who climbed down a rope to leave the Ionian King, which had fetched 1,280 people from Libya.
About 30 men have been transferred to a nearby hospital. Many of them are suffering from hypothermia, Merchant Marine officials say.
The reason why they left the ship are not known, although there is speculation that they did not have legal documents and feared being deported to Bangladesh.
There were about 60,000 Bangladeshi workers in Libya when the uprising against the leadership of Moamer Kadhafi erupted.
About 3,500 had returned to Bangladesh by Saturday with another 26,000 awaiting repatriation in Greece, Tunisia, Egypt and Italy, according to officials in Dhaka.
Many are reported not to want to return for fear that they will not be able to go back to Libya to work.
But relatives of migrant workers have accused the Bangladeshi government of abandoning them after it advised them to stay put unless they were the targets of attacks.
The government has admitted that it does not have the money to bring thousands of its nationals home.
Many of those who have returned did so at their own expense or were repatriated by their employers and the International Office of Migration, officials say.
Two other Bangladeshis have died while fleeing Libya – one in Tunisia and the other in Egypt.