The 24 migrants were ordered off a bus at the northern port of Calais as it was about to board a car ferry to cross the Channel, a local official told the AFP news agency.
The five are expected to be deported, while the others were let go because their papers were in order or their asylum applications were in process or completed, authorities said.
Activists of the Collectif des Sans-Papiers de Paris (CSP 75) decried the move, heading along with dozens of others to the Calais police station beating drums and shouting "Free Our Comrades".
France's Human Rights League (LDH) also demanded their release, saying "these people... are now deprived of their freedom for having participated, in a visible way, in a protest for a more open and welcoming world."
François Guennoc of the French charity Auberge des migrants that co-organised the march acknowledged that the group should not have been allowed to cross into Britain, but "what shocks us is that they were taken to a detention centre even though we were accompanying them and their presence was known."
Speaking from across the Channel in Dover, he added that the protesters had told French and British authorities that they had intended to take their protest into Britain only "for the day".
Guennoc and the other activists were expected to arrive in London later Sunday.
Several hundred people took part in the "solidarity" action for migrants, a series of relay marches that kicked off in the immigration hotspot of Ventimiglia, on the border between Italy and France, in late April.
The organisers arrived Saturday in Calais after the 1,400-kilometre journey.