"If we open the border tomorrow, what will happen? The British, who run their own border, will block them and send them back," Bernard Cazeneuve said on BFMTV.
He said this would only increase the flow of people and "aggravate a humanitarian problem".
"We don't need statements that create buzz on this topic, we need long-term action."
The row broke out Thursday when France's outspoken Economy Minister Emmanuel Macron raised the possibility of camps similar to the so-called Jungle shantytown at Calais springing up on Britain's southern coastline.
He told the Financial Times that a so-called Brexit would scupper a bilateral agreement that allows Britain to conduct border controls on the French side of the border.
"The day this relationship unravels, migrants will no longer be in Calais," Macron told the newspaper.
Asked about his comments during a summit meeting with British Prime Minister David Cameron, President Francois Hollande merely warned that "there will be consequences (to) the way in which we manage migration issues" if Britain voted in a referendum to leave the EU.
Cameron warned last month that a Brexit could mean British border checks being removed from Calais and that "there would be nothing to stop thousands of people crossing the Channel overnight."
But campaigners in favour of Britain leaving the 28-member bloc accused Cameron of scaremongering.
Observers are sceptical that France would make a move that would only draw thousands more migrants to its northern coast.