The French businessman took aim at the wave of such big fashion chains as Swedish giant H&M and Japanese brand Uniqlo that have followed the Italian designers Dolce & Gabbana in catering specifically to the Muslim market.
Berge told Europe 1 French radio on Wednesday he was "scandalised".
"Creators should have nothing to do with Islamic fashion. Designers are there to make women more beautiful, to give them their freedom, not to collaborate with this dictatorship which imposes this abominable thing by which we hide women and make them live a hidden life.
"Renounce the money and have some principles," he said.
The 85-year old's comments came as the French families minister, Laurence Rossignol, sparked outrage on social media as she compared women who followed this trend with "negroes who supported slavery".
Her office later told French news agency AFP that she had not intented to cause offence but was referring to an abolitionist tract by the French philosopher Montesquieu, "De l'esclavage des nègres" ("On the Enslavement of Negroes").
Earlier this year Dolce & Gabbana became the first major western brand to openly aim at capturing a corner of the Islamic fashion market - estimated to be worth 230 billion euros -- with its Abaya range. It included 14 abayas or ankle-length dresses, which it matched with embroidered headscarves and hijabs.
- with AFP