Assembling on Paris's Champs Elysées avenue, the scene of the murder of police officer Xavier Jugelé earlier this year, the Muslim preachers said they hoped to start a Europe-wide movement.
"Our message is clear," said the chief organiser, Hassen Chalgouhi. "You can't associate Islam with these barbarians and these murderers."
He called on "civil society to mobilise" after the deaths of several hundred people in attacks in Europe and after "7,000 young people have left for Iraq and Syria".
Chalgoumi, the former imam of the Paris suburb of Drancy, has attracted criticism and even threats for his opposition to fundamentalism and building bridges to the Jewish community.
French Muslim umbrella group, the CFCM, has refused to back the march but Chalgoumi said he did not want to enter into an argument over the matter, while questioning the motives of anyone who would oppose a "march against the barbarians".
Lisbon Imam David Munir hailed a "historic initiative in Europe", declaring that "Some people commit crimes in the name of Islam, we are here to say 'Not in our name'."
The participants will travel by coach to the sites of several attacks, including Berlin, Brussels, Toulouse and Nice before ending their demonstrations on the Champs Elysées on France's national day, 14 July.