MEPs endorsed the decision by a show of hands after reviewing a 2014 clip the former National Front leader posted on the party website.
In it, Le Pen vowed to put his critics in their place -- including French singer Patrick Bruel, who is Jewish -- and used a pun suggesting Nazi gas chambers.
The European Parliament's legal affairs committee said earlier the 88-year-old's remarks were not in line with his work as an MEP.
The committee also stressed parliamentary immunity "does not allow for slandering, libelling, inciting hatred or pronouncing statements attacking a person's honour."
The 2014 video was also condemned by Le Pen's daughter Marine, who took over the FN's leadership in 2011, as a "political error."
It was the first time she had criticised her father in public. The senior Le Pen has had multiple convictions for inciting racial hatred and denying crimes against humanity.
On Monday, the president of the European Parliament, Martin Schulz, said he had received a request to also end Marine Le Pen's immunity, and this had been handed to the legal affairs committee.
A source at the parliament, in the eastern French city of Strasbourg, said Schulz's request is linked to an investigation opened after Marine Le Pen tweeted graphic images in December of Islamic State atrocities, including the beheading of US reporter James Foley.
She said she was responding to a journalist whom she had accused of having compared her party to the Islamic State group.
Marine Le Pen, who is aiming to run in the 2017 French presidential election, has sought to make the FN an alternative to the conservative Republicans party for mainstream voters.
She has been locked in a vicious feud with her father, who was booted out of the party after rehashing familiar comments about Nazi gas chambers being a "detail" of history and saying France should improve ties with Russia to save the "white world."