It was not immediately clear what sparked the violence at the site known to Muslims as Al-Aqsa mosque and to Jews as Temple Mount.
Tens of thousands of Muslims turned out early Sunday for prayers marking the beginning of the Islamic holiday of Eid al-Adha, which coincides with the Jewish holiday of Tisha Be' Av, or Ninth of Av, which mourns the destruction of the two Biblical temples built on the site during antiquity.
Reports say large numbers of Palestinians gathered at the gates of the compound early Sunday following rumours that Israeli police would allow Jewish visitors to enter.
Other reports say police prevented Jewish visitors from entering the compound before clashes erupted.
The Palestinian Red Crescent said 14 worshipers were injured as authorities dispersed crowds by firing sound grenades and tear gas.
Often a flashpoint in the Israeli-Palestinian conflict, the compound is the holiest site for Jews and the third holiest for Muslims.
Jews are barred from praying at the compound under a longstanding arrangement between Israel and Muslim authorities.
But in recent years Israeli religious nationalists have stepped up visits to the site to challenge the status quo.
In a separate incident on Sunday, Israeli troops killed a Palestinian gunman after he opened fire on them from across the perimeter fence around the Gaza Strip.
The small Palestinian territory sandwiched between Israel and Egypt has been under blockade since 2007, which has gutted its economy.
On Saturday, Israeli troops killed four Palestinian militants who the army said had tried to carry out a cross-border attack, out of frustration at the ongoing blockade.