The US had been negotiating to keep some military bases and a 5,000-strong force in the country to train the army and counter the influence of neighbouring Iran in the oil-rich and strategically crucial region. But defence officials said that talks with Baghdad collapsed over the question of legal protections for American troops.
Members of the government, some of whom are close to Iran, opposed troops staying and scuppered US plans to keep some bases.
"Today I can report that, as promised, the rest of our troops in Iraq will come home by the end of the year. After nearly nine years, America's war in Iraq will be over," Obama said at the White House.
When he was a presidential candidate Obama promised to withdraw all US military personnel.
At present there are about 39,000 US troops in Iraq, down from a peak of 165,000 in 505 in 2008.
Obama announced the pullout after holding a video conference with Prime Minister Nuri al-Maliki, which US officials said included a moving tribute by the Iraqi leader to American troops who died in his country.
But Obama said that despite the failure, US defence officials would still seek ways to help train Iraqi forces, as they do for many other nations.
US Defense Secretary Leon Panetta confirmed the plan.
"Once we've completed the reduction of the combat presence, then I think we begin a process of negotiating with them," Panetta told reporters.
The US will still have personnel at its embassy, one of the largest in the world, and two consults and 4-5,000 private defence contractors will remain, according to White House aids.
Republican presidential candidate Mitt Romney accused Obama of presiding over an “astonishing failure to secure an orderly transition in Iraq", putting at risk victories won through the sacrifice of American soldiers.
"The unavoidable question is whether this decision is the result of a naked political calculation or simply sheer ineptitude in negotiations with the Iraqi government," he said.