France, China and Russia were among the eight countries that voted in favour of the text, but the resolution fell short of winning the nine "yes" votes necessary for adoption within the 15-member council.
France's envoy said he backed the resolution because of an "urgent need to act" and vowed to continue pressing for a UN role in reviving peace talks.
"Our efforts must not stop here. It is our responsibility to try again, before it's too late," said Francois Delattre.
Argentina, Chad, Chile, Jordan and Luxembourg joined China, France and Russia in supporting the resolution.
Australia and the United States voted against,
Britain abstained, along with Lithuania, Nigeria, Rwanda and South Korea.
The failure to win nine votes also spared Washington from having to wield its veto in a move which would have caused it problems with its Arab allies.
The Palestinian-drafted resolution, which was backed by Arab states, would have paved the way for an independent Palestinian state with east Jerusalem as its capital.
It set a 12-month deadline for Israel to reach a final peace deal with the Palestinians and called for a full withdrawal of Israeli troops from the Palestinian territories by the end of 2017.
The council's rejection of the resolution was hailed by Israel, which said the Palestinians would achieve their dream of statehood only through bilateral negotiations, rather than through "unilateral" moves at the United Nations.
"This deals a blow to efforts by [Palestinian president] Mahmud Abbas to embarrass and isolate us," said Lieberman's deputy Tzahi HaNegbi, who is known to be a close associate of Prime Minister Benjamin Netanyahu.
But Russia, one of the three permanent members of the Security Council which voted in favour, said the failure to pass the resolution was "a strategic error."
"Russia regrets that the UN Security Council did not manage to adopt the draft resolution," said Russia's UN envoy Vitaly Churkin after the vote, accusing Washington of "monopolising" decades of Israeli-Palestinian peace talks and leading them "to a dead end."
Security Council member Jordan had requested the vote on Tuesday despite opposition from Washington, which said the resolution did not address Israel's security concerns and set arbitrary deadlines.
"This resolution sets the stage for more division, not for compromise," US Ambassador Samantha Power told the council, saying it only addressed the concerns "of just one side."
"Peace will come from hard choices and compromises that must be made at the negotiating table," she said.
Ahead of the vote, Secretary of State John Kerry had spoken to 13 foreign ministers to explain the US position.
Diplomatic sources said Nigeria had been expected to support the resolution and changed its stance at the last minute.
The vote capped a three-month campaign by the Palestinians at the United Nations to win support for the resolution that would have set a timeframe for ending the Israeli occupation.
After the vote, Palestinian envoy Riyad Mansour accused the council of failing to shoulder its responsibilities and vowed to seek other venues to gain recognition.
"The Palestinian people and the world can no longer wait. That message, despite the regrettable outcome today, is especially clear," he said.
Abbas was to meet with top Palestinian officials at his West Bank headquarters at 1630 GMT on Wednesday to discuss the next steps in their campaign for statehood.