TV footage showed heavily-armed commandos boarding the boat, which was in international waters, after being lowered from helicopters.
The Israeli military confirmed that "more than 10 people were killed" and that four soldiers were injured after shots were fired at them.
An Israeli TV channel reported that commandos had opened fire after being attacked with axes and knives by some of the 700 passengers on board the ships.
Thousands of people have taken to the streets in Turkey to protest Israel’s actions, while the Turkish government has summoned the Israeli ambassador to the foreign ministry.
The European Union's foreign affairs chief Catherine Ashton has called on Israeli authorities conduct a "full inquiry" into the incident.
"By targeting civilians, Israel has once again shown its disregard for human life and peaceful initiatives. We strongly condemn these inhumane practices of Israel," a Turkish government statement said.
"This deplorable incident, which took place in open seas and constitutes a fragrant breach of international law, may lead to irreparable consequences in our bilateral relations.
"Whatever the reason is, such action against civilians involved only in peaceful activities is unacceptable.
"Israel will have to bear the consequences of this behaviour, which constitutes a violation of international law.”
Palestinian president Mahmud Abbas called the killings "a massacre" and announced a three-day mourning period.
The Islamist movement Hamas, which took charge of Gaza in 2007, called on Arabs and Muslims to “rise up” to protest at Israeli embassies across the world.
The boats left from Cyprus on Sunday after Israel had warned it would stop the operation.
"This is a provocation intended to delegitimise Israel," Deputy Foreign Minister Danny Ayalon said on Saturday.
"If the flotilla had a genuine humanitarian goal, then its organisers should have transferred something for the abducted soldier Gilad Shalit as well," he said of the Israeli kidnapped by militants in 2006.
The activists said they had offered to take in a letter for Shalit from his family but received no response from their lawyer.