The attack was claimed by the Baloch Liberation Army (BLA), an ethnic insurgent group who opposes China's exploitation of natural resources in southwest Pakistan.
It is the most prominent attack against neighbor and ally China, which is pouring billions of dollars into the country as part of its vast Belt and Road initiative.
China "strongly condemned" the attack and asked Pakistan to take measures to ensure the safety of Chinese citizens and institutions in the country, as well as the China-Pakistan Economic Corridor (CPEC) mega-project.
In another incident on Friday, at least 25 people were killed in a bomb attack in north-west Pakistan.
The blast outside the consulate occurred in a mostly Shia neighbourhood in Orakzai district.
Police say three gunmen tried to enter the building in the southern port city, but were intercepted by guards at a checkpoint, Karachi police chief Ameer Sheikh told AFP news agency.
"They were holding Kalashnikovs. First, they hurled a small (grenade) and then started firing," said Allah Bakhsh, a guard at a nearby house who witnessed the attack.
Police officials said two of their personnel were killed, along with a father and son from Quetta, the capital of Balochistan, who were seeking Chinese visas and were caught in the crossfire.
At least one of the attackers was wearing a suicide vest which did not detonate, another senior police official said.
Foreign Minister Shah Mehmood Qureshi told reporters in Islamabad that "all the terrorists have been eliminated", and that all 21 staff at the consulate during the attack had been taken to a safe location.
"Situation under control," the military's media wing added in a statement.
China is an 'oppressor'
The attack was claimed by a separatist militant group from Pakistan's southwestern province of Balochistan, which is at the centre of the China-Pakistan Economic Corridor, the major Chinese project in the country.
"We have been seeing the Chinese as an oppressor, along with Pakistani forces," the spokesman for the Balochistan Liberation Army (BLA), Geand Baloch, told AFP by telephone from an undisclosed location, adding they were "destroying the future of Balochistan".
The BLA later emailed a statement to media in which it said the attack was "aimed at making it clear that China's military expansionism on Baloch soil will not be tolerated".
It warned the Chinese to leave or "be prepared for continued attacks".
The group is just one of the militant outfits operating in Balochistan, Pakistan's largest and poorest province, which is rife with ethnic, sectarian and separatist insurgencies.
Residents of the resource-rich province, which borders Iran and Afghanistan, have long complained that it does not receive a fair share of the profits made from its mineral wealth.
Prime Minister Imran Khan said Friday's attack would not undermine the Pakistan-China relationship, which he described in a statement as "mightier than Himalayas and deeper than Arabian Sea".
China, one of Pakistan's closest allies, has poured billions into the South Asian country in recent years as part of CPEC, a massive infrastructure project that seeks to connect its western province Xinjiang with the Arabian Sea port of Gwadar in Balochistan.
The project is one of the largest in Beijing's "One Belt One Road" initiative, comprising a network of roads and sea routes involving 65 countries.
Pakistan sees the project as a "gamechanger", but it presents an enormous challenge in a country plagued by weak institutions, endemic corruption and a range of insurgencies in areas slated to host the corridor.
The subject of economic dividends from CPEC is extremely sensitive in some of those areas -- particularly in Balochistan.
Since the beginning of the project militants have repeatedly attacked construction sites, blowing up numerous gas pipelines and trains, and targeted Chinese workers.
In August this year three Chinese nationals were among six wounded in a suicide attack on a bus transporting Chinese engineers working in Balochistan, in an attack that was also claimed by the BLA.
The Pakistani military has been targeting insurgencies in the province since 2004, and has been repeatedly accused by international rights groups of abuses there.
Islamabad regularly accuses its eastern neighbour India of funding and arming Baloch separatists, and of targeting development projects in the province, particularly CPEC.
India's foreign ministry swiftly condemned Friday's attack in a statement, saying: "There can be no justification whatsoever for any act of terrorism".