The blaze broke out on the ground floor of a residential building where flammable material was being stored, just before midnight when most of the occupiers were asleep.
Many people were trapped and it is thought that the victims include guests at a restaurant and a wedding party.
Firefighters were hindered by lack of water sources in the narrow streets of the historical Chawkbazar district and they fought for more than 5 hours to quell the inferno.
Most of the bodies retrieved were charred beyond recognition and the number of victims is expected to rise as the search continues.
It is not yet clear how the fire started but the district is a hub of chemical businesses and perfume factories and was the scene of a major fire in 2010 which led authorities to ban the storage of chemical goods.