Three others died of burns. Workers said some exit doors were locked. The multi-storey plant is owned by one of the country's largest exporters, Hameem Group.
Piles of clothing fuelled Tuesday's fire. Authorities are still searching for bodies, a local official told AFP.
He said the government has already launched a probe into the blaze and that initial investigations suggest it was caused by an electrical short circuit.
Some of the 100 who were injured were hurt in a stampede, some inhaled too much smoke.
"It happened so quickly, everyone rushed to the side of the building to jump off,” eyewitness Rezaul Karim told the local Sheershanews agency. “People at the bottom were holding out nets and blankets to try and catch them.”
Some 14,000 workers are employed at the factory, which produces clothes for American brands including Gap and JC Penney.