The blaze broke out at the four-storey building in the city's congested old quarter early on Sunday morning.
At least 100 people were sleeping inside the factory, which mainly makes school bags, when the fire started. More than 60 have been rescued.
Prime Minister Narendra Modi called the fire "horrific" and sent his condolences.
A local fire chief told BBC Hindi the building did not have a proper fire licence and was operating illegally as a factory.
Local media reported that the owner of the factory, named as Rehan, had been arrested.
One firefighter, Rajesh Shukla, was hailed as a "hero" for saving 11 people - despite suffering injuries himself.
Delhi's firefighters received the first call about the fire at 05:22 local time on Sunday (23:52 GMT Saturday).
The fire began on the lower storeys, spreading rapidly to the third floor where workers were sleeping.
"We woke up with cries and shouts for help," said Ronak Khan, a 17-year-old living next door.
"I saw people trapped. We asked them to come to the rooftop so that we could rescue them but they were not able to come up."
The area where the factory is located - Azad Market - is a web of narrow alleyways, which made it difficult to reach the blaze.
Rescuers had to carry out victims on their shoulders one-by-one with firefighters cutting away window grills to access the building.
It is not clear what caused the blaze but an investigation has been ordered.
An initial probe and eyewitnesses suggested a short circuit may have been to blame.