The blaze broke out at the four-storey building in the city's congested old quarter early on Sunday morning.
At least 100 people were sleeping inside the factory, which mainly makes school bags, when the fire started. More than 60 have been rescued.
Prime Minister Narendra Modi called the fire "horrific" and sent his condolences.
A local fire chief told BBC Hindi the building did not have a proper fire licence and was operating illegally as a factory.
Local media reported that the owner of the factory, named as Rehan, had been arrested.
One firefighter, Rajesh Shukla, was hailed as a "hero" for saving 11 people - despite suffering injuries himself.
Delhi's firefighters received the first call about the fire at 05:22 local time on Sunday (23:52 GMT Saturday).
The fire began on the lower storeys, spreading rapidly to the third floor where workers were sleeping.
"We woke up with cries and shouts for help," said Ronak Khan, a 17-year-old living next door.
"I saw people trapped. We asked them to come to the rooftop so that we could rescue them but they were not able to come up."
The area where the factory is located - Azad Market - is a web of narrow alleyways, which made it difficult to reach the blaze.
Rescuers had to carry out victims on their shoulders one-by-one with firefighters cutting away window grills to access the building.
It is not clear what caused the blaze but an investigation has been ordered.
An initial probe and eyewitnesses suggested a short circuit may have been to blame.
By Anant Prakash, BBC Hindi, Lok Nayak hospital, Delhi
Mohammad Haider's brother Bablu worked in the factory.
Haider says he rushed there and was told: "Your brother has been saved."
"I was relieved and rescued three to four other people.
"But then I learned that my brother was actually inside. Afterwards, I found him lying in the mortuary."
Mohammad Haider is shaking. Two other men are trying to console him.
"I had never thought that my brother would be separated from me this way," he says.
"Bablu used to inform me about everything in his life, whatever he was up to.
"Even when the fire started, he called me and said, 'Brother, please save me'. But I could not save him."
Victims' relatives have been scrambling for information. One man told India's PTI agency his brother was inside.
"I got a call from his friend informing that he has been injured in the incident. I have no clue which hospital he has been taken to," he said.
Indian cities have often seen deadly fires, with poor planning and lax enforcement of safety regulations major factors.
Indian politicians have been expressing their horror at the blaze.
"The fire in Delhi's Anaj Mandi on Rani Jhansi Road is extremely horrific," Prime Minister Modi tweeted.
"My thoughts are with those who lost their loved ones. Wishing the injured a quick recovery. Authorities are providing all possible assistance at the site of the tragedy."
Home Minister Amit Shah called it a "tragic loss of precious lives".
