"There was a bunch of ashes going up in the air and falling down and everything and it looked like a volcano," Quincy Grant said.
The fire burned into the day on Friday.
Mason Grant said the building had been an eyesore for years.
"That thing should have been gone years ago," he said.
Fire crews walked Eyewitness News a bit closer so cameras could capture a better look at the building. The fire chief said it had been abandoned for between 20 and 25 years. He said it was in bad shape before the fire.
The bulk of the blaze was centered in a five story section of the building.
The fire was deja vu for the fire department. Firefighters said it conjured memories of the Nova Dye factory fire.
"It would be great if the owners would tear them down," said Chief David Martin, Waterbury Fire Department. "Obviously when it gets to the point it's not rehab worthy, it's just sitting there as an eyesore."
In the Bristol factory case, the building had an owner. Sources told Eyewitness News it was Drubner Commercial.
A city spokesperson said it would have been on the owner to tear down the building before the fire. It takes funding for the city to be able to step in and help.
On Friday, Eyewitness News reached out to Drubner Commercial several times. It did not respond to requests to comment.
"There are challenges, a lot of challenges," said Joe Geary, chief of staff, city of Waterbury. "In the case you can find an owner to remediate these properties, they're extremely expensive to remediate."
Geary said the city will be working with the Environmental Protection Agency to come up with a cost for remediation.
Flames tore through the old Nova Dye Factory in Waterbury about one year ago and now officials are focused on cleaning up after receiving a $2 million grant from the state. State officials believe thatMore >
Flames tore through the old Nova Dye Factory in Waterbury about one year ago and now officials are focused on cleaning up after receiving a $2 million grant from the state.More >