Depends what the listing covers Crawford. Think that old bit of the factory was listed because there was a mural in it at some point*
Stuff like the mill conversions or Robertson's building in town are generally ok because its the facade thats seen as important, the interior is just an empty space with no historic features.
Demolition or alteration of a listed building isn't that uncommon if its not deemed unique and important enough to keep and can't be converted like the mills
* See shadows post for more detail