So I've had some old installations of Windows (used to be 8 and before that 7) for quite some time. I've NOT been able to delete these old folders until now. After trying all the tricks I had up my sleeve, I ran into this article that explains it to the t! Thanks Scotties Tech Info for this great HOWTO!
Basically, you have to take ownership of the folder and then remove SYSTEM and TrustedInstaller from the list of users who have access to it. I wonder what happens if you skip one of those 2 steps?