So I've had some old installations of Windows (used to be 8 and before that 7) for quite some time. I've NOT been able to delete these old folders until now. After trying all the tricks I had up my sleeve, I ran into this article that explains it to the t! Thanks Scotties Tech Info for this great HOWTO!
TL;DR
Basically, you have to take ownership of the folder and then remove SYSTEM and TrustedInstaller from the list of users who have access to it. I wonder what happens if you skip one of those 2 steps?