I have had the same problem, and unfortunately, been unable to fix it consistently. Here are a few more possibilities:
Groups have the same issues as components. You can move/delete the entities in a group and leave the 'group' entity behind. If it is unnamed, PartCleanUp will get it for...