Yeah, I was working like this but it is pretty much time consuming also :). I have moved Replace View Icon to the quick access toolbar... this is better but still...
What I would like to have is using popup selection toolbar similar to the filter for objects (point, curve, face, body, etc....)...