I know what you mean, but this is not an easy thing to 'fix'.
What's happening is that when you open a function, which either creates or modifies some object, an 'Undo mark' is set (which will be used later to 'roll-back' your model, by perfroming an 'Undo', to the state that it was in before you created/modified whatever using this function). And if you were to hit the 'Cancel' button while in the dialog, we roll=back to that 'Undo' mark primarily because some functions are actually creating internal objects which you may or may not even see. Take for example in that Extrude example that you mentioned, it might have created some 'Preview' geometry, and we need to get rid of that IF you decide to 'Cancel' the dialog. Granted, we could have used some sort of 'delete' routine to get rid of these temporary objects, but the Undo mechanism will do it for us automatically no matter how many these objects were created.
It works every time with no fuss or muss. And besides, if we did NOT roll-back, the 'Undo Mark' itself would not be deleted unless again we did something special to delete IT as well as any temporary geometry that we had created while in the now canceled function. And if we did NOT get rid of these now unneeded (i.e. 'empty') 'Undo Marks' they would still be there so that when you had created/modified some other objects and you decided to Undo that you might have needed to press Undo several times to get back to where you actually wanted to be since there might be several of these 'empty' Undo Marks (for everytime you opened and then canceled a dialog) in your session, and trust me,
we would hear about that from users as well.
Unfortunately, while inside a function, most anything that you do using what we call 'special functions' (these are operations which CAN be performed while inside of a function without having to first leave that function, such as change Layers, Hide/Show an object, toggle ON/OFF the display of the WCS, etc.) will get cought up in any Undo operations performed after these changes were made even if they were done while performing some other function. Now while it is true that most of these so-called 'special functions', if performed while NOT inside of another function, will set their OWN Undo Marks, we can't set Undo Marks INSIDE of of another 'Mark' since there is NO way to support 'parallel' sets of marks. That is, there is ONLY one steam of actions which you can Undo back through. To try otherwise would be very complex and could lead to other anomalies which could prove to be even more annoying.
And as for this behavior somehow not being evident prior to NX 5.0, I don't think that's true. This is how Undo has always behaved with respect to performing any of these 'special functions' while in a dialog. In fact, at one time is was worse than it is now. It used to be that if you changed you view by doing a Zoom or Rotate (either with a Spaceball or your mouse) or a Fit operation, that those changes is display would ALSO be Undone. Now that was REALLY annoying and so about 10 release ago or so, we took simple display changes completely out of the Undo system. Now no matter how many Undo operations that you perform, the orientation/scale of the display of your model will be unchanged (now this does NOT apply while working in the context of a Drawing as there is a different display mechanism utilized there which cannot be left unchanged by Undo operations).
Anyway, I hope you can see that while we understand that this behavior, as you've described it, can be annoying, it is something that would be very difficult to change and any attempt to do so could potentially create other behaviors which could be even more of a nuisance.
John R. Baker, P.E.
Product 'Evangelist'
Product Engineering Software
Siemens PLM Software Inc.
Industry Sector
Cypress, CA
Siemens PLM:
UG/NX Museum:
To an Engineer, the glass is twice as big as it needs to be.