NX 4.0.2.2 Initially did the same thing, and so I will describe what I found.
If I create a part in NX-3 with the view set to what was called "Invisible Hidden Edges" then it works in NX-3. I knew this I just needed a more familiar starting point.
See also
you need to read it really carefully because things have changed a little.
If I try the same thing in NX-4 setting the view to "Wireframe with Hidden Edges" then I get the same error message as you did. Note that the setting "Invisible Hidden Edges" doesn't appear to exist any more, post NX-3.
But if I set the view back to static wireframe, then I go into Visualization Preferences, find the Visual Tab, and then under that tab are two more tabs, General and Edge Display. You'll find the settings you need under the Edge Display Tab. To get the right kind of display I turned off Silhouettes and then manually set Hidden Edges to Invisible. I think only the second of these is important, but I work with a spacepilot so I rarely display silhouettes anyway.
This was what I meant by reading John's instructions really closely. It isn't set to do what you would want it to using the icons, out of the box so to speak. Not to mention that once you mess about with the settings the icon actions appear to make less and less sense to me.
Anyway after that of course it worked just fine.
It is also pretty much the same in NX-5 by the way.
As a side note I saved my NX-3 file as set to what worked and then opened it in NX-4. Of course the Shadow Outline curve extraction worked just fine, but then the view was behaving strangely when I tried to use the Icons to change modes of wireframe display.
John,
Is there any way we can get the icon fixed to a setting that allows the curve extraction to work by default, or is there a downside to this that we don't know about?
Best Regards
Hudson