I agree that a Finite Element analysis is great for these, even if you have to model the curve in short segments. Be careful with open sections as the AISC Design Guide for complete warping torsion checks are not part of most packages. If you can use a closed shape (HSS Rectangle) life is easier. Finally, consult with a fabricator or steel bender to find out what shapes can be bent before you go too far. I see advertisers in Modern Steel Construction all the time.
I must agree with SethGuthroie, use closed steel sections. They work well in resisting torsion. There are specialized fabricators who will bend and curve them and they come out stunning when done. Of course with help from the architect who will cover up the structure.
The links below will take you to two pictures of two curved stairs that were in the same building:
We design these frequently. We use Risa, since the design examples for stairs assume they are straight spans. We typically use plate or HSS material for the stringers. And depending on the tightness of the curve HSS sections may need to fabricated from plate. There are several subtle considerations in the design and connections. An experienced fabricator or detailer may be able to provide some suggestions. Or, there are designers that have experience in these conditions.
We build these from time to time. Steel channel seems to be the preferred material as the flanges help to highlight the helical shape. Typically oversized stair treads are used to help reinforce the stringers. I will say that shop drawings from which fabricators can produce these are a headache. As a starting point you can analyze the stringers as equivalent straight members in combined bending and torsion.
