petronila The shaft diameter (for any application) is determined by how much mechanical stress is applied to the material and how much stress (tensile and ultimate yield) the material can take. Ideally, the applied stress is no more than 75-80 percent of the mechanical limitations based on material used. Keeping things relatively simple, the designers tend to use rated torque and a multiplier, then compare the result to the material yield limit. More difficult applications (metal rolling and/or reversing duty) have higher multipliers than general industrial (which is the NEMA "standard" sizing approach). This is why you see shaft sizes increase as torque throughput increases.
The shaft extension diameter is also a function of torque, in that the ability of the fit between shaft and coupling determines the amount of torque that can be successfully transferred. The maximum torque transmission capability is also a function of the contact pressure (usually a function of coupling wall thickness) and fit length (i.e., shaft extension length). Since the majority of coupling geometries for "standard" general industrial ratings are well defined, the resulting fit length is as well.
Also, since you're looking at "standard" NEMA ratings, chances are good the shaft material is a medium carbon steel (such as AISI 1040, 1045, or even A36) formed into a bar stock and then machined as required. Some applications (food industry for example) might use a different shaft material such as stainless steel, but how the calculation is made is the same - just apply the properties of the stainless steel instead of the n=more normal carbon steel.
A quick glance through Shigley would give you the basics for the calculation: all you'd then have to do is apply the appropriate material properties and "application" factors.
Converting energy to motion for more than half a century