DimaC1974,
I design high-speed aircraft gear trains, and the (NASTRAN) method I described to you is what we use. Performing an accurate system modal analysis for a complete transmission requires lots of DOF's, which means lots of calculations. NASTRAN is the only application that I know of that can handle such a complex analysis, but as I said I'm just a novice with FEA.
As for the mathematics that NASTRAN uses to derive these solutions, they can be obtained from any decent textbook. The difficult part is writing the code that allows your computer's processor to perform the numerous calculations in an accurate and timely manner. And that's why people pay big bucks for a NASTRAN license.
I'm assuming that since you're after eigenfrequencies in a gear train, you want to check for load effects caused by coupling of structural modes due to meshing frequencies. A static FEA analysis will only provide part of the answer though. Besides torsional vibration effects, dynamic gear tooth loads also can be significantly increased due to profile and index geometry errors, high pitch line velocities, or low contact ratios. In fact, gear tooth geometry errors can result in dynamic load factors much greater than torsional vibrations produce.
Since your parts are modeled in SolidWorks, the most compatible FEA application is probably Geostar. Geostar is relatively inexpensive (at least as far as FEA applications go), and it will give you the various structural frequencies of individual parts, provided the mesh is not too complex. But I don't think that Geostar has the capacity to do an analysis with a large number of elements or DOF's that a full transmission system would require. I might be mistaken, so check it out yourself:
Good luck.
Terry