gwang,
All you need to do is look at the numerical displacement results at a few key nodes, as reported by your FEA program.
Like I said, the actual displacement of the perimeter nodes should be purely tangential (assuming an axially symmetric shaft under pure torsion, of course!). This is a very easy check to undertake.
Simply get the (x,y,z) displacement values at the selected nodes, and see which direction the true resultant displacement vectors point. If the true exaggerated vectors are tangential, your analysis results are as expected; if not, you have some sort of problem.
As per my first example, suppose we have a shaft which has a radius of 10 mm , with its axis lying on the Global Z axis. Take a node on the free end at Theta = 0 degrees - i.e. lying on the X-axis, with (x,y) coordinates of (10,0). We expect the actual displacement of this node to be a small displacement in the Global Y direction - perhaps 0.01 mm say. This is too small to see, so we usually plot with an exaggerated displacement scale.
However, when we plot the displaced shape with a scaling factor of 1000 say, the apparent displacement of the node is now 10 mm in the Global Y direction. That is, the undisplaced node coordinate was at (x,y) = (10,0), the true displaced position is (10,0.01), which is visually indistinguishable from the undisplaced position, but the exaggerated displaced node position is plotted at (10,10).
Repeat this example for all the nodes that lie on the perimeter of the free end, and it becomes apparent that when plotted with a large exaggerated deformation scale, the shaft appears to twist AND expand radially at the free end. Like I said, this is just a consequence of plotting with exaggerated deformations.
Hope this helps!