I can't recommend any books, but there are two suggestions.
1. If you need to connect with an off-the-shelf part, like a coupler or yoke, look at whats already available. If this is going to be a production part, find out what profiles your suppllier normally deals with.
2. Generally the spline count goes up with load and/or rpm's. I'm not sure what the exact reason is, but it probably has to do with having a larger rood diameter for strength and more area to transmit torque.