In my somewhat limited marine gear box work, I found the contact stresses on the gear teeth to be very scary, well over 200 ksi., as you no doubt will too if the gear box is to be kept small.
The dog clutch diameter is probably less than the gear pitch diameter and the torques are the same so the forces are higher yet. Anyway you look at the thing the stresses get to be more than you want. There is probably just edge contact at the initial engagement so I don’t think there will be any reassurance in the stress levels obtained through modeling.
If you still inist on modeling, an estimate found for the interia of the water around the prop was 25% of prop inertia (BICERA handbook on torsional vibration). Combining that with the spring rate of the elastomeric coupling and the prop shaft might get you a number but I wouldn't put too much faith in it. The mechanism for getting the prop spinning probably involves the energy expended in briefly ratcheting the clutch and gear teeth together to create the horrible noise we associate with dog clutch engagement.
Consider S7 tool steel for the clutch.