morris9791,
Dog clutch teeth are basically short, fat, cantilevered beams. You don't usually have a bending issue or a shear strength issue. So all you really need to worry about is surface contact fatigue and fretting. The contact condition under load is very similar to what you would have in a spline.
The most difficult variable to establish for your contact loading is load sharing among the teeth, if you have more than three teeth (with three teeth or less, the teeth should load share equally by definition). And also the effective contact area of each tooth pair can vary due to edge loading. Edge loading can be caused by deflections under load or torsional wind-up, or from manufacturing tolerance errors.
The only other thing you need to account for is how aggressive your driver is. If your driver is not a skilled shifter, you may want to alter things like tooth undercut, backlash spacing, and possibly adding helix to the top land surface.
One effective type of dog tooth geometry is the "Curvic" tooth form from Gleason. Unlike the more conventional straight flank tooth, the curvic tooth is self-centering and thus tends to load share more effectively.
Gleason's engineering has a design guide for curvic dog clutches.
Good luck.
Terry