Ian,
The heat in a gearbox is generated thru several mechanisms. If your gears have high pitch line velocities, then oil windage losses can predominate. Oil churning losses in bearings can also be significant if scavenging and drainage flows are not paid attention to. And if your gear meshes have lots of sliding, they will require high flows.
If your gears are supported in rolling element bearings, then they are fairly well thermally isolated from the housing structure, and most of the heat transfer out of the gear teeth/rims must be accomplished thru oil jetting. The difficulty with doing this is that the oil spends very little time actually in contact with the gear teeth, so it experiences only a modest temperature rise before it is flung off. With a small temperature rise, you must naturally increase mass flow rates to effect your desired heat transfer rate.
My experience is mostly designing aircraft transmissions with aluminum cases. The total mechanical (gears, bearings, splines, seals, etc.) and windage losses of these systems is usually about 4%. As a rule of thumb, I would recommend starting with a bulk oil flow temperature rise of about 40degF, from jet to sump, to be conservative. We typically design for oil inlet temps of 230degF max under normal conditions with mil-spec gear oil, so bulk oil out would be 270degF max. The oil system volume is also normally kept very small to minimize weight, so the oil turnover rate thru the system is naturally very high. Usually around 4X per minute.
Having said all that, the heat transfer variables across the wall of an aluminum casing is something you'll have to make some assumptions about until you get your hands on some test data. You can easily determine the thermal conductivity of the aluminum, you can analyse for the total exterior surface area, you can safely assume that the inner surface of the casing is at (or near) bulk oil temp, but you'll likely have to guess at the local ambient air mass flows and temperature. Once again, as a conservative rule of thumb to start with, I would recommend assuming about 10% of the heat load being rejected thru the case wall structure with an aluminum housing in static air.
The other 90% of the heat load must be rejected thru your cooler. Unfortunately, the cooler core always ends up being a lot larger than you would expect, since you must size it for conditions of max power, on a hot day, at high altitude.
Good luck,
Terry