Interesting problem, I remember a course on this exact thing in college. Many aerospace applications I understand have exactly this type of system. The electronics are inside a sealed box, with the edges of the PCB having cooling air or water flowing past them, such that the edges of the PCB are at a (relatively) known temperature. The PCB is attached to a heat sink, such as an aluminum plate and the heat carried out to the sides where it is removed by the air or water. This is done in aerospace applications for reasons I'm not totally familiar with.
In that course I wrote an FEA program to do exactly what you're asking for in Basic. It was a very cheesy program as you may imagine a senior in college doing it, but it worked. You may consider writing your own if you feel comfortable with that. It doesn't need to have a high tech look to be effective.
To answer your question, I would guess that the vast majority of the heat transfer is conduction through the heat sink. Here I'm assuming the system you have is similar to those I studied for aerospace applications as described above. I'd suggest ignoring convection and radiation heat transfer effects as they are likely to be low and if they are not specifically designed into the package, I'd suggest these effects are not controllable and reliable.
Have you done a web search for software packages?
Sorry I can't be more helpful.