Try Steinberg's book "Cooling Techniques for Electronic Equipment", which talks about estimating electronic equipment temperatures by hand.
Did you mean CFD analysis? Because you won't get very far with an FEA analysis, unless you are depending on conduction to remove the vast majority of your heat - FEA packages are not capable of calculating the heat transfer coefficients from natural or forced convection, and these can change drastically depending on your cooling scheme.
No easy answer for that - it depends! If it is totally closed (no air moving in or out of the cavity), and you have a good conduction path (like a PCB with a metal core, which is then connected by a good thermal path outside of the box), then conduction may be enough. If all of the inside surfaces are at close to the same temperature, than radiation might not be significant.
But remember, the heat has to go somewhere! So at some point the exterior of your box or package will have to remove the heat, and you may have convection/radiation there.
Interesting problem, I remember a course on this exact thing in college. Many aerospace applications I understand have exactly this type of system. The electronics are inside a sealed box, with the edges of the PCB having cooling air or water flowing past them, such that the edges of the PCB are at a (relatively) known temperature. The PCB is attached to a heat sink, such as an aluminum plate and the heat carried out to the sides where it is removed by the air or water. This is done in aerospace applications for reasons I'm not totally familiar with.
In that course I wrote an FEA program to do exactly what you're asking for in Basic. It was a very cheesy program as you may imagine a senior in college doing it, but it worked. You may consider writing your own if you feel comfortable with that. It doesn't need to have a high tech look to be effective.
To answer your question, I would guess that the vast majority of the heat transfer is conduction through the heat sink. Here I'm assuming the system you have is similar to those I studied for aerospace applications as described above. I'd suggest ignoring convection and radiation heat transfer effects as they are likely to be low and if they are not specifically designed into the package, I'd suggest these effects are not controllable and reliable.
