A house (the people in it, actually) needs ventilation, the tighter the envelope, the better. If you are using exhaust fans in the house and have no "air system" - where does the make-up air come from?
If the house is built tight, and then if there is no make-up air system, outdoor air will be drawn in through every little crack it can find in the envelope - in an uncontrolled manner, which in winter time leads to moisture condensation at the cold spots, then mold. If you have bathroom and kitchen exhaust, you need a source of controlled make-up air. Normally if you can supply all the heating needs from the radiant system, the usual method is to provide an HRV or ERV to provide the necessary house ventilation (exhaust and make-up air). And in colder climates, even the most efficient heat recovery unit will need some touch-up heat in the supply air.
Sounds like someone needs to take a good look at the "house as a system" here and sort things out.