Yes. Mold and mildew can evolve into "wood destroying organisms", if left without treatment.
The conditions that allow mold growth (high moisture, food supply (the wood))will cause wood to deteriorate at the cellular level. This will, obviously, affect the structural integrity of the wood.
You can remove the mold/mildew with chlorine bleach, but you need to be sure you have stopped any moisture intrusion or accumulation on the wood, as mold and mildew will return.
Check the vapor drive through the wall. Make sure you don't have a vapor barrier in the wrong place. There are lots of myths considering moisture intrusion, vapor drive and permeance of the materials. Even though theoretically materials may be designed to "breathe" and release moisture accumulations, in reality that doesn't work well at all. If water gets into a wall cavity, there is nothing that will drive it back out in any reasonable time frame. It stays and does huge amounts of damage.