I'm really going out on a limb here (I won't pretend to know a great deal about fatigue), but another possibility is that the plain material isn't really all that 'plain' on a microscopic level, especially along the rolled edges. The fatigue resistance is sensitive to the surface condition, so the presence of little flaws, mill scale, scratches, etc. would all reduce the fatigue life. A specific example is steel reinforcement bars - there are studies that have compared the fatigue life of conventional deformed reinforcement, plain round reinforcement, and also the same reinforcement bars turned on a lathe and polished to a mirror finish. The difference in fatigue life is a minor improvement going from deformed to round, but there is a large increase in fatigue life for the polished bars.
What I'm getting at is that potentially, where you have neatly drilled or punched holes in the steel, this leaves you with a relatively clean/smooth surface on the inside of the hole compared to the bare steel which might somewhat compensate for the 3x stress increase.
Another point is that the 3x stress multiplier only really applies for an infinitely wide plate, doesn't it? So for more realistic specimens, the stress multiplier may be closer to 2 or so.
