It's not a "pressure" thing.
Imagine removing all but a narrow annulus next to the circumference of the hole. Apply temperature. Since the linear expansion along the circumference of the annulus is positive, the annulus must grow in diameter to accommodate the new circumference, so the hole grows. Look at the next annulus, same thing.
Note however, since its circumference is longer, the second annulus has a larger diameter growth relative to the width of this annulus, so there's no "push" from the outer parts of the washer.
Note also, that growth in width of the annulus from expansion is miniscule compared to the growth in diameter from the circumference increasing, so, again, nothing pushing to make the hole smaller.
TTFN
FAQ731-376