Drilling two thru holes issues with perpendicularity, same spindle speed and feed all for all holes, but having issues with 1 or 2 holes being misaligned?
drilling thru 2 walls like that causes to wander you been lucky, slow down the feeds and speeds that is your fix.
are you using carbide drills ? which would be more rigid but are easy to break when there is moment or shock loads.
sometimes its easier to drill and ream from one side , then flip over use tooling pin and diamond pin. use the tooling pin as zero.
drill second side and ream. shorten the tools
some times its easier to use center cutting end mill to machine the holes.