there may be some more variables at play here. are you actually measuring the hole after drilling to see if the drills are drilling the hole to the specified size? if not, you may want to check this.
are you drilling the hole using an alignment bushing? are you spot drilling the hole prior to drilling? if not, you could have a condition where the hole could be oversized for a bit at the top of the hole. since you are drilling 3/8" thick plate, the drill will not have enough material to "straighten" itself out. basically, your hole may be egg-shaped.
what type of tapping fluid, if any, are you using? sometimes when tapping low carbon steel, the material will gall or stick to the tap causing the threads to be oversized. you can fix this by using a tapping fluid with higher amounts of sulfer to help with the galling.
i would strongly suggest that you by a go-nogo gage to properly check the threads. standard metric threads are 6H/6G.
the 8.5mm drill you were using in the beginning is the proper drill size and i would not deviate from using this size. just double-check the hole size after drilling to see if is correct. we produce these threads every day with no size problems. we use a 8.5mm, solid carbide, through coolant guhring drill. we use YMW taps and our threads have an effective depth of 20mm(3/4"

.
hope this helps a little.