As mentioned by swall, a heavy scale can indicate excessive decarburization, which will result in low surface hardness. Typically, for as-rolled bar, you can expect between .015"-.030" of decarb. In excessive cases, maybe twice that. If you are performing a Brinell test, I'd guess you would have to grind about 0.060" to get a wide enough flat to perform the test, so this should not a problem. If, however, they are not grinding deep enough, you can get low hardnesses.
Descaling the material prior to the temper will not do any good, since if it is directly responsible, it is affecting the quench, not the temper. A rough turning operation on the OD will eliminate this problem.
Not knowing your end use or volume, this solution may not be pratical, but have you considered just buying some heat treated bar stock? 320/340 BHN in not all that uncommon of a hardness range and while 4150 may not be all that common, 4140/4142/4145 grades are readily available in the heat treated condition.
rp