mikewesterfield,
Are you tring to apply a tolerance, or interpret one?
If you trying to interpret an existing tolerance, make sure the drawing has a reference to ASME Y14.5M-1994 or its ISO equivalent. Otherwise, the tolerance is not based on a standard, and cannot be interpreted.
If you show your tapped hole on the centre line without a positional tolerance, I think you are on your own. I am not sure how I would interpret that under ASME Y14.5M-1994.
If this were my drawing, I would specify a geometric tolerance, and call up the pully face, the bore and the top surface of the keyway as my datums (data?). My tolerance would be fairly sloppy, since probably, I don't care that much about where the set screws go.
Getting tapped holes into the right place cannot be that difficult. Regardless of GD&T and your drawing, perhaps you need a new vendor.
JHG