DawidWas,
You said you don't care about orientation, but you do likely care about [u,v] ie: rotation about the x and y axes respectively (assuming x and y axes are in plane with B and z is perpendicular to B), just not necessarily orientation/clocking about the z-axis aka [w] rotation. Position tolerance also controls both orientation AND location of each hole axis to within the stated tolerance. This may all be intuitive, but I just wanted to clarify that when you say you don't care about orientation, you are likely only talking about one form of orientation (rotation about the z-axis).
As mentioned above it is important that our datum features are robust for simulation ie: the low length to diameter ratio of A could potentially make it unstable/unreliable as a primary datum feature but we also want to make sure our datum features reflect part function/design intent. You mentioned the example of a flange, which could also be your textbook example - if A is a clearance fit and is secured with a series of bolts the clamping force will tightly pull your surface B in contact with the mating part surface. In this case your datum feature B is likely the best candidate for the primary datum feature as it will constrain your rotation [u,v] and translation [z] in practice, and then A can be specified secondary to constrain [x,y] translation/location.
Your position tolerance can have one datum feature. It can even have no datum features (datumless). Whether or not that "suffices" is up to the designer and what the design intent is.