The point I was trying to make is that FRF is output spectrum divided by input spectrum; which variables they are is not as relevant, perhaps I worded it wrong.
If you obtained the displacement response at location B due to a unit sinusoidal load over a frequency range at any input location A, because its a unit force, the magnitude will be correct for FRF (divided by one), but you still need to include phase shift. Remember when you divide one complex number by another, the magnitudes divide out, but the phase angles are subtracted. Thus his output will contain the magnitude same as FRF but not the phase.