You can do it within a layout viewport.
Place your x-ref on a specific layer.
In a viewport where you want to see this x-ref, make sure the layer is thawed, on, and not frozen in viewport.
In viewports where you do not want to see it, freeze in current viewport. You must be in the active viewport to reach this setting.