For me, 1 inch is reasonable for retaining sensitive areas and 2 inches for less sensitive areas. Anything more would require input from the contractor and owner about potential issues regarding deflection.
Keep in mind that your calculated deflection may not be accurate. The software we use in-house seems to predict high deflection, especially if we're required to use conservative soil values. Often, we have to caveat design submissions that, based on experience, we anticipate will have less deflection than shown in the software's report.
Regarding references, FHWA-RD-75-130 "Lateral Support Systems and Underpinning," dated 1976 by Goldberg, Jaworski, and Gorden, discusses deflection and settlement in Volumes II and III. In Volume III, they provide a summary of:
For Competent Soils (granular soils, very stiff clays, etc.):
"Maximum displacements are typically in the order of 0.25 percent to 0.35 percent of wall height. The lower range is associated with granular soils; the upper range is related to cohesive soils.
Typically, maximum horizontal and vertical displacements are about equal."
For Weaker Soils (soft to medium clays, organic soils, etc.):
"Maximum displacements typically exceed 1 percent of depth of cut for flexible walls such as steel sheet piling. Concrete diaphragm walls dramatically reduce the magnitude of displacements to about 0.25 percent of the depth of cut -- or about the same as those observed for competent soils."
Typically, the maximum vertical displacements exceed maximum horizontal displacement.
When the excavation is underlain by deep deposits of weak soils, the cumulative total of all displacements occurring below the last placed strut level amounts to about 60 percent of the total measured movement."