After spending hours playing with calculations and digging through the NEHRP design guide, I've found some solid answers to my question. I'll post them here for future reference for others.
NEHRP recognizes that some walls will be shear-dominated no matter what you do. In such cases, an R value of 5 per ASCE 7 is not justifiable. NEHRP recommends using an R of 1.5 when shear-dominated behavior is expected. Most low-rise construction falls into this category. R of 1.5 is essentially elastic behavior. The effect of the flanges on the wall ductility becomes a moot point, although they still do affect stiffness and load distribution.
Quoting the NEHRP Design Guide:
Design of Shear-dominated Walls
It is the implicit intent of ASCE 7 and TMS 402 that special walls be fexure-dominated, with behavior that justifes the response modifcation factor R=5 assigned to them. However, as discussed in Section 3, walls designed in accordance with the provisions of these codes may still be shear-dominated. There is little direct guidance in TMS 402 to address this situation, which raises concerns that require the engineering judgment of the designer. The frst is the possibility of brittle shear failure in elements of the gravity load bearing system, albeit at lateral loads that may be much higher than the code-required design loads. This Guide does not address the complexity or repercussions of this possibility, nor does it advocate one approach or another to address it, because too many scenarios are possible.
Another implication of shear-dominated behavior in wall elements is the understanding that such elements may behave nearly elastically, attracting forces much higher than those calculated using a response modification factor R=5. This has implications for both the wall element itself and for other elements of the seismic load path that will have correspondingly increased demand. Such elements may include diaphragms, diaphragm chords, connections of diaphragms to shear walls, and collectors.
One approach available to the designer would be to design all elements of the seismic lateral load- resisting system associated with shear-dominated walls using a response modifcation factor less than the code value for special reinforced masonry walls (for example, R =1.5).
And again, in Section 3 of the design guide:
The implicit goal of TMS 402 is that special masonry shear walls be fexure-dominated and ductile. The code indirectly encourages designs that meet this goal through prescriptive requirements for distribution of reinforcement, limitations on bar diameters, maximum reinforcement restrictions, and other provisions, but these requirements may not be suffcient to produce ductile behavior. When a special shear wall has a shear-span-to-depth ratio Mu/(Vudv) greater than one with a well-designed plastic hinge zone (Paulay and Priestley 1992), these requirements plus the required capacity-based design for shear generally result in fexure-dominated, ductile behavior. However, when a special shear wall has a shear-span-to-depth ratio less than one or a high axial load, the same combination of prescriptive requirements may still result in a wall that is shear-dominated and brittle. This is often the case for low-rise masonry buildings, which constitute most masonry construction in the United States.