paulmech01: The equation in my first post was found to be incorrect.
I investigated further and found that when one tube is eight times more flexible (per unit length) than the other tube (as in your case), the required engagement length appears to be 5*D. Greater than 5*D seems to exhibit little or no change, theoretically. (Note that 6*D might be effective if both tubes are relatively stiff and of comparable stiffness, which isn't the case in your assembly.)
The peak longitudinal shear stress on your fibreglass tube is 1.47 MPa. I think the longitudinal shear strength of the polyester resin, alone, is no less than 8 MPa (maybe higher), so this indicates your fibreglass tube wouldn't split longitudinally. And, your pultruded tube has E-glass matting (fabric), so the weft fibre adds additional longitudinal shear strength, as well as some hoop strength, since there would be significant reinforcement fibre in the hoop direction.
It appears that the contact pressure exerted on your fibreglass tube by the lip of your aluminum tube is concentrated in only a small, 12-mm-wide band, and peaks at roughly 53 MPa on the fibreglass at the top 5 mm of your aluminum tube. (The aluminum tube is below yield.) I'm not sure, but I think slightly over 20 MPa of uniform external pressure exerted over a larger area of the fibreglass tube might cause a local stability problem (?), assuming the fibreglass tube modulus of elasticity is 14 GPa. I don't know if the smallness of the 53 MPa spot will (a) precipitate a stability problem, or (b) be able to sufficiently redistribute causing no problem. If you're concerned, you might consider gluing a 15 mm high, 5 or 6 mm thick piece of aluminum tube inside the fibreglass tube adjacent to the outer aluminum tube top edge, which might prevent the fibreglass tube from being able to collapse inward locally at that contact pressure point.
Regarding your last question, hoop stress is typically calculated as force per unit area on a pipe wall projected to a plane and divided by wall thickness; e.g., p*D/(2*t) for uniform internal or external pressure. But the stress in your tubes is due to bending, not uniform pressure, so the stress changes to mostly shear stress as you go around the tube 90 deg to the neutral axis. In other words, it may not be straightforward (nor meaningful) to calculate a hoop normal stress. The contact stress at the aluminum tube lip is almost a point load (or cosine contact band) exerted on the fibreglass tube, and dissipates as you move away from that point.
