AISC Code of Standard Practice provides acceptable erection tolerances (somewhere in the L/500 range which would put you at about an inch for the 40 ft. height).
See Section 7.13.
For your situation, I would suggest the following course of action:
1. First determine what you've actually got - this would involve a survey of all the building columns, identifying the degree of out-of-plumbness as well as the direction of the tilt.
2. Each column, then, will have a vector force due to that mis-alignment at the top of the column. This will take the form of Dead(x) and Dead

along with RoofLL(x) and RoofLL

at each column.
3. Then - map out your diaphragm in plan, with each column force.
4. You can then perform another lateral analysis, using the code prescribed wind/seismic loads in addition to the appropriate lateral forces from non-plumb columns. Include all required load combinations from your governing code.
5. With each load combination, re-check you diaphragm design and lateral bracing system design.
6. Add strengthening if required.
Now the above assumes that you aren't going to make the contractor take everything down and re-erect the structure square and plumb. You probably would want to consider this as a first choice as 3 1/2" out of plumb is a bit much.
But the steps outlined above are a rational response to a given situation that cannot be avoided by re-construction.
If the columns are all out-of-plumb in one direction, it could be that the whole frame simply needs to be drawn back to a better alignment. This can be very difficult with a large building. With this condition, the lateral forces from the mis-alignment would all work one way and probably diminish your lateral bracing capacity.
If the columns are out of plumb in various directions, they may simply counteract one another and have minimal effect on the overall frame stability.
Finally, with out-of-plumb columns, you would also generate local lateral forces on your framing. The beam or joist connections would be the first thing that would transfer the lateral non-plumb forces to the diaphragm and these should be checked.
Sorry for the rambling above - just some thoughts as they came to me.