To expand on my previous post- if you have a test program, there should already be some additional factor of safety built in via the statistical reduction (A-Basis confidence level for example). If you have a riveted sheet you have a combination of test allowables (rivet) and a well understood joint behavior in the sheet; it is highly unlikely that a rivet joint designed to be critical in bearing will fail at or before the design load (assuming good practices are followed). So in these cases, the allowable load is not the load at which you would predict the failure to occur at, rather it is the load at which you would predict - with high statistical confidence - that there would NOT be a failure.
Now compare that to the case of a machined or cast "fitting", where you are determining the allowable load via a stress anlysis based on material properties and joint geometry. This allowable load DOES represent the load at which you would predict failure of the part. In this case, there is no built in safety factor. (I wouldn't consider material properties as having a built in factor of safety for stress analysis since they represent minimums for material cert - though if I were to place a wager on a failure load I might). So in these types of cases, I would apply the fitting factor.
I would apply the same general principals to most joints, especially critical ones, whether or not they meet a given definition of "fitting".