I think some caveats are in order.
a) The bolt holes are always lager than the bolts. Therefore, you have the potential for those bolts to move around in the hole a little. This can potentially increase deflection. Probably not by much, but it would depend on the type of connection, the type of loading, et cetera. Way worse for a cantilever with a flange plate moment connection at one end. Not so bad for a redundant joist with clip angles or shear tabs.
b) If the bolt is fully pre-tensioned this will greatly reduce any movement within the bolt hole. It may not be designed as slip critical, but it shouldn't slip almost at all under service level loads.