So, there is an eccentricity between the centroid of the applied axial force P (which coincides with the centroid of the beam) and the connection. Therefore, this produces what is hopefully a small moment (P*e) in the connection.
There is also an eccentricity between the shear V (applied at the bolt line) and the face of the column flange. This produces a similar moment (V*h using Rb1957's sketch).
So, yes, there is some additional eccentricity that could / should be considered in the design. There are a number of ways to go about doing this.
There is a book, "Handbook of Structural Steel Connection Design and Details" by Ankar Tamboli. I believe this book suggests that when the axial load is small you should just combine it with the shear load Veq = sqrt (V^2 + P^2) and continue designing it as a pure shear connection. That should be fine when the shear force dominates the behavior. I would think that would be the case for your connection.