Have you established a threshold damage level? What amount of damage do you induce above that level?
How much product are are you losing from ESD damage? How many product returns are traced to latent ESD damage?
Are your workbenches and surfaces already ESD guarded?
The simplest answer is that if you are not losing any product, then there is no immediate damage, although there might be latent damage.
If you are already losing product, then it would seem to be prudent to use proper gloves.
TTFN