Don't make the mistake of addressing the symptom rather than the cause. The first thing you should do is perform a kinematic analysis of your cam and follower system. Once you have determined the forces, sliding velocity, and contact pressure at the cam/pin interface throughout a cycle, you can make a more informed decision on what corrective action to take. The best approach might be to modify the pin diameter and/or the cam profile. Or it might involve using a different material for the cam.
If I were to venture a guess of what is causing your problem based simply on the limited information provided in the OP, I'd say your cam profile and pin diameter are creating a very high pressure angle at this particular point. And the high pressure angle is producing excessive contact stress on the plastic cam surface.