All elastomer material is not the same. Just because you call it Buna-N, do not assume it. Compounds can very greatly and have the same hardness. When spec'ing/ordering, look at both the Shore A hardness and tensile strength. Suppliers often only talk in hardness.
Tolerance on the parts is also very large. A ptfe option may be something to consider. Assuming you are using a dashed unsplit o-ring. The ptfe may need to be replaced more frequently from cold flow, but will also act as a dry lubricant. Just a thought. However, this depends on parameters such as temps, speeds & pressures.
As discussed in earlier threads, do look at the dimensions, both theory and actual and also try without the o-ring as earlier recommended.
Surface finish where o-rings ride should also be inspected/reviewed.
Cheers,
MECOMAN