Engineering analysis (alternative to the 3% rule) is explained in API 520 Pt. II, 6th ed., section 7.3.6. Included is a high-level explanation of the suggested steps and suggested analyses. There are two analyses. One is a force balance and other is an acoustic analysis. I suggest you get a copy of the ioMosaic papers in references 31 and 32.
Notice that API 520 doesn't prescriptively define a particular engineering analysis methodology. But this is a complex topic, so I don't suggest going off on your own and creating an engineering analysis. Over the last decade a lot of research has been done on PRV stability analysis, and there are process safety engineering companies that are practicing analysis methodologies based on that research. ioMosaic's website advertises a seminar in which they teach how to do these engineering analysis calculations. Alternatively, you can pay one of these process safety companies to do this work for you.
This is still an evolving technology. There's no guaranteed way to be certain that a PRV installation will not cause the valve to chatter. However, what is known based on research and testing is that the legacy 3% rule isn't sufficient for predicting chatter. This research has shown that a PRV with 2% inlet loss may chatter, and a PRV with 10% inlet loss may not chatter.