From what I can see now you have a mechanism which has contact. I understand that an FE contact model is possible but it is not the same as a standard linear FE model.
I am not familiar with FE contact modeling. I know a few book that theoretically develop come contact mechanics models. However, the only thing numerical I can suggest for this is to try another dynamic and static modeling software like MapleSim.
Other than this, I could not say for certain.
If you have a nonlinear model (as with contacts that open and close during excitation), you can't do a modal transient solution. You would have to do a direct (full) transient solution.