Dear Nate;
The people who make the rivets also make washers which fit between the flange on the rivet head and the plastic part to be secured so that you dont crack the plastic when you set the rivet. If the plastic is a good deal thicker than the diameter of the rivet, even this might not be necessary.
Rivet manufacturer will tell you what the grip length is, and the pullout strength, so long as you observe those specs, it should work fine with aluminum rivets.
I am assuming of course that the connector will not be used to secure a stampeding elephant or that the connector will be used on the surface of the planet mercury. This might modify the design parameters somewhat.
Regards Adrian