With ss rivets and the aluminum skin, you have a large anode area and a small cathode area. This is the "least worse" situation. The adhesive won't aggravate the situation, since I assume it will be mainy on the inside of the aluminum sheet, which would be the dry side. What environment will this vehicle be exposed to and how long do you want it to last? If you really want to be safe, you could put a sealant on the rivets or in the rivet hole before you set them, which would help isolate the rivets from the aluminum.