I made the switch about 4 years ago, and now design both bridges and buildings on a regualar basis. The AASHTO code is a beast, and depending on whether your state DOT does LRFD or not, it can even be more confusing. Most DOT's have design guides/bridge design manuals that assist in the process, but again, the analysis is sometimes tricky. There are some good design books out there, just make sure you buy the one that will suit your particular DOT.
What I dont like about bridge design is the seemingly slow process of DOT review and approval, at least in my state. I also dont like dealing with Right of Way issues, that always come up with bridge design. If you can stay inside a box and just design the bridge, and have others worry about the other stuff, it would be great, but working this way is not practical. Also, you have to get familiar with issues such as line of sight, roadway safety, and the geometrics of roadway layout to be effective. None of which is difficult, however, they are important concepts to grasp, which can affect your bridge design widths. Also, bridge roadway surfaces normally aren't flat, and are usually built to a vertical curve profile which much be determined so that screed elevation can be given to the contractor which account for the DL deflection. Again not too difficult, you just need to understand what is going on. You may also find yourself doing quantity takeoffs for all of your bridge designs, which is normally not done for buildings, as most DOT's require quantities on the plans.
Fees are usually better than dealing with Architects, so that is a big plus.