Replacement batteries for such phones are sometimes available at your local 'Dollar Store', for, ah, $1 (each).
Then again, a complete new set of FOUR cordless DECT 6.0 phones might be around $50 at your local Big Box store (ahem, Costco, ahem).
If you do decide to get new ones, just make sure they're the latest DECT and stay away from 900 MHz or 2.4 GHz or anything not DECT. You can wiki DECT to learn more.