Important questions: what kind of serial interface do the modem and computer have - USB, RS232, RS422? What data rate are you expecting to run at?
If you have an RS232 interface between the computer and the modem it's not good for more than about 10 metres at a maximum data rate of 9600 bps, as - in theory - the line drivers in each device can't drive longer cables at high data rates without errors. If your speed is lower or you can find VERY low capacitance cable you may have better results.
If you have an RS422 interface you can easily drive several hundred metres of cable. The best cable type for this is, in order of preference, screened twisted pair, followed by twisted pair. Connect the screen to ground at one end only.