No, I do it all the time (more often connecting a minidisk recorder headphone socket to the mic socket on the pc, but sometimes the other way round). Works fine.
Just remember to start with the volume turned down, then wind it up gently until you reach a sensible signal level.
A.