There are a number of problems - and for the sake of completeness, I'l be going over some of the same ground as above:
1) Durability - Winter tires are designed to operate in cold weather. This means that when the ambient temperature goes up, the tire operating temperature does, too. Heat is the tire killer. So there is an increased risk of a heat related tire failure.
2) Tread wear - The tread compounds in winter tires are designed to be softer than normal tread compounds - and that means they will be soft and not wear well when the ambient temperature goes up.
3) Traction - While the tread compound in a winter tire is soft, there is a temperaure where the traction properties peak and at temperatures higher than that, they degrade significantly.
4) Handling - Winter tires don't not have as much built in sidewall stiffness as regular tires - and the larger the difference in speed rating, the worse this is.
5) Wet traction / hydroplaning resistance: If we confine our discussion to hydroplaning resistance, then we can talk about circumferential grooves and their affect on wet traction. Circumferential grooves are not needed for snow traction - may be for "slush" traction, but if slush is your problem, you're probably running all season tires and this discussion becomes a mute point.
However if we stay away from hydroplaning resistance (meaning slow speeds and /or fairly thin films of water), then winter tires will be good up to a certain temperature and this falls off fairly rapidly (this is related to the tread compound).
Overall conclusion: There are a number of reasons not to use winter tires in conditions other than winter.