Your question is hard to answer without looking at the geometric data but I'll throw out some suggestions to try (disclaimer - I've never had that particular error message while analyizing a dam breach in RAS). First of all is your dam set up as an in-line structure or as a lateral structure? For breaching purposes the dam should be set up as an in-line structure. Second, are you describing your reservoir through x-sections or as a storage area? If described as a storage area is the storage area properly connected to the river reach? In the storage area option you need at least 2 x-sections of the river reach inside the storage area and then comes your dam. Third, have you checked your stationing data for the inline structure to make sure it matches your x-sections stationing? Lastly, run your model first as a steady flow model without breaching the dam. Just run some flow like the 10 year or 100 year storm through the dam and down the reach. It is a lot easier to catch errors in Steady Flow analysis. You may still have errors when you go to Unsteady Flow, but alot of errors can be eliminated via Steady Flow check.