Nothing unethical at all as long as you don't, as GeoEnvGuy pointed out, have a non-compete in a contract with anyone.
I've been going through a somewhat similar transition. I was working for a firm but got an offer in a connected but different industry. I was able to leave on excellent terms and, as soon as restrictions lift and it's safe to do so, we're all still planning to go get a beer together. Well several months after I left, I started getting some calls from people asking if I wanted to do some work on the side. I decided to go all in with it and get the appropriate licenses, insurance, etc. It's still a part time deal (my current employer doesn't offer services to the public, so I'm not competing with my employer or violating any no moonlighting policies), but it's technically in direct competition with my old employer.
I went ahead and contacted the leadership of my old firm and let them know that I was doing it. I'm making a point to not market directly to their clients, but that's a matter of personal choice rather than ethics - and I'm also not turning them away if they call. Before you make the plunge, you may want to reach back out to those old clients and remind them of the conversations you had and see if they still feel the same way. Maybe they've settled in with your replacement and don't want to change. Or maybe they found another firm and they don't want to go through another switch so quickly. Or maybe they're sitting around every evening drowning their sorrow over you moving away. (We'll hope for some version of the last one!)
I have the luxury of folding up shop and just working my regular job if I so choose, so I can afford to maintain that bridge better than you may be able to. I'm also not in a position to really compete with them on the big money makers, anyway. That's an important thing to consider. What size projects are you taking on? There comes a point where it's just not feasible to do the design, calculations, drafting, etc. within the time frame of many projects if you're by yourself. So for now I'm one of the detestable "bottom feeders" that picks up the minor inspections, repairs, the odd addition, etc. One day I'll grow out of it, but for now I'm not capable of operating at the same level and won't try because I'll end up doing a disservice to the client and the engineering community. That may be something to think about. After all, ours is an industry of relationships. A firm's true value is in the relationships it is able to make and maintain, which means the value is in the people. If they can't keep the people who hold the value, then that's their fault (usually). You have value and you owe it to yourself to use that value for the benefit of you and your family. If your old firm can't respect that, then are they really the sort of people you want to maintain close ties with?