First what software are you using? It sounds like you may have general purpose finite element software, not topopology optimization software like OptiStruct.
There are two approaches. First is a general design optimization where a parameterized model is run through a DOE-type optimization that alters values of the critical parameters. You are starting with a shape and changing proportions. The other approach, which is I think what you want, is to start with a blob of material that represents the part's maximum envelope, then whittling away until you get an optimized shape. This is usually referred to as topology optimization.
Back in the middle '90s I experimented with the element birth and death feature in Ansys. What I did was create a block of elements, apply loads and boundary conditions, run a structural analysis, then go into post and get the strain energy of the elements. I then killed any elements with strain energy below a certain value, and reran the analysis. I iterated this process until a critical level was reached, typically a stress level was exceeded or a certain percentage of the elements were dead. This was simple to do in Ansys with the APDL macro language. I used a mapped mesh of brick or quad plane elements, with no tet or tri elements. This produced a general shape determined by the load path. From there I made a parameterized solid model of this shape, and analyzed it with the Ansys design optimization module. This procedure worked ok, but sometimes produced shapes that were not easy to make with the intended process. Shortly after I left that employer, Ansys added a topology optimization module to Ansys. I've never used it, but I believe it works in a similar manner.
Ansys implements element birth and death by reducing the element's stiffness, I think by 1E6. The element is still there, but just so flexible it contributes virtually nothing to the structural stiffness. At the time, Ansys would not allow you to change material properties after the first solve, but that now has changed. An approach that should work better is to gradually reduce stiffness in increments. Something I worked on was element resurrection, where certain dead elements were reborn. I did this to reduce the effects of checker-boarding. Gradually changing element stiffness would I thing reduce or eliminate checker-boarding.
Alas, things are changing at Ansys. As of R14.5.7, design optimization is complete gone from MAPDL (aka Ansys Multiphysics, Classic, black screen), and topology optimization has been undocumented. Ansys seems intent on removing the Multi from Multiphysics (Monophysics?!!) so they can sell you an add-on that do the same thing. You can always program this with APDL, but there has been talk for years that the marketing buffoons want to take that away too.
Rick Fischer
Principal Engineer
Argonne National Laboratory