Practice, practice, practice - but of course you need to start somewhere...
Some books I've found helpful are Site Engineering for Landscape Architects by Strom and Nathan, Land Development Handbook by Sidney Dewberry of Dewberry & Davis (very comprehensive reference to land/site development), and Grade Easy by the ASLA (American Society of Landscape Architects). For whatever reason, civil engineering students don't seem to be formally taught grading design in school (at least the civil engineering programs I'm familiar with) but landscape architects have classwork in it, so you're more likely to find a landscape architecture textbook on it. I've found all these books in nearby university libraries but I'm not sure if you'd be so lucky. Strom and Nathan, and the Dewberry book, are in print.
If you can track down any reference or review material for the L.A.R.E. (Landscape Architecture Registration Exam), they have some good material, practice problems and worked examples on site grading.
There's a recent similar thread in the "civil engineering other topics" section, and several posters recommended getting your hands on example site development plans from your area and studying them. If you're in a design firm, you probably already have access to tons of previous plans.
A cool thing about site grading is that real world examples are free for anyone to look at, all around us. A trip to the regional mall, strip mall or big box store is an opportunity to see examples of site grading for commercial development. A walk around your neighborhood may offer good examples of residential grading. There are some fairly hilly areas around where I live and work, and one thing I like to look at is how grading transitions are handled from lot to lot, development to development, lot to street, etc. - for example, retaining walls or more gradual slopes.
Get familiar with what different slopes and elevation changes actually look and feel like. For example, if you can get your hands on a plan for a detention basin in your area with the typical 3:1 or 4:1 side slopes, walk the site and see what those side slopes actually feel like and what their visual impact is.
I remember parking in a small parking lot in my city and noting how steep the lot was, to the extent that an open car door tended to swing itself shut if the car was parked parallel to the slope. It made an impression on me, that that lot was about as steep as you'd ever want to make a parking lot, if not a little too steep. Later on I was able to go to the county online GIS website, check out the 1 foot contours of the parking lot, and estimate that the pavement slope in the direction of steepest descent was about 10-12% (if I remember correctly).
Those are some of my initial thoughts.
