We had a sloped backyard in the house we owned and I figured it would be profitable to put a retaining wall and fill during the bubble years. The same project, though, may not make sense in the current market. Besides, we had to contend with a (possibly uncooperative) HOA.
I think you should start by asking what will you do with a flat backyard. If you are adding a room etc., it may be worthwhile. A garden may not pencil out. If you are planning construction, a good architect may design a structure that hugs the slope nicely and avoids the need to fill.