You can do a head to head comparison at
www.dpreview.com
Basically, the image quality is the same. I think the 20D's sensor is just a little bit bigger with a few more pixels. 8.2 vs 8.0 So if image quality is a top priority, then you might be better off getting a 350D and putting the extra money toward a good lens or two.
The 20D is built a little more sturdy but I've never heard that the 350D is delicate. Still, the 20D is made with more metal and the 350D has lots of plastic.
I got the 20D over the 350D, mostly because of the feel and the layout of the controls. The 350D is quite small, although there is an optional battery grip that will make it easier to hold. The 20D has more buttons and controls, which allow for quick and easy control of the camera. The 350D is more menu based. The control wheel on the back of the 20D is wonderful.
Also, did you know that the 20D has recently been discontinued? The 30D is the new model. It will cost more than a 20D...but it's still cheaper than the 20D was, when it first came out. The 30D is basically the same as a 20D but with some improvements. A bigger LCD screen, spot metering etc.
The best thing to do, is to go into a camera store and hold them both. A lot of people just don't like the small light feel of the 350D...and that's why they go with the 20D. If you feel comfortable with the feel of the 350D, then you might be better off saving the money for lenses.