This is a matter of some debate!
Personally I'd say the following:
No, it isn't prohibitively expensive to start (unless you are on the poverty line!). Set yourself a budget - say £20 per month and stick to it. £20 will get you a few films developed and printed (and put onto CD for sharing with the internet).
Buy one cheap book on the principals of a manual SLR. Get an idea of how focussing, aperture and exposure work and how they affect an image. There's a good FAQ at the top which covers all of these. Children's photography books are often very good as you don't really need to know about complicated things, just vaguely how it works!
Buy a good value, good marque, fully manual 35mm SLR such as a Canon AE-1, Nikon FE, Pentax K1000 etc. Get it with one standard (50mm) lens. Check it for mould, rust and major damage, smoothness of operation - if it feels right it usually is. You should be able to get a great SLR like this for under £100 including the lens and some accessories like a bag for it. Don't get a zoom lens to start with as you've got enough new things to worry about. Also, don't buy an automatic camera as it won't teach you to think as well. If you've got time on your hands, look at car boot sales - don't pay more than £10 for anything though.
Take pictures, recording the exposure, aperture and conditions (v. tedious). Compare these criteria to the quality of images you get back.
Here's a blatant bit of opinion: Don't bother with courses - they are all rubbish. Spend the money on film development instead and you'll learn much more. If they are any good, then they'll probably cost you a fortune.
Don't buy expensive books, go to a charity / second-hand shop and get them there for £1. Buy some traditional magazines such as B&W photographer and look at the settings people have used to take great photos. Copy them, recreate them etc.
Good luck - if you need any help with second-hand cameras, just ask as there are lots of collectors and experts here.
Rob