Leica is fairly well known for being of the highest quality. Yes, you are paying extra for the name, but I think that is typical with top of the line products.
I've always been of the opinion that Pentax, Nikon, and Canon are all really about the same, but you'll definately pay a little more if it says Canon or Nikon on it.
Leica is also very collectable, and that drives up the price. It is possible to find better deals on "unpopular" Leica models. It's also very possible to find the same optics (Zeiss or Schneider) on other great cameras forgotten by history, and therefore afordable.
The Leica camera that most people think of is their rangefinder camera. Rangefinders don't use through the lens viewfinders, so they don't have the mirror and prism of an SLR. This cuts down on size, movement, and noise. They also usually have leaf shutter shutters rather than focal plane shutters (like SLRs). Leaf shutters cause less camera movement so you may be able to hand hold the camera at slower shutter speeds. Leaf shutters also can sync with a flash at higher shutter speeds than a FP shutter.
KSmattfish