There is a difference in sharpness and build quality, but it all depends on what you need. I have two "L" lenses, the 24-70 2.8, and the 70-200 2.8. I use them because I take them where I want them to hold up. They are always in a backpack sledding, dirt biking, and on my motorcycles and hiking. They have to survive lots of vibration and me falling on them when I fall of.
I also have a 10-22 EF-S lens which is not the L build construction, but it's one of my favorite lenses and the quality and IQ is still quite impressive.
With anything in life, you have to consider your needs and wants. If you need it, save up, if you want it, and don't need it, then consider the alternatives.
For sports you will want F2.8 for sure.