But in most sport event or fast moving objects my shutter would be pretty fast so would IS really matter?
A little, but not a lot.
Yes, IS is the main difference between those two lenses...and yes, IS won't really help you for sports because it only helps prevent blur from camera shake, not subject movement. You need a fast shutter speed for that.
There is the issue of the focus motor, as mentioned above, and yes...having USM is a nice advantage. Although, there is a Canon version of the 75-300mm that does have USM, but does not have IS...(that may even be the one you looked at...at it should be around $200). However, note that there is USM and there is Ring USM. The Ring type USM is much better (quieter and faster)...but it's hard to really know which the lens has, unless you look up the specs on Canon's website.
Further to this, from what I've read, the version that has IS, also had a better lens design and produces better image quality. So besides better AF and IS, it's also a better lens overall. Of course, it's a lot more expensive.
But both lenses present a problem for you...the maximum aperture is only F4-5.6....which means that it would be hard to get decent shutter speeds in less than ideal light. This makes them a bad choice for sports.
Shooting sports is actually one of the most demanding (on your gear) types of photography. You want fast shutter speeds and you are also likely a fair distance from the action. So you'll want long lenses that have a large maximum aperture. Most 'pro' shooters are probably walking around with lenses like 400mm F2.8, 300mm F2.8 or at the very least, 70-200mm F2.8. Add a couple 'pro' level cameras that have very advanced auto focus and a very fast frame rate and they have enough gear to equal the price of a nice used car.
I haven't see that you mention what your budget is...so it's hard to offer a recommendation. The best 'bang for the buck' lenses for sports might be something like the Canon 85mm F1.8 (if that is long enough for your type of shooting). Otherwise, I might suggest a Sigma or Tamron 70-200mm F2.8. Still rather expensive lenses, but cheaper than the Canon option.