The recycling time of the flash is a result of how it's designed, the type & power level of the batteries and how much power it used for it's previous flash. The last point being the key here.
When you fire a 580EX at full power, it can take a couple seconds to recycle...but when you fire it at 1/64 (or something like that) it takes almost no time to recycle.
When it's on-camera, you are likely using E-TTL metering, and unless your subject is rather far away or you're using small apertures, then the flash is probably not working very hard....so you get quick recycle times.
So the question is, what power levels are you using when you fire it off-camera? (If you leave it in E-TTL mode, it will fire at full power.)
Flash power is just one part of the exposure equation, so you can require less power from the flash if you use a larger aperture, use a higher ISO or move the flash closer to the subject. So by requiring less power, you get faster recycle times and longer lasting batteries.