As I understand it, it's not the size of the sensor directly, but the short focal lengths they have to use. The f numbers are a ratio, not a direct measurement of size. f2.8 on a 50mm is an opening about 19mm in diameter. On a 9mm lens (one camera I looked at has a range of 5.8mm-17.4mm for its lens!), f2.8 is only 3mm. 3mm on a 50mm lens is f16! Cropping comes into play, which might enlarge the effective area in focus; but still, f16 is a huge difference from f2.8, even if you crop in quite a bit.
As an aside, f2.8 on a 300mm lens is a whopping 107mm across, which is why large aperture telephotos are so expensive. That's like having an f.45 on a 50mm lens. It would be more than twice as wide as it is long!