f stop is a ratio of aperture diameter and focal length. All other things equal, a 200mm lens at f/2.8 needs to be about 4x larger than a 50mm lens at f/2.8.
Complex lenses like zooms, heavily corrected lenses, ultra wide angle rectilinear lenses, and retrofocus (a design used for focal lengths wider than the distance between the sensor and the mount, i.e. roughly <50mm) also have more glass to accomplish some of the weird things they do and to compensate for the amount that bounces away in the midst of 15-20 different complex elements.
And yes, as the OP suggests, full frame lenses with equal specifications to a crop frame lens will also be larger, to cover a larger sensor in back. That's definitely one of the main contributors.
From the small amount of experience I have designing some of my own lenses and mucking around for many hours in simulator programs, the above three factors all seem to have a similar influence on lens size.