I'm no expert on lens construction and as far as I know, nobody in this forum is.
But as a general impression - lens construction is an extremely complicated matter. Theres a lot of possible lens errors, and its extremely hard to compensate for them all, and impossible to compensate for any of them perfectly. And you can do the same thing in many different ways, with different results. And of course there are always new materials and new refinement techniques developed which change the game.
For example Panasonic mentioned they've been unable to create a camera like the LX100 before, because it only became possible with new materials to make a lens this compact.
So in general, if the lens designer wants a larger maximum aperture, obviously they need to make the lens more complex, which makes it larger, have more elements, etc. A 50mm f2.8 lens will be a LOT more compact than a 50mm f1.0 lens.
On the other hand, if the lens designer wants to make the lens more compact, for example they can put all lens elements BEHIND the aperture, which logically will result in a very compact lens, but also makes lens construction harder.
As Solarflare said size vs speed is one of the main trade offs. There are a host of other trade offs involved in lens design, if your willing to reduce resolution slightly, or accept slightly more distortion then it becomes easier to make the lens more compact. Pancake designs are more practical at moderate focal lengths, and AFAIK currently impossible for focal lengths above 100mm.
Despite the trade offs my pancake lenses give results good enough that I can't see any issues in the photos they produce. If they where both the full size equivalents I wouldn't have room for both in my bag, and would probably still have to loose a filter or two just to get one in .
Flatbed scanner image of my favorite modern pancake lens, the Nikkor 45mm f/2.8 "Ai-P Nikkor", which has a CPU. This is a modern Tessar lens design which was computer-optimized and released as a companion lens for the Nikon FM3A manual focus 35mm camera. The Tessar design uses four lens elements, in three groups. Very good wide-open at f/2.8, a bit better at f/4, and excellent at f/5.6, f/8, and f/11, with a very high degree of center/corner equality. This lens does not have the high drop-off at the edges of the frame the way many newer, faster-aperture lenses have. Due to the very few air-to-glass surfaces, this lens has very low flare or ghosting, and also has high contrast.