Ok....
First, the sensor size is critical for another primary aspect. That being absolute resolution. You have more absolute area to record information. This is akin to a desktop to a server.
For those not able to relate to that, think of it this way: Look at the threads on your bed sheet. The finer the thread count the finer the cloth. The larger the cloth, the more information that can be recorded, PERIOD!
Secondly. In regards to the lens issue: in regards to that, why is it that the much desired Canon or Nikon lenses are so friggn large?
More than that, notice that many of those lenses are limited in zoom or are prime. Why?
To understand that, think of sucking soda through a straw. The larger diameter the straw the more you can suck through at one time.
Ok simple enough. So what about light?
In most modern lenses you have many multiple elements Though very high quality, that glass is not (just like the air) fully transparent. The more elements, means the more in the way of light refraction and blockage. Also you have more in the way of actual lens coating which is NOT transparent.
Third: Because of this aspect, here is where you have to put on the ol' thinking cap.
The concept is known as Modulation Transfer Function. or MTF. This little known aspect is what drives the lens size movement.
MTF - Luminous Landscape
You are correct that the center of the lens means sharper focus and a larger element means a large in focus area.
But it also means more actual light is coming in and reducing light drop off, vignetteing (as previously mentioned) because it increases actual light circle size, and because like the straw, your allowing in more actual light. This helps increase contrast.
There is another factor involving lines per millimeter and this take a while to understand. But its actual area that is the concern.
A Med. Format lens on an APS sensor will drive high quality light, but alas it also depends on the quality of the lens itself.
To finish this off, (and keep in mind I have played with micro, mini, small, med. large and VL Formats) I have a 1Ds and a 70D.
Putting them both to the test, the resolution of the 70D smokes the 1Ds in actual defined detail. (Here is the Mp count argument)
But the 1Ds smokes the 70D in actual area resolution. Any Med. format lens used on either or both will have diff. results.
It whether or not you want to play the game.