yes, FF is noticeably higher IQ and ISO performance usually. But that's nothing compared to what a large format 4x5 sensor would be like.
ISO performance, for example, is almost directly related to pixel size/area (for the same generation of sensor technology). When you double the area of light gathering, you can go to about twice as high of an ISO with the same acceptable noise level threshold.
A 4x5 sensor would be over 13 times larger than a full frame sensor, but could still fit theoretically in a camera of a similar size to mid-upper range modern DSLRs.
So if a modern FF camera can go up to, let's say, 3200 ISO with reasonable noise levels, a 4x5 sensor could go up to
43,000 ISO with reasonable noise levels, 3 2/3 stops higher than full frame.
(And almost 6 stops better than micro 4/3 of the same generation technology).
Assuming manufacturing gets to the point of being able to provide affordable 4x5 large format sensors, would YOU give up 6 stops of ISO performance just to have your camera fit in your pocket?
For reference, this is a difference of six stops that I took just now:

Similar improvements would be seen in dynamic range and in resolution (if the pixel count was high, and the future camera was pooling across pixels dynamically/intelligently in high ISO scenarios, etc. etc.)
I find it extremely unlikely that smaller sensors can possibly win out in light of such possibilities. The logical "sweet spot" of sensor size, in a world where any sensor size is cheap to make, would be the size that is as large as possible while still being portable in a reasonably hand held device. Which is probably about 4x5 or slightly smaller.