To me, one of the defining characteristics of a mirrorless ILC--like the Panasonic Lumix DMC-GF3 or Olympus PEN E-P3, just to name two recent ones--aside from the interchangeable lens, is that it produces better photos than a point and shoot. As far as I can tell, people see the lens flexibility as a bonus, but for the extra couple hundred dollars they want better photos.
Which is why I'm confused by the trend toward even smaller sensor sizes in this space as evidenced by Pentax's Q with its 5.5x multiplier and the rumored forthcoming 2.7x-multiplier size from Nikon. (As an aside, I find the easiest way to internalize differences in sensor sizes is by the multipliers and assume that pixels shrink correspondingly. For a comprehensive visualization, check out this Sensor Size Cheat Sheet.)
Some people complain that even the Four Thirds sensor (2x), which uses the Micro Four Thirds mount in Panasonic and Olympus' ILCs, is too small.
In image quality terms, that means the Pentax will likely deliver image quality on par with other 1/2.3-inch sensor models like the Nikon Coolpix P300. The rumored ILC Nikon sensor will likely be about 74 percent the size of a Four Thirds, which is a new size. (Through some wonky and probably wrong math I'm guessing it's equivalent to a 1/1.1-inch sensor. Interestingly, there was a discussion in the dpreview forums a couple of years ago speculating on that size sensor.)
The drawback of the "larger" sensors, though, is that they require larger lens mounts with correspondingly larger lenses, so while the manufacturers can attain extremely compact bodies, when equipped with a zoom lens the cameras cease to to be pocketable. I think this is one of the biggest issues with Sony's NEX series, generally characterized by tiny cameras with huge heavy lenses.
So I ask you: How small is too small? How low would you go to obtain an ILC that, when equipped with a zoom lens, fits in your pocket? And as long as you're OK with a sensor that small, why not just get a compact megazoom with manual controls?