The whole point of the mirror less systems is to be smaller and lighter but better than P&S. I second the notion of taking a look at the micro 43 system as Kundalini mentioned. I've had my system for a bit longer and it does fit my needs; with advantages and disadvantages. If you need more info, search my threads, the one Kundalini started, and/or just ask. Remember, its not just the size of the camera but the size of the lenses as well.
* NEX system has complaints against user interface but a bigger sensor. Bigger-sensor often means bigger lenses however. NEX native lenses are not that great.
* Digilux 3 is/was a rebranded DMC-L1 Panny of the 4/3 system (not micro 4/3). Lenses are expensive AND the system is not any smaller than your Nikon DSLRs (in fact bigger than D40). Its not a good fit for your original intention of smaller and lighter. It was not popular among the Leica crowd.
* Digilux 2 was popular and continues to be somewhat of a cult classic in the Leica crowd. It does fit the smaller and lighter but a bit larger than P&S and micro 43. It has sensor problems but Leica has (not sure if they continue to) to replace the sensor. The fixed zoom lens on it is very nice.
People I have run into with DSLRs that have been searching for a smaller second camera, I usually steer them to one of three solutions. The first being the least compromising down to the most:
* Smaller entry level DSLR of the same system.
* Mirrorless. Quality performance slotted above bridge cameras and right at entry level DSLRs. (Olympus Pens, Panasonic G*)
* High end point and shoot (Canon S95, G, Panasonic LX5, Olympus Zx-1, etc).
There's also a mix of options coming in that might work for you and not particular slotted in a category (niche market). Among those are the Fuji film X10, X100, Ricoh GXR etc...
There are some mirror less cameras that are approaching the size of the High end P&S so I'm more inclined to pass on the P&S now a days. Mirrorless has a significantly better feature set, interchangeable lenses (adaptable too), and larger sensor.
NOTE: Other than the S2, M9, and X1 (and upcoming mirror less in 2012), all other Leica digitals are rebranded Panasonics. As much as the Leica community insists there is a quality difference, I have not personally observed. My high end point and shoot is an LX3 not the rebranded Digilux 4.... happy with that too.