We have an E3 and E520 at my store, the E3 is OK, the 520 is a peice of junk.
A 50mm f/1.4 lens on film or full frame with similar framing is going to act completely different then the same 50mm composed accordingly on 4/3rds. Worse yet, to get the same angle of view as that 50mm, you need a 25mm wide angle.
The thing about olympus is that they tote about how they're lenses can be smaller and lighter while still offering the same angle of view. They're 35-100 f/2 is friggin enormous, about double the size of my 80-200, wont' work on full frame, is rediculously expensive, and at similar working distances, doesn't have anywhere near the seperation I get with my 80-200.
If your main schtick is portraits, I can't reccomend the olympus system. The good lenses are usually more expensive then nikon's or canon's, the viewfinders are like tunnel vision, the sensors are a quarter of the size of film, thus incredibly noisy, it's a lose-lose unless you need the fastest AF system in the world.
On photozone they have a preview of the E620, ISO 400 looks like 1600 on my D700, and 3200 looks worst then 25600. Even the D40 blows it away.
In short unless you plan on shooting at ISO 100 all the time, need the fastest AF system ever, or actually like the 4/3rds square (yuck), then by all means shoot olympus. otherwise, there are much better, less expensive alternatives. especially for portraiture. All the people who have come into my store who own olympus wish they got someting else, the main reason they get them is because the kits are so cheap.