The biggest and most dramatic difference I've noticed is when shooting into the light. Even a small amount of dust can cause a noticeable difference in image quality if the lens is otherwise in mint condition. As usayit has mentioned, the effect of large scratches and chips may be reduced greatly by blacking them out with wax, a sharpie or paint. It's the small, shallow scratches or "cleaning marks" all over the surface that cause the most reduction in contrast, but do not necessarily cause a loss of sharpness.
Best,
Helen