Going from warm to cold won't bother the camera. But going from cold to warm may just cause a lot of moisture to condense on the camera parts (including internal parts.)
For this reason, I put the caps on and put the camera in its bag and zip everything up to keep any warm moist air away from the camera until it warms up gradually.