You do need to change it. If you take a picture and the exposure is spot on, then take another picture on top of that then it will all be over exposed won't it. What you need to do is decide before hand how many exposures you want to take, then reduce the exposure by that amount. So if you're taking two photos reduce the exposure by half. If you're taking three shots reduce each exposure by two thirds.
Because your camera won't know how many exposures you want to take you'll have to do that by hand I'd imagine. All the feature does is stop the film from winding onto the next frame. So make sure it isn't selected before you take your last shot. That way the film will wind on straight away after you take it.
Obviously check your manual for specific instructions for your camera.