Yes it takes practice.
It also takes positioning your body correctly to use your body to pan smoothly.
Stand with your feet a bit wider than shoulder width apart.
If you want to pan right to left you want your left foot forward of your right foot, your right foot toe out at about a 45° angle to your straight ahead left foot, and you want a majority of your weight on your left foot. In other words you lean into the shot.
You move right to left at the waist, so make sure your feet are placed so you don't wind up all scrunched up on your left side at the end of the pan.
For a left to right pan put the right foot forward, the left foot back, with a majority of your weight on the right foot.