There are a lot of other reasons things can go wrong:
1) If you changed any of your exposure values between shots, it can confuse the software and potentially make it "jump" suddenly when it shouldn't, etc. You should always shoot panoramas on manual mode, and meter one way for the whole panorama. It's easy to forget to do this and to have your camera set to spot or evaluative metering or something on aperture priority mode, and then every time you take one of the photos, it gets metered differently than the last one and you end up with a huge headache for the software.
2) If you moved your camera overly weirdly during the panorama. Did you do it handheld, or on a tripod? Photomerge can correct for a bit of deviation: a few degrees of tilt difference, a bit of perspective change, an inch or so of up/down sway, etc. But it can't correct very well for 10 degree tilts to the side, or if you took a step or two in between photos, or if you drew your arms in or out in between photos by much. Doing these sorts of things will change the relationship of objects to each other. Photoshop isn't magical, and it can't change 3-dimensional relationships that differ between images. It can only stretch and skew the images it gets. So if Aunt Mary is hidden behind Uncle Tom in photo #1, but not in photo #2, then it's impossible to stitch those together seamlessly.
Thus, a tripod is always best. And better yet, a tripod with specific panoramic brackets to center the correct part of your optical system over the pivot point.
Or if you do go handheld, you need good technique in how you pivot, to ensure that the optical axis of the camera comes as close as possible to always intersecting at a single pivot point, and minimizing angular tilts and imperfections as much as possible.
3) Which mode did you choose in photomerge? the first one that lets it distort and everything as it sees fit? Or one of the lower options? If you chose one of the more constrained options (not the first one), then it might not be able to come up with a solution, even if there were only miniscule errors in how you moved your camera. Definitely try using the first mode if that's not what you did already.