Here is my two cents:
juli32, I want you to look out your window on an object fairly close. You have two eyes (I assume) and as your eyes focus on an object, they no longer look straight ahead, but rather slightly cross-eyed.
Looking at an object further away, your eyes get less cross-eyed to give you depth perception again.
Looking at things even further away, your eyes start making smaller and smaller adjustments until you finally set your two eyes to watch straight ahead. Your eyes stop being cross-eyed entirely. Your brain cannot make so fine adjustments to your eye location and it doesn't need to: something so far away is unlikely to be a danger to you. To be exact, you don't have depth-perception for objects more than 40m away. No human does.
Well your camera doesn't get cross-eyed but it has a much larger front element. As you focus, you select light coming from certain angle to be in focus. As you focus to infinity, you receive light coming from very distant objects. Your lens receives them all at angles so similar that your lens can't tell the difference. This means that they are all in focus, regardless of their distance from you. This is focusing at "infinity".
I tried to explain it in layman's terms. But at the end of the day this is nothing more than the concept of limit, and as far as focal distances go, the limit is finite.