Back-button focus is
supposed to work as you describe. It's only active while the button is held down; it does not start an AF mode "on."
It does auto-focus, but only while the button is pressed.
That makes it easy to switch between AF-S and AF-C modes when shooting stills. Hit it and let go, or hold it down. You don't have to make a menu or settings change.
The camera's behavior of ignoring focus until you press the button is
not manually focusing; that's triggering the auto-focus at a specific time.
Manual focus is turning the thing on the lens and judging for yourself through the viewfinder.
All of that said, I don't see the point of back-button focus for video. I'm not sure the manual even describes the button's behavior in Live View or video.
Consider also that the AF operation is through a completely different system for video than for stills. the super-duper ultra-hot-shot 51-point AF system is locked out of view when the mirror is up for Live View, and unable to contribute to video in any way. The AF system during video is contrast sensitive to what it sees on the sensor, a much slower system, and more prone to "guessing" which way to move the focus, which results in focus hunting, none of which you'll see in still photography.