Designer, you are correct. That is how a focal plane curtain would works.
However, to answer the OPs question,
it is called shutter speed by accepted photographic convention, or as my mother would say, "Because we have always called it shutter speed".
Returning to my previous post. Looking at both my 1910 view camera lens and my 1970's Twin Lens Reflex, the leaf shutter is directly in front of the aperture iris.
So when I press the button the shutter leafs start to open and allow light to strike the film, once the shutter leafs have opened to the size of the aperture iris, the film receives full illumination until the shutter leaves close.
Therefore an f32 aperture will reach full illumination ever so slightly ahead of an f1.8 aperture, as the shutter leafs open to the larger aperture. So to the smaller aperture will remain fully illuminated just slightly longer as the shutter leafs moves from f1.8 to f32 before closing. Hence the ever so small change in shutter duration.
Indeed while restoring my mom's 1940's vintage Agfa-Ansco Sure-Shot. Even it has two aperture choices behind a shutter plate, with a hole passing over them at a fix speed.
Of course all of this is simply an academic exercise, because once you change aperture size, you also change the shutter speed, so any minute shutter travel time delay is a moot point. It is a great way to spin your wheels, to use anther idiom.
Perhaps the finite shutter speed might explain why my 1910 view camera lens goes to f45 but only 1/100 sec. while my TWL leaf shutter goes to 1/500 sec and my similar vintage focal plane shutter can do 1/1000.