Of course each voice is different. But you are not looking at a random sample, but at a selection of singer with similar features, that is high voices.
Now, passaggio is more or less a posh term for "at this point you cannot continue with having lots of your vocal muscles vibrate (what people call 'chest voice'), so you need to reduce the amount of vocal muscle that vibrates (mixed voice, head voice)". And usually having a physiologically higher voice will also mean that your chest voice will go higher. Sure, people are different, but that is mostly why we have different voice types in the first place.
Surely within one voice type there might be some deviation, but we can still observe some general behavior. And maybe there are a few high tenors who’ll start getting into passaggio slightly more early or slightly later. You can see it as a sort of "rule of thumb".
About "what is supossed to happen on D4": You should start feeling less comfortable singing with your full chest voice around that point, and you’ll need to adjust your voice for higher notes. Note that at some point in training you want to have the control over your body to not switch between two or three "modes", but to continuously add and remove vocal muscle mass as you go up and down, effectively allowing you to get from one range into the other one without a break, because you can constantly adjust for the note you are currently going for.