This is just from 2 years of class voice at the uni, so if there is anyone better qualified to answer, take their answer over mine.
The different warm ups do different things.
Most warmups stretch range, and you certianly don't want to belt at the end of those, because you're already at the top (or bottom) of your range, and belting at the edges of your range can hurt your vocal chords. In fact, at the highest pitches of your range, you should be soft enough the notes are almost "breathy," because pushing can cause nodules to form.
So, most warm ups are done "at a comfortable level." You should never push to note and/or volume that has you straining.
There are warm ups for dynamic range, and these should be done at the volume intended. Typically the involve staying on the same pitch in the middle of your range with a cresendo and decrescendo or to hum or buzz a simple melodic pattern (typically arpeggios).
"Lip trills" or "buzzing" are also meant to increase the breath support and thus the volume, but it doesn't need to be loud as the second set of vibrations created forces you to push air like you were belting without actually belting.