You are right. When it comes to measuring frequencies (which is at the core of listening), there is the uncertainty principle: The shorter you measure your signal (or the shorter the signal is) the less accurate it can be measured. This is a fundamental principle and it is not depending on whether one uses a fast fourier transformation or a mechanical device (ears) for measuring.
It is stated that the uncertainty of the frequency times the uncertainty of the time is roughly 1.
That means that a sine wave with 100 Hz can be estimated to within 1% accuracy if one hears it for 1 second. A sine wave with 1000 Hz can be estimated to within 1% accuracy in a tenth of a second.
There is a really great resource available online, unfortunately it is in German. It loosely translates in the introduction "The uncertainty principle": "Notes are written... as if pitch and duration could be created completely independent from each other. However, experienced composers know for a long time that the low notes of an organ or a
tuba have to persist for a certain time to be perceived as well-sounding. Sequences of such low notes are therefore only playable at limited speed." (from:
Karrenberg U. (2017) Das Unschärfe–Prinzip. In: Signale – Prozesse – Systeme. Springer Vieweg, Berlin, Heidelberg).
The German wikipedia on pitch also points out that from the uncertainty principle "it follows that in music practice intonation accuracy is much more important (because audible) for slow passages (long notes) than for fast passages (short notes). String players often claim - to the surprise of the layman- that it is by no means easier to play slow pieces."
Todd Wilcox' statement on the overtones is relevant here: Because of the presence of overtones, we can estimate pitch faster than if one would listen to pure sine waves, but still one needs more time to estimate the pitch of a low note.
Regarding Topo Morto's statement about the cilia in the ear detecting soundwaves directly without doing a fourier analysis: It takes time for the cilia to tune into resonance. If they are exposed to a pure sound wave for a long time, only the cilia with the right resonance frequency will vibrate. In the onset of the soundwave several cilia "around" the right frequency will start to vibrate, making it impossible to get the exact pitch of a short note.