"If there are studies out there about frequency sensitivity more rigorous than me making noises in my bedroom, I'd like to see them."
Well, there are literally hundreds of studies in psychoacoustics on the perception of musical pitch. You may want to do some basic searches on topics like psychoacoustics, pitch perception and cognition, etc.
To try to answer your question(s), there are a few somewhat unrelated threads. The first is the question of how wide a range of frequencies a single hair cell on the basilar membrane can sense. And the answer (as with almost all questions in psychoacoustics) is "it depends." At the very low and very high ranges of hearing, neurons may fire over a wider range of frequencies.
But let's assume we choose a frequency in the sensitive middle region of hearing. At very low sound intensity (less than 10-20 dB), a particular auditory neuron may only fire at a rather specific frequency, known as the characteristic frequency of that nerve fiber. But raise the sound level to anything reasonable, and that frequency response range widens considerably. If you get to 40 dB, the response may be over half an octave in width. Raise the sound level to 90 dB, and the neuron may fire over a range of several octaves (though it will generally fire at a higher rate for frequencies near its characteristic frequency).
The point is that individual resolution of nerve fibers is a terrible indicator of humans' ability for pitch resolution. There's a huge amount of processing that goes on in the auditory cortex to convert all of those signals into the thing we think of as a "pitch."
So what do we actually know about your central question concerning pitch resolution? Well, again, it depends. The first thing you'd want to look at is so-called Just Noticeable Difference (JND) studies. This refers to a type of study where, for example, one pitch is played, and then another is played quickly afterward, and the subject is asked, "Is this the same pitch?" How small a change is perceptible again varies over the frequency range of hearing and the loudness of the stimulus. It also varies greatly on the nature of the stimulus -- a pure tone (sine wave) vs. a complex tone (with harmonics).
For humans overall, in the most sensitive central range of hearing, assuming only pure sine tones, the JND is usually about 0.5% of the frequency. In musical terms for this range, this turns out to be roughly 10 cents or 1/10 of a semitone. In lower and higher registers, the JND is wider. If you introduce complex tones with harmonics, the JND shrinks considerably. For trained musicians in the central range of music pitch with complex tones, their JND is likely only a couple cents. (That's how professional musicians are able to tune their instruments by comparing their instrument to a reference pitch played by another instrument without any electronic "tuner" telling them they're sharp or flat; they wouldn't be able to without pitch discrimination on that level.)
But that's only comparing two single tones in quick succession. When you start playing simultaneous tones, it's a whole different ballgame. There, with complex tones, beat perception begins to play a role for pitches that are very close. Further apart, and you encounter sensory roughness, a result of critical bands created by the low effective resolution of the nerve groupings in the cochlea.
If you start doing tasks that ask whether two simultaneous pitches are the same or how they are changing, people do a lot worse. With pure tones, the so-called "limit of frequency discrimination" is often closer to a semitone for non-musicians, and it may be as much as 30 times as large as the JND threshold. Complex tones make it easier to hear beating and roughness, so most real musical contexts will allow musicians to discriminate sounds more accurately. But in terms of basic perception of pure tones, things get really complicated with more than one pitch when we want to hear differences.
Although this answers your questions about the cognitive limits of frequency perception, actually none of this so far really gets at the kind of "experiment" you did. You were trying to see at what point a given musical interval sounds "off" or "different" categorically from what you heard before. That gets into musical context.
The problem with complex tones and trained musicians is that perception of musical intervals is fundamentally categorical. That is, through years of training, you've learned that sound X is a "perfect fifth" while sound Y is a "tritone" and sound Z is a "perfect fourth." Even if you didn't learn the exact names for these intervals, you played music that was roughly divided into 12 semitones per octave, and your brain was entrained to categorize sound in this fashion.
Non-musicians do not display such marked categorization in their perception of intervals composed of complex tones. But musicians will naturally gravitate toward hearing a "perfect fifth" vs. a "tritone," and only something far off in-between those intervals will be perceived as ambiguous or weird. It's a bit like color perception: small children don't naturally come with words dividing up the spectrum into bands of named colors. But through language usage, they come to associate certain frequency ranges with specific words. Musical pitch (aside from those who have "perfect pitch") is relative, but a similar process goes on among musicians as they learn to sing and play intervals in tune. They learn to ignore somewhat minor deviations (often up to about 20 cents either way from equal temperament) and still hear the "same interval." It may not sound as well "in-tune," but it will still sound like a "perfect fifth" or whatever. In real-world musical performance, we often accept even wider pitch variation when bent artistically according to musical norms.
What you perceived in your "experiment" is the result of this categorization process, which has little to do with anything going on in your cochlea. It's the result of much higher-order pitch processing in the auditory cortex and elsewhere in the brain.
Anyhow, musicians can train themselves to perceive smaller changes as significant. It's certainly possible to notice smaller pitch deviations (as noted in the JND thresholds). The question is whether they are perceived as musically relevant and thus something we deliberately "notice." For complex tones, I personally will definitely hear the difference between a 700-cent "perfect fifth" and a 680-cent one or even a 690-cent one. When played in isolation, the latter will sound noticeably out of tune to me. I've also spent a lot of time working with tuning of scales. Yet if you had such tones played smoothly in a certain musical context where it makes sense to bend the pitches like that, I likely won't notice a slightly smaller fifth at all. That's the categorization working. And if I notice the difference, it will still be "acceptable."
Ultimately, it's like apple pie. You may really like the taste of that certain type of homemade apple pie your grandmother made when you were a child, and you immediately recognize it as the "best" apple pie. But you may also like a lot of other kinds of apple pie as well. There's a wide range of apple pie types you might consider "acceptable" for eating. It doesn't mean you are incapable of noticing the difference between them, but they all are in the category of "acceptable apple pie." You found the threshold approximately where you hear an "acceptable perfect fifth." That doesn't mean all possibilities in that range are equally "in-tune" or that you couldn't tell the difference between some of them if they were played in isolation in a lab experiment. But from a musical categorization experiment, you found where your personal boundary for "perfect fifth" lies, likely based on years of listening to and perhaps playing/singing music where the size of an "acceptable perfect fifth" can vary significantly in performance.
