I found an excerpt from a chapter written by Johan Sundberg in The Psychology of Music, a compilation of articles, presumably all related to psychoacoustics.
Here are some quotes and explanations:
Although F0 [the fundamental frequency of the note] varies regularly in such tones, the pitch we perceive is perfectly constant as long as the rate and extent of vibrato are kept within certain limits.
This means if a vibrato is too wide or too slow or too fast, it will not be perceived as vibrato of one note, but instead a fast glissando or glide back and forth. I doubt that human-generated vibrato is likely to ever be too fast, but electronic vibrato can easily be sped up to the point that it is actually frequency modulation that causes a dramatic change in timbre, as used in synthesizers.
Our conclusions are that the pitch of a vibrato tone is practically identical to the pitch of a vibrato-free tone with an F0 equal to the geometric mean of the F0 of the tone with vibrato.
That suggests that upwards vibrato from a fretted note should sound a bit sharp, and downwards vibrato from a bent note should sound a bit flat.
In your particular situation, you can't entirely trust your perception of pitch, because you are overly informed about how you're creating the sounds and therefore are susceptible to confirmation bias. By that I mean, you expect to hear a certain pitch and that can create the auditory illusion of hearing exactly the pitch you expect to hear.
That said, unless you're playing in an ensemble with other instruments that are playing the same nominal pitches as you, naive listeners are unlikely to notice the difference.