I don't have perfect pitch, but I liken perfect pitch to seeing pitches on a piano keyboard. And based on that assumption, if you can learn to name an interval immediately without counting keys by seeing it on a piano keyboard, it's possible to learn to do the same with perfect pitch. Myself, I can see in my mind what I hear on a piano keyboard and with a slightly worse hit/miss rate on a guitar fretboard, and I can transpose the vision to a different key at will. It is no problem to tell which interval I'm looking at on a keyboard, because all the intervals and how they look are very familiar to me. I've developed these skills entirely by practicing playing by ear, playing and accompanying songs as melody+chords, in different keys.
Each chord and each note in a chord has a role (function) and harmonic taste within its key. And each chord has a position within a key, with the chord's root note and third being good "known positions" to compare to. I suppose your perfect pitch doesn't in any way hinder your ability to feel e.g. a V - I dominant chord motion? I'm suggesting you to play (and/or sing) melodies and chord accompaniment (perhaps also arpeggiating the chords) in different keys. When you have a song where the melody goes from a I note to a III note, and you have to produce that jump in different keys, it should make you associate that jump i.e. interval with those pitches. Just like if you're playing it on a piano - when you've played that I - III jump in all keys enough times, I think you'll begin to see that interval visually without having to count keys. The same goes for all intervals. When you've played a 6th interval in all keys many times, you'll recognize that interval, particularly if you can relate the pitches to other known sounding pitches. Third and sixth intervals have a functional meaning, because you can see them as jumps between a major or minor triad's third and fifth, or some other triad's 1st and 3rd, e.g. C-E can be a part of C major or A minor triad. These things become more meaningful when you have to produce them, not just observe them.
So, to summarize the "prescription". You want to make the intervals - not just pitches - important and meaningful for you, and so you get into situations where you have to produce the intervals. The practice routine is: play or sing intervals and entire melodies and chords in different keys. When you have to produce the intervals, you get to know them, and the question of recognizing an interval becomes: in which harmonic and/or melodic situation would I have to make that jump. If you just listen to music, you get the pitches as given because of your perfect pitch, but now you reverse the chain of command. Your task is not to play or sing a specific pitch, the task is to play or sing intervals and arpeggios, like in instrument practice, except you're not given ready-made notation for the exercises.
This is just a theory I made up after thinking about your situation, and it hasn't been tested on anyone. It would be great to know if it helps you. Interestingly, that's the same exercise I've recommended for people who don't have perfect pitch, but would like to learn to recognize chords and intervals by ear. Produce, don't just observe.