Yes, you are thinking of the key in a way that's a bit rigid. This is perfectly understandable. When you see a song, you're trying to look at its notes and figure out what key it's in so you can understand it. It almost seems like those out-of-key notes would throw off your calculation and make it harder to figure out.
I would say even more important than key, you should understand the concept of tonal center. The tonal center is the tone that all other notes in a song will gravitate or fall back to. That's just because in most music, the writer is creating a series of notes and chords which tend to return to a main note, and a chord built from the main note. For instance, a song in the key of E has the tonal center of E. But the tonal center does not say what notes are right, or wrong, or good or bad. It just names the tone (a single note) which everything will keep gravitating back to.
Now, let's think of key not as a set of notes that are right or wrong, but as a set of notes, normally expressed as a scale, which are drawn from to write the song. That means most of the things I write to put in the song, most of the pitches and chords will by tend to come from this place. But they don't really have to. If I use other notes, it may or may not be pretty obvious to the listener that those other notes are unique and don't fit in the key. Because music has been based on the major scale for 500 years, the listener may just sort of "assume" the pitches are based on the major scale.
Over time, the concept of key has tended to revolve around a major scale this way, and any use of a chord or melody that didn't follow the major scale of the tonal center has tended to be described by its relationship that major scale. For instance, if a song made from E major scale notes always alters the 7th note of the major scale to be flat (in the key of E, this means D# would be lowered to D), this is called a b7, but you can see that this is implying that D# is the "normal" 7. We tend to use this language to understand special notes and chords even when they're used consistently. So even if the song never uses the tone D# and only uses the tone D, we will tend to say that the song is in E and it's got alot of b7. So from this I think maybe you can see why it's important to think about tonal center.
One thing that will help is learning about the minor keys, and the chords that are harmonized from them, and also learning about the different modes, and the chords that would be found in those modes. I found that when I began to understand harmony in this way, I was able to see things pretty differently because now everything falling outside of the key could be explained by saying "Oh, this comes from the minor mode of the tonal center" or "Oh this comes from the mixolydian mode of the tonal center." But that will take some practice and you may not be ready for that just yet.
As for Day Tripper, the roots of all of the chords are from the key of E, and much of the melody as far as I can recall is in the key of E. But many chords are made from pitches outside of the key of E.