No, it's not changing "key."
To really answer the question you would need to see the main melody of the song.
Typically the melody would be in a blues mode with a tonic of
E. That will not be diatonic, like the key of
E major. It's more chromatic. But that doesn't mean the tonic is changing during the song.
When you talk about "play this scale over that chord" it's some version of the "chord/scale system." That is more of a cheat sheet approach for what to improvise over a chord progression rather than a way to analyze the "key" of a song.
Another way to look at it with just the example you give is this. When you go through the progression...
E7 A7 B7 E7
...you start on
E7 and then return to
E is your tonic. Whatever harmonic diversions you take going from
E and returning to
E do not change the fact that
E is the tonic, the "home."
Actually, you want to embrace this idea of an
E tonic in an improv. You want to move through the
B to get to an ending on
E. If you treat
B like new keys, as new tonics, homes, points of focus (however you want to think of it) the melodic line will probably not flow well. You might end up with 4 disjointed bits instead of one connected line in