I think the most important thing for many aspiring guitar students is to enjoy playing the instrument. If it becomes too complicated, it's easier to give up. I am pulling for you to make the 5th time taking up guitar - the one where you find so much joy in the process, that you won't be able to put down your instrument.
There is nothing inherently wrong with your idea of leaning heavily into one key and learning to play chord progressions and riffs in that key before moving to another common guitar key. Others have stressed that to become a true "musician" you will need to be well versed in more than one key.
But you did say "one key at a time" and I don't see that as a major mistake if you don't need to make a living as a professional guitarist. Realistically most guitarist rarely if ever play in certain keys because the open chords for certain keys are few and far between. You won't see many guitar arrangements written for the key of E flat for example. I don't have a clue how to play an E flat chord in open or first position (nor do I feel the need to ever learn to play any of the chords in the key of E flat).
While it is true that scale patterns are movable and the same pattern will work for every scale if you can play it at the right place on the neck, the idea that as a guitarist you MUST to learn every key is not useful in my opinion.
I like to teach beginning guitarist by focusing on one key at a time in the beginning and then get them playing songs they enjoy in that key so that they can discover the joy of making music without making it more complicated than it needs to be! Once they can learn the three major chords and a couple of minor chords in one key, with the help of a capo, they can play at least a basic version of most of their favorite songs, in a key they could sing them in. After they get comfortable with that key and develop the requisite passion by enjoying learning to play many songs, then we can move on to other keys and learn the chords for those keys.
I will agree that in more advanced guitar training it is helpful to learn scales in multiple keys simultaneously so you can see how a particular scale pattern can be used in any key. So yes - when you get to the point where you want to be able to improvise and accompany other musicians on the fly, regardless of what key they are playing in, you will want to focus on learning how the licks based on scales can be played in any key.
My go to key is G - because that key fits my voice, the chords are easy, and I have used it more than any other so I instinctively know where to find the notes in that key on my guitar. But having said that, I can front my own band - but I would never make it as a hired guitarist for someone else's band because they might want to play some songs in other keys and then I would face a learning curve. But that's fine with me because I don't aspire to be a guitarist in someone else's band. And if I hire guitarist to play in my band - I can say "I play this song in the key of G" and if they are a pro, they will say "no problem".
So if you just want to have fun and play songs for your own enjoyment and the entertainment of your friends (or front your own band so you can decide what key to play the songs in), you can do that by only learning the chord progressions and licks and scales in just a few of your favorite keys. And I see nothing wrong with starting with the key of E if you like that key. But it might not be the best path to becoming a professional session guitarist.
The most important thing is to keep it fun. If it ain't (isn't) fun, it ain't (isn't) worth doing! Here's hoping that for you - the fifth time is the charm!