Most brass band players I know can play more than one instrument: after all, this is the main point of the "instruments in different keys" system for three-valved instruments. That said, when a player ends up playing different instruments in different bands (so they have to switch over the course of each week), they usually find that a little uncomfortable and try to resolve the situation by switching seats in one band.
There are two big differences between instruments. One is the amount of air. It takes a lot more air to fill a tuba than a cornet, but I think this shouldn't be a problem going from large to small. Sometimes players of cornets have trouble going the other way, because they're used to stretching the air out for long, twiddly passages, rather than pushing lots of air out for one long note.
The other main difference is in embouchure. Different brass instruments require completely different embouchure to each other. This, I think, is the main difficulty in switching. It's not that one instrument makes your technique worse at the other, or makes you forget how to play: more that, if you've been practising with one embouchure all week, and you sit down to play, you have to remind your lips to do the other embouchure, and it can make them tire quickly.
Thinking of it like changing car from a manual (stick) to an automatic is probably a helpful analogy. Driving an automatic doesn't make you forget how to drive a manual, but you have to get back into the habit of working the gears and giving it more or less gas in different situations.The gears are the embouchure and the gas is your breathing! A one-off change is easy to get used to, but if you keep switching, it's easier to get confused and stall at a junction.
One more thing to bear in mind is that the style of music is quite different. Cornets and trumpets often have very "nimble" music: fiddly, with lots of short notes, double-tonguing, and interesting rhythms. Tubas tend to have boring music, with lots of long notes. (Maybe this is why you want to try something new!) Unless you already play other instruments too, you may find that learning to read faster and change fingerings more quickly is a bigger hurdle for you than the change in instrument.
In summary, I don't think you should be afraid that learning a new instrument will hurt the old one. In the long term, the new perspective, and opportunities to play different music, will probably help your playing overall. Just be prepared for your tuba playing to feel harder until you get used to switching back and forth, and be ready to do just as much "lip practice" on trumpet or cornet as if it were your first instrument.