The simple answer is that you do all of that and more. I guess you don't have to but the chances of being a good improviser without working out your ears and fingers will be greatly diminished.
First, understand that there's not exactly "true improvisation" because everything you might play is based upon things you've heard or learned before. A common refrain in teaching improv is to "play what you hear" and that's good advice. But "what you hear" doesn't come from some magical creative vacuum. It's based on things you've already learned, music you've listened to and the idiomatic patterns in it, and the underlying harmony.
So when you practice things like "box patterns"—which btw don't have to be single position boxes, there are linear/horizontal/diagonal patterns as well—or intervals what you are doing is getting those patterns and sounds drilled into your brain. And most importantly you are connecting the sound and theory of all that to your muscle memory. This improves both "what you hear"—your capacity for imagining appropriate melodies—as well as your ability to quickly get your fingers to play those melodies.
As for what you should practice, the answer is to practice as much as you can with as much variety as possible. Variety is key to having flexibility in how you move around and think about the fretboard. Here's a non-exhaustive list of ideas off the top of my head in no particular order:
(This doesn't cover everything you'd want to study or practice for improvisation, but just the fretboard work. You'd also want to work in some theory work, ear training, and a lot of learning, listening, and transcribing of tunes)
- Sing/say the note name or interval or scale degree or whatever you're trying to learn while you play it.
- The chromatic scale. Yes that's all the notes which doesn't sound helpful but practicing playing up and down the neck using as many different "pathways" as you can is actually pretty helpful for breaking out of position-based boxes as you mentioned.
- Scales and associated modes (major, melodic minor, harmonic minor, pentatonic, diminished scales, whole-tone, etc, etc)
- Triad arpeggios including all inversions
- 7th chord arpeggios including all inversions
- Intervals. Not just 2 notes. Try playing a particular interval all the way up and down the neck as well.
- Chord voicings (triads, 7ths, drop voicings, etc, etc). Go beyond just using these shapes for comping and try to relate them to your melodic work. Play the chord and then work on the melodic structures contained and surrounding them.
- Idiomatic patterns - licks and patterns popular in a given genre, transcribed solos, techniques like enclosures, sequencing, or whatever.
The take those patterns and more and find ways to mix it up.
- Play both ascending and descending
- Mix up the order with inversions or starting from the middle of a pattern rather always using the root.
- Mix 2 or more scales or patterns together. For instance play 1 bar of one scale and switch to the nearest note of another scale on the downbeat of bar 2.
- Play in several directions. Rather than just playing "box patterns" try connecting the boxes linearly (horizontally/diagonally). And also try playing on a single string sometimes.
- Pick a different starting point and order for anything you can. Don't always start in key of C and then G, etc. Use the circle of 5ths but not always, mix it up. Don't always start low on the neck and move up to higher positions. Mix up every aspect you can think from day to day.
- When playing licks/patterns, learn them in as many positions and registers as possible. And play them in all keys. Ideally you should be able to play the lick from wherever you happen to be on the fretboard. This is not only good for the sake of recalling the lick, but it helps you learn the fretboard.
Lastly, try not to think of any fretboard work you do as a task to complete. It's not as if you'll suddenly be a great improviser once you've learned all the scales. Think of it more like an athlete working out in the gym each day. They're never "done" working out nor does working out replace what they do on the actual field. So do your fretboard workout a bit each day trying to push yourself and become more flexible and it will come out in your playing during the real work—playing over actual changes in an actual song.