I can only imagine you are referring to the modes that are related to the major scale and the melodic minor scale, as those are the more common ones. There is a simple way to organize your thoughts regarding these. Seven of these modes can be generated from the major scale by simply starting on a different note and playing the sequence up an octave.
For example starting on the C major scale (C, D, E, F, G, A, B, C)
The D Dorian mode is (D, E, F, G, A, B, C, D)
The E Phrygian mode is (E, F, G, A, B, C, D, E)
The F Lydian mode is (F, G, A, B, C, D, E, F)
The G Mixolydian mode is (G, A, B, C, D, E, F, G)
The A Aeolian mode is (A, B, C, D, E, F, G, A)
The B Locrean mode is (B, C, D, E, F, G, A, B)
This pattern extends to ALL keys. The Dorian mode in the key of X starts on the second note of the X major scale. From this you can figure out the whole step and half step pattern if you want. Which one formula is "easier" to remember and apply is partly a matter of taste and experience.
The second set of modes are related to the Melodic minor and are commonly used in Jazz. Starting with the Jazz-Melodic minor (which descends the same way it ascends) we have the following note (using the Key of A minor/C major as an example) (A, B, C, D, E, F#, G#, A).
All you have to do to generate the other modes related to this scale is to start the sequence on a different note, e.g. (C, D, E, F#, G#, A, B, C) is one of the modes.
From this perspective it is pretty easy to recall all seven diatonic modes in any key. Just remember the sequence in order and map it to the degrees of the Major scale in order, and similarly for the other set of modes. The real question is how does this help with "improvise over them in any key". To improvise I think your time would be better spent developing a good ear for voice leading. From the point of view of classical music theory we generally tend to gravitate towards the use of arpeggios in developing melodies and those naturally fall on the chords that harmonize the melody. It could be thought of as a Chicken vs. Egg argument that no one will ever win but IMO the melody comes first and chords are added later to harmonize or support the natural movement of the melody. That doesn't mean one cannot start from chords and develop original melodies from chord tones but... chicken/egg.
Trying to pick a mode for a solo line takes a lot of mental energy. In Jazz it is often taught that we should pick a new mode that best matches each chord. This can be a challenge since there are dozens of chords in a single tune. But the chords follow well defined sequences within the key of the tune. For example the progression C -> E-7 -> A-7 -> D-7 -> G7 -> C never leaves the key of C major. It would not make a lot of sense to think Ionian -> Phrygian -> Aeolean -> Dorian -> Mixolydian -> Ionian as you solo. Better to realize you are in the Key of C, without any modulation, and stay on C major but use voice leading to generate melodic themes that gravitate to the chord tones. Simple diatonic sequences move linearly through the circle of 4ths and 5ths very nicely. In fact walking right up the scale matches chords tones in this sequence. The above progression can be made a little more interesting by inserting a modulation to the relative minor, i.e. C -> B-7(b5) -> E7 -> A-7 -> D-7 -> G7 ->C. Now the A-7 doesn't just sound like part of the sequence of chords leading back to C but as the ending point. This is due to the presence of the E7 before it. The B-7(b5) is strictly in the Key of C but its function is as a ii chord for the Key of A-. The grouping of chords (B-7(b5) -> E7 -> A-7) calls for an A melodic or harmonic minor scale all the way through.
In some cases committing the patterns to memory can be useful on an instrument. I have them in my muscle memory for guitar. But in reality, after decades of playing, these patterns are not very useful from a practical point of view. Learn how to analyze a chord progression to find the key of each section of a tune then stay on the primary scale of that key using voice leading to land on "good" notes as you go. This will take you much farther in improv that modes.
If you really want to get into interesting melodic ideas spend more time on the major and minor Blues scales, Be Bop scale, 1/2-whole and whole-1/2 scales, etc. The ones that DON'T fit into the diatonic sequence are more interesting.