Modern classical music and jazz heavily rely on chromaticisms (notes outside of a piece's main key and scale). Playing a scale different to the piece's/section's main scale on top of a chord played by the rhythm section is one of the ways to add chromaticisms to a musical composition. These days, I mostly play "old-school" jazz, and I rarely see more than 2 chromaticisms per bar. When there's more, it's usually a chromatic progression from one note to another or a trill.
When composing or playing music, I mostly employ this technique for solo phrases, to add more "spice" to a phrase or to emphasise a transition from one chord to another (similar to drum breaks, which usually signify a change of chords or moving from one section of a piece to another).
The scales you employ and how often you introduce chromaticisms depends on the style that you play and your instrument and its role within the ensemble. In most musical genres, the rhythm section is not expected to move outside of the main key and scale for the piece/section. For example, the bass is expected to outline the tonic and the dominant of the chord being played. Piano in some styles of jazz is expected to be very "inventive" and play maj7 chords and their related scales/modes.
Free jazz and some styles of modern classical music depend on chromaticisms, and this is where I would personally expect every instrument (sometimes even the rhythm section) to play unrelated scales over a chord, or change key and/or signature every bar/phrase, or some unconventional logic to be used for musical progression. Probably, one of the most famous jazz composition progressions is "Coltrane Changes".
Usually, when it comes to jamming, you are either given a note sheet that outlines the chords per bar or the other musicians tell you what conventional progression you are going to use (12-bar blues being one), what key the rhythm section will be playing in, and what scale they are going to use. If you practise your scales every day, it should not be too difficult to take a look at the note sheet and find out what chromaticisms you can add to any given bar. If you have your personal printout, you can even add them to it in advance. I do not have perfect pitch or an eidetic memory, so that's what I normally do.
I started out as a percussionist (drums, kettle drums, xylophone) in a folk orchestra for kids and teenagers, playing in some metal, grindcore and mathcore bands and composing videogame music on the side, so I understand the difficulties you are facing. Jazz can become very complex, but most of the time it's important to know the chords per bar and the scales commonly utilised in jazz (harmonic minor, be-bop major (ascending), be-bop dominant (descending), blues hexatonic major and blues hexatonic minor). You might even add whole-half/half-whole diminished scales to your phrases, to add more of a "metal" feel to them.
To give you a more concrete example, the piece might be in, say, A blues hexatonic minor, and the rhythm section plays a min7 chord in the first bar (A-C-E-G). You decide to impress the audience and play outside of the piece's main scale. Your pick that day is the half-whole diminished scale in A. You quickly remember that its notes are A-B♭-C-C#-D#-E-F#-G. As you know, the half-whole diminished scale has the min7 chord as well. But you can add a few "blue" notes (chromaticisms) to the rhythm section's chord and play A-C#-D#, for example. That way, all of you will start on the same tonic (A). Your sound will be a bit "dissonant", but the listeners will still perceive it as "viable", because it is still within the main key of the piece and the root of the chord (A) and within the scale you have picked for your little "solo" (A half-whole diminished).
I would also like to add that I prefer to be consistent with scales that I "go away" to within a musical piece and not overuse this technique. If you jump between scales all the time, the average listener won't be able to follow. But if the rhythm section plays one scale, and you only ever change to another one, there will still be some audial consistency to what you're playing. Naturally, this rule does not really apply to free jazz or neo-classical music that is fully or almost fully chromatic in nature.
Hopefully, this makes sense.