...I-IV-V-I chord prog, in C mixolidian: C-F-Gm-C...
You're mixing up symbols and tonalities.
If C mixolydian, then the
V - the triad on the dominant/^5 scale degree is a minor chord, so lower case
C mixo: I-IV-v--I
If we speak properly of a dominant chord then the chord must have a leading tone the ^7 scale degree one half-step below the tonic.
Mixolydian mode doesn't have a leading tone, therefore it doesn't have a dominant. Strictly speaking then you don't have a dominant to substitute.
Side note: to me modal generally implies music without a real dominant chord. Ionian mode of course it just the major mode so that more or less excludes Ionian from the general sense of "modal" harmony. Except for the Lydian mode all the other modes do not have leading tones and so they don't have dominant chords. From that perspective tritone substitution may be foreign to modal harmony.
Back to your chords.
If you change your harmony to use a real dominant you will be in C major, you would get...
I suppose you could have that
G7-C change and still have the music sound like it was Mixolydian, but it would depend on exactly how you handled
G7-C. It would need to sound like a temporary tonal shift. It would help if the melody somehow clearly sounded Mixolydian. Perhaps by avoiding the raised ^7 degree. You would have to handle the harmony somehow to make Mixolydian the clear tonality and the
bII7-I substitution seem temporary. You would probably need the minor
v in the progression to make Mixolydian clear. You would have to play with that Mixolydian/major and
Finally, the other way to look at this is to say your progression isn't really in C Mixolydian, but rather in F major. In that case you have to re-label your chords...
We have a proper dominant now, but it's in a different position in the order of the chords. The substitution might be like...
Back to part of your question...
...In F major the dominant note is the C and its tritone is Gb. So It
would be C-F-Gb7-C. It seems wrong to me...
You were correct. The only problem is you substituted the wrong chord! You substituted the
ii instead of the
So, make sure you identify a real dominant chord then make the tritone substitution.