The problem in that particular part of Fux's The Study of Counterpoint is not that G doesn't belong to the mode at all, but that the student is putting the G below D in the first measure. The first measure of a species counterpoint exercise must make the modality completely clear, but a G on the bottom would cause the exercise to sound initially as if it was in G Mixolydian instead of D Dorian. This is a problem in this context because the exercise will clearly end in D, and species counterpoint exercises don't modulate.
The upshot of this is that, although the initial consonance can be a P5th when a student is writing an upper counterpoint to a cantus firmus, it is not an option when writing a lower counterpoint. That's why the teacher says it wasn't the student's fault, he just hadn't told him about the rule yet.