The reason why a Bb is used instead of a B has to do with the key aspects of the Lydian mode itself which is the #4 and how it acts.
The only difference between the Ionian mode and the Lydian is the 4th which is perfect in the Ionian mode and augmented in the Lydian mode. In Fux's counterpoint whatever mode you were in, you would want your cantus firmus and its counterpoint to reflect that. That's why you'll see a D# in E Phrygian, C# in D Dorian, ect. They are not natural to the mode, but emphasise and lead to there respected home note. While the E in F Lydian does this nicely, because of the way the scale is laid out the B (the augmented 4th) can also lead to the C very easily and makes it seem like the home note.
To counter this, the B is typically avoided and when introduced it is flattened to avoid this.There are a few sources that talk about this in depth and are worth a read. Here an excerpt from two that explain it pretty well:
Counterpoint in Composition: The Study of Voice Leading
In writing a modal cantus firmi, the students should bear the following in mind:
- In relation to voice leadin the Ionian and Aeolian modes (on C and A) can be considered equivalent to major and minor.
- The distinction between authentic and plagal modes becomes problematic in polyphonic music; the student, therefore, can disregard this distinction.
- In polyphonic textures the Lydian mode (on F) regularly employs B flat; this mode, therefore, becomes equivalent to transposed Ionian.
Source
Music from the Middle Ages Through the Twentieth Century
Despite the fact the b flat in lydian mode gives the appearance of the modern "major" mode which was an anomaly in medieval theory, one must take into account the evolution of modal theory that, in Adam's day, readily accepted b flats in lydian mode. Moreover, the use of b flats in refrains "He! Robechon" and "Vous l'orres" may be justified on the basis of the understood principle of fa supra la or modal commixtures.
Source