While the guest answer makes some useful points about the edition, it doesn't really address the broader question (or Fux).
The basic answer to this is that the modern set of scales taught as "modes" are somewhat fictional, having little basis in historical use, particularly in polyphony. The 8-mode system of church modes was primarily employed as a classification system for monophonic chant, never intended for polyphony. Basically, the system was originally developed so people could know which chant formula to use to recite formulaic elements of the liturgy (like psalm tones) which would correspond to the other chants surrounding it in the liturgy (e.g., antiphons). That's really the main reason the 8-mode system was invented in the first place -- you hear a chant with certain cadences that are called mode X, and then you recite the psalm using this formula Y.
Polyphony developed pretty much without regard to any notion of mode as a foundational scale that a piece was "composed in." At most, one might be concerned about the mode of the chant tune that was often used as the basis of a piece of polyphony, but the other voices certainly didn't need to conform to any particular "modal scale." It was only in the mid-1500s that some composers and theorists tried to graft the old church mode names onto polyphonic pieces, and that was only done sort of haphazardly, and with due respect to the pre-existing practices of the 8-mode system.
You see, the 12-mode system was really unnecessary when it was introduced. The medieval gamut had one true accidental: B-natural and B-flat. That meant that a chant starting on D could have a B-natural (what we now think of as "Dorian") or it could use a B-flat (what we now think of as "Aeolian" or "natural minor"). But, in medieval theory, both of these would just be various uses of "Dorian." And some chants used both versions of B. Similarly, a chant with a final on F could have a B-natural or B-flat or both. All of it was called "Lydian." No dire need for Ionian and Aeolian at all.
If you spend a lot of time looking at Gregorian chants, you'll find a LOT of chants with a D final that have a lot of B-flats in them. And you'll find a LOT of chants with a F final that have a lot of B-flats in them. That was just fine and dandy. No one worried about it, until some systematizing efforts of theorists decided maybe they should add C and A as "legitimate" finals. (Note that there were also plenty of examples of medieval chants with C and A as finals too. But they were considered "transposed" versions of Lydian and Dorian respectively.)
All of this is background to say that "Dorian" mode as understood for both monophonic and polyphonic practice historically could very well include both versions of scale degree 6. And "Lydian" could include both versions of scale degree 4. That's how those modes worked, and those names still stuck around for such uses long after the introduction of the (somewhat unnecessary) 12-mode system with its Ionian and Aeolian.
Now, fast-forward to around 1700 and your particular pieces. At that time, as guest remarks, key signatures were not yet standardized. In fact, there was a trend for a while where the "Dorian" key signature seemed like it might become the default "minor" key signature, along with or even instead of the "Aeolian" that was ultimately settled on. Around 1700, one frequently sees these "Dorian" key signatures, even in pieces where an Aeolian/natural minor signature seems like it would make more sense.
And as for Fux, his treatise is ultimately a mish-mash of overlapping old modal practices thrust into a somewhat tonal context, and a hardy helping of stuff that he likely just made up (which had nothing to do with Palestrina or modes). It's really important not to read Fux as representing anything close to the actual practice of Palestrina. Fux was trying to teach people how to write something like 18th-century music while also paying homage to the "strict style" of "modal" music that wasn't really current practice anymore.
All of that said, his use of B-flat in a "Lydian" context is far from unusual, when compared to the use of B-flat in Lydian chants and in polyphonic works labeled Lydian in the rare 16th-century collections that actually labeled pieces in modes.
As to the last question -- if this were transposed into C, wouldn't it be "Ionian"? Sure. Except, as I said, "Ionian" was never really necessary as a mode to begin with. Look at real examples of polyphony that end on an F final from the 1400s and 1500s. Pretty much all of it has B-flats all over the place, even before anyone called anything "Ionian." (In fact, one might argue that one impetus for making "Ionian" into a mode in the first place was the fact that so much polyphony was already being written in that scale before anyone had a unique name for it.) I'll give Fux credit here in that his examples do reflect some realistic elements of historical practice, one of them being that F-based pieces frequently made use of more B-flats than B-naturals.