However, if the assumption above is correct, the octavi toni would end up in Ionian mode.
As you've probably deduced, the assumption is incorrect. In fact, this question is based on a couple of incorrect assumptions. The first is the identity of the first mode, which is Dorian, not Ionian. Ionian didn't even exist when the modes were initially numbered and named.
The second is the relationship of the numbering scheme to the tonic notes. There are two modes per tonic, not one. This is possible because each of the modern modes corresponds to an authentic and a plagal mode. The authentic and plagal modes come in pairs that have the same tonic, but they have different ranges and different reciting tones (or "dominants"). In particular, the authentic reciting tone is a fifth above the tonic (sometimes a sixth in Phrygian), while the plagal reciting tone is a third above (sometimes a fourth in Hypophrygian and Hypomixolydian).
The names of the plagal modes are derived from those of the corresponding authentic modes by adding the prefix "hypo-." The original eight modes, therefore, are
- Dorian
- Hypodorian
- Phrygian
- Hypophrygian
- Lydian
- Hypolydian
- Mixolydian
- Hypomixolydian
By the late renaissance, these modes were on their way to becoming the major and minor modes we know and love today, but theory was slow to catch up. In 1547, modes 9 through 12 were added, namely the Aeolian, Hypoaeolian, Ionian, and Hypoionian modes, respectively.
Not long thereafter, there were a couple of renumberings of the modes, and conflicting systems were used in parallel for some time, but fortunately for us Cavazzoni predates all this craziness.
From the point of view of modern analysis, the plagal and authentic pairs were effectively the same. A modern analyst would say that a Hypodorian magnificat is in the Dorian mode or perhaps just in D minor (or whatever minor key if transposed). The difference lies in the chant tone of the cantus firmus on which the piece is based, because the reciting tone (the "dominant") will be a different pitch: F instead of A.
how is octavi toni different from primi toni?
Octavi toni is Hypomixolydian, so basically a major key with the reciting tone being the third or fourth scale degree. Primi toni is Dorian, so basically a minor key with the reciting tone being the fifth scale degree.
Wikipedia has a decent summary. In particular, take note of the table.
To apply this to the two pieces cited in the question, the Magnificat octavi toni is in Hypomixolydian transposed down a whole step, that is, with the tonic being F and the reciting tone being B flat. The lack of an E flat in the key signature corresponds to the frequent need, for harmonic considerations, to raise the leading tone (that is, to use F# accidentals when the tonic is G). That practice eventually led to major tonality. The Magnificat primi toni seems to be in a major mode at the beginning, but at the end you see that it's in D Dorian, albeit with some chromatic alteration, namely both B flat and B natural, as well as a raised third in the final chords. The tone begins on F and ascends to A, the reciting tone. In the end, it descends from A to D.
It's worth adding that the tones are more than just modes. The tones are specific melodies that are used for reciting psalms and canticles. In addition to the numbered tones, there is the tonus peregrinus, so called because the reciting tone of the second half is a step lower than the reciting tone of the first half. There is some variation in the melodies for any given tone, especially from one region to another, but they are substantially similar. For example, the first half of the first tone is generally F, G, any number of A's, then B-flat, A, G, A. Then the second half is any number of A's, then G, F, A, G, F, E, D.
