Was the term “octave” coined after the development of early music theory?
No. As shown below, it was already in use by the 11th century to denote the musical interval (although the principal name for the interval at that time seems still to have been diapason).
What system was in use in medieval Europe when the term octave arose, and what did the term octave refer to?
I'm a little hazy on the state of music theory before Guido d'Arezzo, who worked in the early 11th century, but it's pretty clear that the letter names A through G were in use before he published his system of solmization with the six syllables ut, re, mi, fa, sol, and la.
In his treatise Micrologus, published around 1026, he mostly refers to the interval that we now call the octave as the diapason, but it's also clear from chapter V that the term "octave" had come to its musical meaning. Keep in mind that the day one week after some other day was also called the octave, which is simply Latin for "eighth," and, as has been noted elsewhere, the periods of a week and two weeks are to this day called eight and fifteen days in at least some modern Romance languages.
As an aside, I note that the only difference between numbering the intervals of the second, third, fourth, fifth, sixth, and seventh, on one hand, and the octave on the other, is that we use our native (Germanic) ordinals for the first set of intervals and a word derived from the Latin ordinal for the last. There's no theoretical reason not to call it an eighth; it's just custom, perhaps influenced by the special status given to that interval by octave equivalence.
Guido's system had eight pitch classes, the seven letters of the "white key" scale, A through G, plus B-flat. To distinguish, B-natural was written with a square form (𝇒), while B-flat was written with the normal round form. The two forms of B would never be used at the same time, though, so there were seven distinct pitches in the diatonic scale.
The system comprised roughly two and a half octaves, the lowest in upper case letters, the middle in lower-case letters, and the upper in double lower-case letters. Oh, and because he needed one note below the lowest octave, that note was designated with a Greek letter. There were, in total, 21 notes:
Γ A B C D E F G a b 𝇒 c d e f g aa bb 𝇒𝇒 cc dd
Guido, Micrologus, chapter V, on octave equivalence:
Diapason autem est, in qua diatessaron et diapente iunguntur; cum enim ab A in D sit diatessaron, et ab eadem D in a acutam sit diapente, ab A in alteram a diapason existit: cuius vis est eamdem litteram in utroque habere latere, ut a B in 𝇒, a C in c, a D in d, et reliq. Sicut enim utraque vox eadem littera notatur, ita per omnia eiusdem qualitatis, perfectissimaeque similitudinis utraque habetur et creditur. Nam sicut finitis septem diebus eosdem repetimus, ut semper primum et octavum diem eumdem esse dicamus, ita octavas semper voces easdem figuramus et dicimus, quia naturali eas concordia consonare sentimus, ut D et d.
Corrections to this translation are welcome. To maintain the distinction between the Greek and Latin interval names in the original, I've used Greek and English names here. Diapente is fifth, and diatessaron is fourth:
The diapason is, moreover, that in which the diatessaron and diapente are joined, for from A to D is a diatessaron, and likewise from D to acute a is a diapente, so from A to the other a there is a diapason: whose effect is having the same letter on either side, from B to 𝇒, from C to c, from D to d, and the rest. Thus as both sounds are notated by the same letter, so both by their property they are ascribed the most perfect similarity. For as, after seven days, the same [day] repeats; as always the first [primum] and eighth [octavum] days are called the same, so octaves [octavas] are written and called alike, for we perceive their natural harmony together, as D and d.
The question assumes that the seven-note scale arose by taking a subset of the twelve equal divisions of the octave, but this is not the case. Guido's introduction of two forms of B was the beginning of what later came to be known as chromatic alteration. With the advent of harmony, alterations of notes other than b and 𝇒 became necessary. There was E♭, then A♭.
As the raised leading tone became important, F♯ was invented, then C♯. Eventually the sharps and the flats met somewhere around D♯/E♭ or G♯/A♭. Although there were some experiments with split-key keyboards to enable those pairs of notes to have different pitches, the winning strategy in the long run was to devise acceptable temperaments in which they could have the same pitch, leaving us with the 12-key keyboard we know today. In other words, the 12-tone system arose as an extension of the diatonic system, which came first.
There is another misconception in the question:
(during definition and use of the ionian major scale -I know modes weren't defined up until much much later-).
In fact, the modern Ionian mode was not identified as such until the sixteenth century, along with the Aeolian. The weird Locrian mode came later still (weird because its fifth degree is diminished). The other four modes are in fact older, being directly descended from their medieval authentic counterparts, developed in the 9th century. (These modes were named after the Greek modes, but they don't descend directly from them.)
In fact, the major and minor tonalities that were firmly established in the Baroque period developed from the four medieval modes during the Renaissance period. The modes were much earlier, not much later. Major and minor tonality developed as a result of the influence of harmonic considerations on various contrapuntal conventions, and the Aeolian and Ionian modes were subsequently invented to reflect this.
The basis of minor keys is the Dorian and Phrygian modes (primarily the Dorian), while the basis of the major keys is the Lydian and Mixolydian modes. Harmonic considerations led to the raising of the leading tone in Mixolydian and Dorian, the lowering of the fourth degree in Lydian, the raising of the second degree in Phrygian, and the lowering of the sixth degree in Dorian (and even today, the sixth and seventh degrees are so commonly raised or lowered in minor keys that we are taught three different forms of the minor scale, one of which even changes depending on its direction).
