The brief answer is no, at least nothing like the modern sense.
The more nuanced answer is sort of and sometimes.
First, let's note that score format did exist historically early on. Almost all polyphonic Western music from its earliest forms (ca. late 8th or early 9th century) to the early 1200s was in score format. (See, for example, Notre Dame polyphony, as in this example by Perotin.) The introduction of clear rhythmic notation around that time allowed polyphonic parts to be split into choirbook or partbook formats. I think the common assumption among historians (though I don't know if we have any contemporary documentary testimony to this effect) is that scores began to decline after ca. 1225 because choirbook and partbook formats required less space to write music. As musical styles of the time often had a part or parts moving much slower than other parts, there was a lot of wasted space in a score with the slower part that had fewer notes.
The development of the motet (which emphasized some independence of parts, including eventually completely separate texts) also was involved in the emergence of choirbook format -- where all the parts are written out in separate places on two facing pages of a book, so that a choir gathered around could simultaneously read the parts. However, other genres continued to be written in vertical parts (sometimes more-or-less aligned) throughout the 13th century and occasionally beyond.
But the modern score notion first starts showing up in instrumental music of the 14th century, even including regular barlines. This is a somewhat rare practice and confined only to instrumental music. It's not until the mid-1500s that we begin to see similar examples of scores in vocal music, and they don't start to become standard until around 1600.
So, there are several centuries there where scores are quite uncommon. How did composers compose?
The classic study on this topic is Jessie Ann Owens, Composers at Work: The Craft of Musical Composition 1450-1600 (Oxford, 1998). If you want to read about the rather fragmentary evidence in terms of extant artifacts, sketches, fair copies, revised versions, etc. of renaissance composers, this book was the first to really look at a lot of it in depth. It built on Owens's groundbreaking article a decade earlier on the compositional process of Cipriano de Rore.
Among other things, Owens clearly shows that composers could and did work without scores. (Through much of the 20th century, a lot of music historians had asserted -- without evidence -- that the complexity of renaissance music simply implied that composers must have had and used scores, but like many autographs of later composers, these were simply lost to time. All we had left were the works of scribes and printers.)
Owens showed this first through the discussion of a set of partbooks in the hand of Cipriano de Rore, which display sketches, mistakes worked out, etc., all in the partbooks themselves. Clearly Rore was capable of thinking and composing in multiple parts and putting them directly into partbooks.
But there was more than that. The partbooks also had an affidavit of authenticity penned by composer and organist Luzzasco Luzzaschi, who discusses a cartella that Rore apparently used while writing compositions, which had been gifted to him. While that particular cartella doesn't survive, it likely referred to a slate tablet (as phoog discussed), though wax tablets were also used. As discussed in Owens's book, fragments of these have been discovered in archives and archaeological digs: some of them have pre-drawn staves on them (often 10 lines) and some have even been found with a few notes scribbled on them. We also have iconographical evidence of this practice (e.g., in illustrations showing composers with a cartella).
These were clearly not large enough to notate a complete piece. Owens argues that they were primarily used for solving more contrapuntally complex segments of music (like coordinated points of imitation). Those could then be transferred to partbooks or choirbooks, and a transitional passage could be composed leading toward the next more complex section. Owens argues based on documentary evidence (mistakes in partbook drafts, other compositional drafts, and sketches found in marginalia on other manuscripts) that this was the likely mode of composition; the higher rate of errors and part-writing problems in transitional passages seems to support such a methodology.
Note that the fragments that would have been written on the slate or in the margin of some scrap paper weren't really "score" in the modern sense, but what Owens calls quasi-score, with parts jumbled together often -- generally without barlines and not always vertically aligned well -- on a 10-line staff (or staves) that could allow the full standard Guidonian gamut of the time.
Another well-attested practice from the time is having choirs actually sing the pieces, with the composer making corrections based on the performance. This is likely one reason why almost without exception renaissance vocal composers were also prominent singers in choirs themselves. We have accounts of several composers "workshopping" their pieces like this (including prominent composers like Josquin), and perhaps this process was used more actively during the compositional process for some.
For some genres, it may have been common practice to compose a two-voice structure first (or even a single voice, to which a second was added, like a cantus firmus), then compose a third voice (or more) against that structure, making modifications to the existing voices where necessary. This could help contain the variables in free composition without a score. We have accounts from theoretical treatises stating that this was the "old way" of composing, though how common or practical it was in renaissance counterpoint is an open question.
But we also need to confront the other issue with understanding historical authorship processes, which is that human memory was simply different (and used in different ways) at a time when books were expensive and difficult to make. (Every time I see a medieval manuscript, I immediately think of how many cows or sheep had to die to make the thing, and how many laborious hours went into scraping and stretching and preparing the parchment even before scribes made any annotations.)
We have accounts of Thomas Aquinas, for example, supposedly dictating separate works to different scribes all at the same time. For many years, these were viewed as simple exaggeration, but Mary Carruthers challenged that thesis in The Book of Memory (first published in 1992). Through painstaking documentation of medieval treatises on memory techniques and processes of working with written words she began to reconstruct the medieval "Art of Memory," a set of practices that greatly influenced the way medieval and renaissance people read and also how they wrote. At a time when visiting a book in another monastery might be the only time you ever saw the book, the ability to effectively memorize it was essential.
In a culture where written books were such a luxury, it would not be uncommon for a skilled person like Aquinas to compose an entire chapter or perhaps even an entire treatise in his head at once, to have it effectively "memorized" so he could work over it and revise it in his head, only then to take it to a scribe who could put it to parchment. (Keep in mind how much was memorized at this time by even your average monk, who would chant/recite the entire book of Psalms from memory, often on a biweekly cycle, for example. Obviously various tricks and techniques would be developed to facilitate the memorization process.)
There's little doubt that music composers were able to work much the same way, and we have accounts at least of composers spending time working out pieces in their head, perhaps punctuated by work with the slates or whatever for particularly complex passages. We know this because performers apparently could do it from choirbooks or even partbooks. We have accounts of organists and other musicians at job interviews from the 16th century being asked to play motets (even up to six parts!) at sight from a choirbook. According to many sources, this was a common expectation particularly of organists to be able to not only work with their own music but to study and learn the music of others. Frequently even theory treatises of the time would give examples in choirbook format, assuming the reader could piece them together (though we begin to see more score formats emerge there in the late 1500s, first in examples with more than two or three parts). A final piece of evidence that a significant portion of music must have been composed directly into partbooks is Nicola Vicentino's instructions for checking for part-writing errors (like parallel octaves and fifths), where he encourages each part to be checked individually with each other part, note by note. The phrasing about "part by part" clearly seems to imply that a student normally would not have a score. (A good account of sources about this can be found in Anne Smith, The Performance of 16th-Century Music: Learning from the Theorists (2012), particularly Chapter 2 on "Part-Book versus Score Culture.")
Such accounts may seem somewhat incredible, until you think about how mystifying it is to a beginning student to see a pianist read four-voice SATB open score along with bits of the two staves of the accompaniment all at the same time, a rather common skill for choral accompanists. It takes practice, but it seems rather "natural" once you can do it. Good conductors can often extract information from a much larger score in real-time or even play a version at the piano too. I myself became somewhat adept at one time at reading two voices concurrently from separate parts (and have occasionally done the fun exercise of trying to condense a four-hand piano piece into something playable myself), and I've certainly puzzled out short sections of a choirbook piece with maybe three or four parts in my head when looking at a manuscript in a library... though the thought of actually playing through a choirbook format with four or more parts at sight seems quite daunting to me too. But it's apparently a learned skill, which we know organists of the time were asked to do, so talented composers during the renaissance likely had it as well.
