How did composers "test" their music, especially in multi-instrument works (eg. chamber, orchestra) or in pieces for instruments they didn't know how to play? Did they really have an orchestra (for example) next to them? Now, we have DAWs and score-writing softwares that come with MIDI playback, but I'm really curious about the past. Any answer is welcome.
Not all composers nowadays write using computers. Many still write by hand using ink or pencil.
Multi-instrument works (chamber, orchestra, etc) were written as either what's known as "piano-score" or "short-score". Many, many composers are / were pianists and so were able to check their music by playing through the piano score. Once they are satisfied with the music, they orchestrate it out to the size of the ensemble they're writing for.
To learn how to do this, a common practice at the time for students was to hand-copy well-known composers' music. From this practice they would learn what worked for different types of instruments. During those times, brass and woodwind instruments were not as technologically advanced as they are today, so many of the wind parts (especially brass) were not especially complicated (though there were obviously virtuosi on those natural instruments, thinking about Hadyn's trumpet concerto or Mozart's horn concerti).
Aside from these considerations, there are many sounds developed over the past century that weren't being used. For example, advances in technology have enabled composers to experiment with: sampling, recording, live-sound manipulation, electronic music, noise music, performance art, and multimedia art to name a few. As a consequence of science (i.e. electricity), many of these technologies did not exist and therefore the sounds / means of expression did not exist. This means a different level of experimentation, which means things are more standardized, which means things don't need to be checked as often.
If you're involved with music deeply enough and for long enough, you become very familiar with the instruments, their techniques, and the sounds that come from them, which brings me to my last point:
Experience. I've played horn though I wouldn't consider myself a horn player. When I write for horn, my experience tells me through my mind's ear how the horn will respond throughout it's range to different musical gestures. I don't need to go running to a horn player every time I write something for horn. That said, if I do write something and am not sure how it will come out, I'll reach out to a horn player for help; a whole orchestra is not needed.
Other composers have done / do / will do the same thing: use prior experience (mistakes) as a guide and ask questions of individual players when they need them. Many university programs offer orchestral "readings" for students, which can also offer invaluable feedback.
The more you listen to and work with music the more you "hear it in your head". After you become proficient at transcribing, it's perfectly possible to see a melody written out and "hear" it in your internal ear.
Doing the same for 2, 3 ... n voices is basically the same thing, just more of it (and of course requiring more practice and experience).
In fact, the real answer to this question varies by composer. Some had more or less a captive orchestra or choir to work with (like Bach for much of his career). Some like Beethoven couldn't even hear (eventually) and yet could write. Some worked mostly on piano, some (such as Tchaikovsky apparently) weren't particularly great at playing anything.
It's also instructive sometimes to look at scores (not only classical) and see how much, or often, how little, is going on. Sometimes there are many interwoven parts, but sometimes, although there is an orchestra, there is not much counterpoint happening. Then the artistry lies more in having a melodic idea which develops and remains interesting, and skill in orchestrating. Less is (sometimes) more.
In any case, it really comes down to the "internal ear" before anything else. Without this, you are unlikely to write anything that sounds much better than random as a start point.
First, I would note that any proficient classical composer likely knew the basics of how to play all the instruments being written for. The composer may not have any great ability to perform on all instruments, but any good composer needs to understand how different instruments produce sound, where difficulties in execution arise (e.g., difficult fingerings, notes that will be more out of tune, etc.), as well as the capabilities of the ensemble being written for.
That's not to say that composers would always try out parts on said instruments themselves. (They might do it, particularly for more difficult passages, perhaps even trying out a solo part with a skilled soloist on a specific instrument.) But experienced composers, as noted in other answers, tended to be able "hear" music in their head. It's not unlike how most people today commonly read written words without speaking. Note that historically this wasn't always the case: there is some evidence that in the ancient word, reading silently was unusual and in some cases considered remarkable. But obviously with practice most people can learn to read text and understand it without vocalizing it aloud. Similarly, one can read music -- not only single melodic lines, but even large scores with many instruments, and understand them, knowing what they would sound like, without literally hearing them aloud.
I'd just add that a strong element of this is fluency in traditional music education, which gave lots of rules for producing music that "sounds good" by following technical principles. For example, if one knows the detailed rules of renaissance counterpoint (as practiced by composers like Palestrina, and described in treatises such as those by the historical music theorist Zarlino), it's a lot easier to write music that will sound like good renaissance music simply by following the rules. That's not to say there aren't other elements in making the music sound artistic and interesting, but simply following formulas and rules automatically will produce something that's likely to sound vaguely like "correct music" in that style.
Similarly, when it comes to other instruments and orchestras, composers were taught principles of orchestration: what instruments blend well together, what instruments contrast, how to create balance (dynamically, timbrally, etc.) within an ensemble, how to create well-voiced chords using different instruments, etc. Most of these principles were at first passed down informally from teachers to students, but by the early 1800s, you had some teachers writing detailed books on orchestration.
Again, these principles can't guarantee that what you write will be interesting, but following these ideas and understanding these concepts will allow a skilled composer to create orchestral blends and interactions among the instruments that will generally sound competent. Today, orchestrators for things like movie soundtracks often still make use of similar concepts: the actual "composer" for a movie may sketch out the ideas of melodies, themes, general instrumentation, and then a sketch of the overall piece, but orchestrators may fill in the details using standard principles. Orchestrators in this case are often specialists, who worry about the actual details of real-world performance. The trumpets and flutes playing together on a computerized score may seem fine, but in the real world, you might not be able to hear the flutes at all. Or, what about the fact that wind players need to breathe? It's easy to write a wind part that goes on forever on a computer, but it may not work in the real world.
Actual composers even today thus need to be able to evaluate music outside of computer simulation, particularly when writing for large ensembles. My experience working with a lot of students who grew up in the era of computerized music writing is that they tend to write a lot of stuff that seems fine on the screen and in "playback," but could never work in a practical sense. In some cases, things that sound reasonably good on computer will sound very different (or even bad) when executed on real-world instruments by real-world performers.
Still, composers did "test" their music in performance. Today, we have a stronger sense of a finished "musical work" than was often true historically. Composers in the past might be more likely to keep revising a work based on experience (and things that they noticed didn't work so well), but also based on the event, the specific ensemble including what players/singers were available, etc. Look at the many versions of Handel's Messiah for example, where the piece was frequently altered from year to year based on what types of soloists might be present. In the real world, the specifics of the performance -- even the specific players available -- can make a difference in what is most successful.
And it's not just historical composers that needed to "test" their music. If you're orchestrating a score for a Hollywood film today, studio time with an entire orchestra is incredibly expensive. You can't afford to realize in the middle of a studio session: "oops, this blend doesn't sound anything like it does on my computer" or "wow, the harmonics of that instrument are conflicting with those other instruments in the higher register" or "I didn't realize a bassoon will have real difficulty playing that particular trill, followed by that leap." Such mistakes can literally cost hundreds or thousands of dollars in lost time. That's not to say they never happen -- studio sessions will make changes as they are recording. But skilled composers and orchestrators can avoid many of these problems just from their knowledge of "what sounds good" in the real world and what works practically, regardless of what a computer output might sound like.
Some write with a piano to test out rhythms and chord progressions. Some just hold all the sonorities in their heads.
But to your point about orchestral works, I've had the pleasure of playing pre-publication pieces at summer music camps or other venues, often conducted by the composer. The composer often modifies the final version after such "test performances." In fact, if you read through enough "liner notes" you'll find that many classical composers rearranged symphonies or operas after initial published, public performances.