These kinds of situations happen often—people's performances get assessed and compared—and have happened for centuries. And yes, when the stakes are high, people have tried to find ways to make them as just as possible and eliminate opinion and bias as much as possible. (But, spoiler: "as much as possible" still leaves a lot to chance or preference.)
One question has to be what the goal of the assessments is. I've been a part of violin auditions at a university music department, where one could audition for the college orchestra, for private violin lessons, or for both. We would see a broad range of abilities, from near-beginner through to the Tchaikovsky concerto. Part of the goal of the auditions was to sort and distribute the students—who got into the orchestra, who didn't, who went to the first violin section vs. second, and which teachers got which students, trying to distribute them in ways that were fair to the teachers, but also didn't give lessons to the beginner while denying the more advanced student. I've had students perform as part of their grade in a course, and had to figure out how to grade all students fairly. I've also judged auditions for a large kids' music school that had three orchestras and chamber groups; there, the question was not "did you get in" but "into which ensemble." Other auditions might be for a job—say, a concertmaster in a professional orchestra, or a part in a musical or opera. And then of course there are competitions that exist for their own sake, like the Tchaikovsky Competition—simply to see who will win.
These all have different goals. The music-school auditions are like Harry Potter's "Sorting Hat." They can have many right answers and assess many different levels. An audition for an on-stage part is more complicated; it's actually casting, and besides simple talent it must take into account whether the person fits the part well. Auditions for orchestra section chairs might also take into account a person's communication, leadership, and organizational abilities. Auditions to join a rock band might emphasize "soft skills"—sure, the applicant has chops, but do they "mesh" with the existing members, in personality as well as musical idiom?
For some of these assessments, "fair" matters less than others. For a leading role in a show, you can totally deny a person a role because they're too old, or too tall, or just lack that "star quality." For joining a small combo, the people-skills might actually matter more than the chops. But for events like an audition for a concertmaster or a musical competition, the thing that (supposedly) matters most is performance and the stakes are high, so people take a lot of steps to try to ensure justice (or appease accusations of injustice).
You've touched on a lot of the best ideas already. Yes, these events usually use a panel of judges rather than just one, and average their scores. For an orchestra, this often includes people who have an interest in the outcome, like the conductor, any owners or board members, and other section chairs. Ideally, organizers might intentionally seek out diversity in this panel; for instance, including an old-school traditionalist teacher and someone with more modernist tastes. I'd like to hope they sometimes seek out racial or gender diversity on the panel, though the cynic in me doubts you can find much in an orchestra board to begin with.
Measures are taken to avoid biases about the performers. At early levels of a competition, a section of wall or opaque screen is placed between the performer and judges, so they can't see gender, race, or any physical characteristics. A carpeted runner goes from the door to the wall to muffle footsteps that might give away, say, high heels or body weight or a wheelchair. The effectiveness of these measures is somewhat limited because, once the applicants have been narrowed down to a short list, the wall is removed; in fact, finalists might even be "tried out" for one concert each to see about that "good fit." So any potential biases are suppressed only long enough to keep a candidate from getting close.
And, on the other hand, some argue that this anonymity impedes diversity. In 2020 an editorial in the New York Times by Anthony Tommasini argued that the screen should be removed. The "blind audition" perhaps smacks of dated rhetoric about "color-blindness"—that we can get a just society by ignoring race (or gender). But Tommasini and others would suggest that if we value diversity we must seek it out actively, with open eyes. (There was a predictable backwash of controversy following the editorial, ranging from "nuh-uh" to "it's complicated"—suffice it to say, if the screen impedes diversity, it's not the only part of the system that does.)
And yes, it can be challenging, as a judge, to compare performances, especially as they stretch over multiple days. To keep us from saying "Gee, I dunno, I guess maybe a 7?", the kids' music school had us consider a rubric, giving scores to intonation, rhythmic accuracy, steadiness of tempo, and expression. (I used the same rubric when grading private students, but assessed them at the beginning and end of the term and graded on improvement.) These categories made sense at student levels, where they are definitely not guaranteed; at professional levels, you'd hope everyone had the right rhythm and a steady tempo—and hopefully, good intonation. Other attributes might take their place, like "clear articulation in fast passages" or "warm tone in lyrical passages." And of course judges judge in different ways. Not only might they disagree about what they like, but they might have different scales; one might say "That was great; 10 out of 10!" while another says "Well, it's not the greatest thing the human race has ever produced, so I'd better reserve '10' for if that ever comes along." But as long as each judge is consistent, then averaging across them should help equalize this. (One concerning study found that judges on a parole board judged more leniently at the start of the day, but more harshly as they got tired and hungry. Hopefully this effect might be offset by holding multiple rounds of auditions!)
But ultimately, especially in a pure competition, much is left to chance and to subjective taste. For one thing, just like the Olympics, a performer might have an "off day," or even be off their stride throughout the rounds of auditions, while they might play better another time. For piano competitions, all performers play the same piano, but for violin, flute, etc., the instruments vary wildly—not in their quality, necessarily (since that's a subjective matter too), but in their nature—"bright" or "dark" tones, different powers of projection or clarity. And ultimately, the highest echelons of competitors will have very similar abilities, and the prize may come down to the judges' preferences, even as a panel. How fast "should" the piece be? How reserved should Brahms be? How outré should Mozart be? What matters more, clarity or nuance?
And ultimately, the outcome has to be just that. Not "this is the best player in the world," but "this is the player who best satisfied the body that created this competition... with the performance they accomplished on this day."
