As with most foundational questions in music, the answer is rarely "it's just an arbitrary convention." Usually, there is some sort of impetus for why these conventions arose historically, as is the case here.
The summary answer is that the whole tone emerged very early as a rather common interval in scales (generally tuned to a 9:8 ratio) and was standardized in size over 2000 years before a standard semitone size was settled on. From a practical standpoint historically, the semitone was treated as an afterthought in scale construction, effectively the "error" leftover after the other notes had been tuned.
More details:
The answers that discuss the pentatonic scale (which is found in many cultures) give perhaps a start at why semitones are not a necessary component of music, but they really don't explain why the whole tone became the standard measurement in Western music.
The simplest explanation is that the ancient Greeks realized superior consonances were to be found in ratios of small numbers. Specifically, they recognized consonances like the 2:1 octave, the 3:2 perfect fifth, and the 4:3 perfect fourth. (Note that at that time, they couldn't measure frequency, but these were the ratios of strings with the same size and tension but different lengths, which also create these intervals.)
The Pythagoreans in particular experimented with ways of combining these perfect consonances, and one of the most common arrangements was to divide an octave both with a perfect fifth and a perfect fourth. If you build a 4th and 5th up from the bottom note, as well as a 4th and 5th down from the top note, you end up with four notes in a 6:8:9:12 ratio.
All the intervals in that ratio are octaves, fifths, and fourths except for the one in the middle, which had a 9:8 ratio. As this interval was so easy to create from the other consonances, it became a standard fourth interval to use for tuning purposes. It was called a tonos, which comes from a root related to "stretching" or "tension," as the stretched string was the model for tuning scales.
The Greeks experimented with many other intervals, but the tonos became one standard measure, used to tune what we now call the diatonic (tuned "through the tones") scale.
Originally, what we now call "semitones" were not half of this tonos. The tonos was impossible to divide precisely in half, according to the Pythagoreans, who didn't believe irrational numbers were practical. Instead, a typical Pythagorean would start filling in tones of the scale by starting with the 6:8:9:12 framework I mentioned above, which could be thought of perhaps as spanning E to E on a modern scale.
The Greeks would tune downward, starting at the top to fill in the topmost perfect fourth (4:3). They would place two whole tones with 9:8 ratios (such as notes E-D-C), and then there would be the leftover bit (C-B). A 4:3 perfect fourth minus two 9:8 whole tones leaves an interval with a ratio of 256:243. That last interval is not a nice ratio, and the Greeks often treated this bit as almost a kind of "error" left over when all the other notes had been properly tuned.
At some point, some Greek musicians noticed that the 256:243 ratio (~90 cents, in modern interval measurement) was roughly half the size of the 9:8 whole tone (~204 cents). There were some vague murmurings in ancient Greece about a potential 12-fold division of the octave using these smaller intervals, but it never went very far. No practical tunings of this nature (to my knowledge) were ever recorded at that time.
Fast forward a thousand-plus years, and the Greek scale was borrowed as the foundation for medieval music. The 256:243 interval was still treated as a kind of dissonant "leftover bit" in the tuning of scales, but it acquired the name semitonium, denoting the fact that it was roughly the size of half of a 9:8 tonus.
As more tuning experimentation went on over the centuries, the semitone was almost always an afterthought. In addition to the perfect octave, fifth, and fourth from ancient Greece, eventually musicians started experimenting with 5:4 major thirds, 6:5 minor thirds, etc. For example, if you divide up a 4:3 fourth with a 5:4 major third, you end up with a 16:15 interval left over, which was 112 cents. Again, this was roughly half the size of the 9:8 whole tone, so it too was referred to as a semitone. But note the wide variance of these "semitones" -- even with the examples of 256:243 and 16:15 we already see a range of 90 to 112 cents. Actually, the term semitone was sometimes used to refer an even greater range of intervals, as again, this semitone was generally just the "leftover bit" (of whatever size was convenient) after all the other intervals had been tuned.
With that context, it becomes clear that the semitone could never be used as a standard unit of measure. It varied wildly over time and from tuning system to tuning system, whereas the 9:8 whole tone had been standardized for thousands of years. (I should note that with these tuning experiments, the "whole tone" also could refer to intervals of other sizes too, especially 10:9 or some other compromise. Yet, despite different size "tones" sometimes being used in scales, there was always THE tone, the 9:8 one the ancient Pythagoreans originally settled on when deploying the octave, fifths, and fourths together.)
It was only around the years 1700 to 1900 that Western scale tuning systems gradually moved toward equal temperament, where there are exactly 12 semitones in an octave (each exactly the same) and a semitone is always exactly equal to half of a whole tone. From the perspective of the past couple centuries, the semitone can seem like an obvious standard measure for the scale. But historically, it was very ill-defined and varying in size, the "leftover bit" after all the important notes of the scale had been tuned.