I'm not sure if there IS a definitive answer to this question, but here are some thoughts.
In the Baroque court, composers were under such time-constraints there would have been little time for soloists to memorize the music. While the orchestra was drawn from the court entourage, who were already on the payroll, professional musicians were hired to play the solo parts in concerti grossi. Because of the expense they would not have been allowed the time to learn the music, so were undoubtedly reading it. They must have been extremely good at reading.
Classical piano concertos were often first performed by their composers, who probably knew them by heart. Perhaps that led audiences to expect the soloist not to be reading the music.
But playing concertos without the music isn't necessarily simply a tradition. By the time you have practised (US: practiced) a concerto enough to perform it well, you HAVE learnt it! Also, if you're a pianist or a string player and the piece is at all demanding, you need to see where your hands/fingers are on the keyboard or fingerboard, and there isn't time to keep looking away at the music. It's difficult to play difficult music while looking at the music!!
Accompanists don't have technically-challenging music to play, and they usually read. Singers don't need to see their hands/fingers while performing, so they can have music. So can wind/brass-players. The only reason they, and sometimes singers, don't is that, as I said above, by the time you've practised it enough you've learnt it.