There's a long history to both of these forms. And I'm going to assume you're talking about the standard forms of the late classical and early romantic periods (around the year 1800). Before and after that time, things called "symphonies" and "concertos" might have a varying number of movements, not just four or three.
Briefly, the tradition of alternating slow and fast movements goes back to the early days of large-scale instrumental pieces in the 17th century. But both the classical concerto (derived from the baroque solo concerto) and the classical symphony have their roots in the opera sinfonia tradition. A sinfonia in early opera was the name for an instrumental prelude (what we now call an overture) or an interlude or postlude.
Opera sinfonias, particularly the ones that served as overtures, developed a typical three-part structure, beginning with a fast section, followed by a slow section, and then concluding with a fast dance-like section. Eventually composers started writing separate instrumental pieces using this model, even when they weren't connected to operas. Early solo concertos followed this three-movement fast-slow-fast(dance) model, as did early symphonies. (In the mid-1700s, many symphonies were also written with three movements, including early symphonies of Mozart.)
The introduction of the fourth movement is usually credited with the so-called Mannheim School of composers, who were most prominent in the 1750s-1770s. There, the minuet was added as the third movement of a symphony. Previously, minuets had been part of baroque dance suites, another instrumental genre. Occasionally composers inserted a different dance other than a minuet into the four-movement structure of a symphony or sonata, but the minuet was most typical. Later, the triple-time minuet movement was often sped up to the point that one couldn't even theoretically dance to it anymore, and then it became known as a scherzo.
In any case, the added movement gradually spread and became common in symphonies, making a four-movement structure standard, while classical concertos typically retained the standard three-movement structure.
(Again, note that this movement structure is only true of these particular traditions. The Vivaldi Concerto mentioned in the question is part of this solo concerto tradition. Other baroque concerto types did not necessarily have a three-movement form: the ripieno concerto and the concerto grosso common in the baroque period could have three movements, but frequently also had more.)