A professor of mine in graduate school liked to use the phrase elevator operator to demonstrate that there is a subtle metrical hierarchy. Imagine that phrase set in eighth notes in a 4/4 bar. "Op-" comes on beat three. It carries the principal accent of the word "operator," but it (normally) receives less emphasis than the first syllable of the first word. Both words also have a secondary accent on their third syllable, which corresponds to beats two and four, and the unstressed syllables correspond to the even-numbered eighth notes.
I say "normally" above because the choice of primary stress actually depends on the context. For example it would be different depending on whether you're answering the question "is that the superintendent?" or "is that the elevator technician?" In the second case, you would emphasize "operator" over "elevator."
As a blatant oversimplification, but a good starting point nonetheless, consider this: the second instance of anything is less important than the first. Beat 3 is less important than beat 1. Beats 2 and 4 are less important than beats 1 and 3. And the off-beat eighth notes (the "ands") are less important than the on-beat eighth notes.
Now that we've got that out of the way, on to the terminology. It is unfortunately not very strict. In the strictest sense, a measure can have only one downbeat, the first beat after the bar line. But people don't always use the term in this strict sense. Suppose there are 16th notes in the measure: in a very real sense the 13th one is the "downbeat" of beat 4.
Furthermore, the meaning of "upbeat" is even looser and more flexible. In your example, it would not be unusual to hear someone refer to beat 4 as "the upbeat" to the next measure, but it would also be perfectly unexceptional if someone called the last eighth note "the upbeat." Indeed, I probably wouldn't think twice if someone referred to any of the off-beat eighth notes as the upbeat to the following beat. It depends on your "zoom level," if you will: what's the tempo? How long is the phrase? Are you talking about large-scale structure or more detailed articulation?
And now to your question:
When I have an eighth note in between those quarter notes that are not on the quarter note, is that the upbeat?
I would say that it is an upbeat, but it isn't necessarily the most important upbeat to you at any given moment.