What does it mean to be "in a key"? That's really at the heart of this question.
The first question I'd ask in response is, "Why do you care? What are you trying to determine by saying piece X is in key Y?"
Some people are looking for a simple answer, often determined by the key signature and what local key/chord progression a piece ends on. They will just declare that to be the key and be done with it.
Other people are interested in something more nuanced. One might ask what local key a piece is in at a particular point in time. Some of those people might look at the local context, including accidentals, cadences, etc. and again just declare on the basis of the score that "In measure X, the piece is in key Y." If there are some conflicting factors, they will weigh various parameters and make a determination based on a global view of the score, as if they were solving a puzzle. This is the type of "analysis" frequently done in an introductory music theory course, where it seems like the goal is often to label a particular place with a Roman numeral in a particular key.
Of course, sometimes the ambiguity gets stronger. Some pieces aren't even tonal at all. But assuming you're working within the general framework of tonal harmony, we might ask as a listener, "How do I know what key I'm in at point X?"
And that's a much harder question. It depends partly on your priorities in analysis. Some people may just look at the local scale and declare an answer based on that. Certainly if you're trying to determine whether you're in D major or B minor in a classical piece, the use of A-sharp as leading tone will often determine the key (i.e., B minor if A-sharp is frequently present and used as leading tone; D major if not). Local cadences, points of rest, and emphasized harmonies might also give clues.
But that may still not be enough. To go to the Brahms example in the question, the beginnings of pieces of music in particular may often be quite ambiguous. Suppose you place a single pitch D, maybe in multiple octaves. What key are you in? D major? B minor? What about G major, if D is the dominant? Or what if the notes that coalesce around D are actually B-flat and then F and A-flat, and suddenly you resolve to E-flat, which is your key? Retrospectively, you realize that D you started on was the leading tone.
But you can't know any of these things with a single pitch D sounded. In that sense, you might spend a few notes (or a few bars or even longer) at the beginning of a piece not actually knowing as a listener what key you're going to end up in. Many composers actually seem to enjoy creating sort of "fake-outs" at the beginning of pieces, and this became more common in the 19th century.
In that sense, it's very easy to say that a piece can be in an ambiguous key for a while. If you have a score-based perspective on analysis, you may look ahead and say, "Oh, obviously that D is a leading tone, which you can see four bars later." But as a listener you don't yet have that information.
I'd compare the beginning of a piece of music to the creation of a "world." The composer can give you clues about the rules of that world -- what it looks like, what places you can go, what places you are unlikely to go, what the rules are about how this world operates (is there gravity? which way does it pull?), etc. These are all expectations set up not only by key, but also by instrumentation, texture, dynamics, articulations, etc. To use an analogy, it can be like being dropped down by a video game into a new world with a slow graphics card. In some simple cases, everything appears at once, and it's very clear what the world looks like. In other cases, you drop in with a field of white around you -- and then gradually out there you see a splotch of green, and then there's a tree. Later you hear a trickle of something, and a stream appears. As the field of view fills in, you get a sense of what the landscape is. And the you can begin (as Tim says) your journey in this musical world.
To be "in a key" as a listener in an emerging world usually means that there are signposts or pointers that designate a particular note as having greater importance or "home." A stereotypical gesture is a leading tone resolution or a V-I cadence: if you have one of those, it's like a bright red flashing arrow pointing at a note in your world and saying, "There! That's home!"
But then you look at a piece like the opening of the Brahms Intermezzo. The first note is F-sharp. (The splotch of green.) Are we in F-sharp? No, a few notes later we've outlined a descending B minor triad. (A tree appears.) Are we in B minor? Well, the next notes might be an E minor triad, which could be iv. But in the next bar we get an A7-D implication. (The stream has emerged.) We might make a new guess: perhaps the piece opened on vi, and we just heard a circle-of-fifths sequence (vi-ii-V-I in D) that points us to D major. But rather than a bright red arrow cadence pointing us to D major, we don't get that strong chordal bass motion of a 5-1 leap, just a hint at V7-I in D major.
And the next bar starts doing some weird stuff: F-sharp minor? In D major, that's an unusual iii chord. In B minor, that would be an even more unusual minor v chord (for classical harmony). What the heck is that? It's like our world opened with some trees and a babbling brook, and suddenly a giant jellyfish dropped out of the sky and landed with a plop in this forest landscape. What's going on?
The only way to know is to "stay tuned" and try to make sense of the developing tonal landscape. At this point, as a listener, if you asked me what key I'm in, I'd say, "I don't know. Could be B minor. Could be D major. Definitely not clear." In essence, ambiguous. Key choices need not only be ambiguous between relative major and minor. As noted above, a limited set of notes may not make clear what key you are in for a while. The opening B minor triad by itself could be a i in B minor or a vi in D major or a ii in A minor or a iv in F-sharp minor or even something more exotic. Only further context can determine that. And sometimes the context can be ambiguous for quite some time, or even for an entire piece. Brahms continues the play with listener expectations for quite some time in this particular piece, introducing various chromatic bits in the subsequent measures that twist and turn and hint at unseen destinations within the dark forest.
Some people resist this idea of ambiguity, generally because some early experience with music theory has taught them that there's supposed to be "one right answer" for an analysis. Introductory exercises in theory often are chosen specifically because they lack ambiguity and are easy to understand (and easy to grade). Thus, students get a false sense that a piece must be in a particular key, and a particular chord must have a particular function in a particular key. Some people might therefore insist that this Brahms opening is NOT "ambiguous," but rather oscillating between keys or something. Maybe. That's one interpretation. But if you believe in the concept of tonicization (i.e., a temporary move toward a non-tonic note, without changing key) or even just an idea of applied/secondary dominants, then one could easily analyze a lot of this piece in at least two different keys. Which one is the "true" governing tonic? Does one have to oscillate or move between keys? The real listening experience of music, particularly for a piece like this Brahms Intermezzo is not always that determined. The ground in the landscape may never seem firm and always be shifting under your feet.
So yes, in my opinion, it's very possible for segments of a piece and even an entire piece to have an ambiguous sense of key. For other analysts who want a "one true answer" and a label to put under every chord to "solve the crossword puzzle" (as I generally think of Roman numeral "analysis"), they might want to weigh the various options and argue for a particular tonic at any given moment. If you forced me to, I could make arguments for what Roman numeral to put under every chord in this piece and what local key you're in. But personally, I'm often happier just to live in the shifting tonal landscapes as the stream babbles on and leaves blow, wafting wistful strains in various directions accompanying me on a journey that may or may not ultimately take me to a "home"....