The colloquial understanding of music being in a particular major or minor key depends solely on the tonic ("home note") and whether the 3rd scale degree is a major 3rd (for major keys) or a minor 3rd (for minor keys) away. If both the major 3rd and minor 3rd for 3rd scale degrees appear, whether the resulting music is in a major or minor key often depends on genre (e.g. blues would still be in a major key, classical music is labelled with the key, video game boss themes would likely be in a minor key).
By this classification, modes are also categorized into major or minor keys. For example, E Phrygian and F Sharp Dorian are both "minor keys" due to their 3rd scale degree being a minor 3rd away from the tonic, while G Flat Lydian and C Mixolydian (and even D Phrygian Dominant despite this being a mode of the G Harmonic Minor scale) are all "major keys" due to their 3rd scale degree being a major 3rd away from the tonic. In practice, quite a lot of today's music flip-flops between modes that share a tonic (e.g. uses both the regular 2nd scale degree and ♭2), so this categorization of modes into keys is more justifiable.
You can even categorize music that uses all 12 notes of the chromatic scale into a key as long as you know the genre, which note feels like "home" (or at least is used the most often), and which of the 3rd scale degrees a major or minor 3rd away from the tonic is used more often. Note that some music that uses all 12 notes of the chromatic scale is designed to be atonal and therefore be in no key at all.
In classical and ragtime music, and also in pop, folk, and rock music and EDM to a certain extent, knowing the key of an excerpt strongly corresponds to knowing exactly which notes you will consistently be putting accidentals on for all music in that key. Your previous intuition that music being "in a key" means that you sharpen/flatten notes according to the key's key signature is correct for the most part. Major-key music indeed often follows its key signature, while minor-key music notably sharpens its 6th and 7th scale degrees quite often compared to what the key signature indicates. For example, expect to see F♯ and G♯ quite often in music in A minor despite A minor's key signature being blank.
Especially in classical and soundtrack music, music in a certain key is not always notated with that key's key signature. Soundtrack music notated with a blank key signature is commonplace. Classical music (e.g. Bach preludes, Mozart sonata 1st movements) often changes keys without changing the key signature. Trust your ears and the accidentals on the page and in the key signature combined more than the key signature alone when it comes to determining the key of passages of music.
Determining the key of entire pieces gets significantly more complicated due to pieces often changing keys partway through. Music often - but does not always! - begin and end in the same key. While a rule of thumb is that you're often safe declaring a piece of music to be in its initial key, music like Chopin's Ballade No. 1 in G Minor disobeys that rule of thumb (it notably starts in A flat major). You can trust single-movement music that is labelled with its key to actually be in that key, but you cannot trust the key label of any movement after the first movement. For example, the 2nd movement of Beethoven's Symphony No. 5 in C Minor is in A flat major, its 4th movement is in C major, and the 3rd movement of Tchaikovsky's Symphony No. 6 in B Minor "Pathetique" is in G major.
In practice, as a performer who does not improvise live, you don't need to care what key the music you're playing is in as long as you get all the notes right. As a composer, you need to care more about what key your piece is in, at the very least because you're probably notating your music and you want your notation to be understandable. If you're improvising a solo live on a given chord progression, you don't need to care about what key the chord progression is in as long as you stick to the notes of each chord † - and if you opt for the chord-scale system when you improvise, you still don't need to care about what key you're soloing in. If you're improvising a cadenza live for a concerto, you need to care about which key you're currently playing in because you need to eventually arrive in the movement's home key. If you're improvising ornaments live for Baroque music, you need to care about which key you're currently playing in, if only to know which neighbour tones of notated but ornamented notes to use in your ornaments.
† https://www.berklee.edu/berklee-today/summer-2000/Chord-Tone calls doing this when soloing the "chord-tone method", a more restrictive rival of the chord-scale system.