I used to think it is necessary to at least follow the chords of a scale until I came across the Nirvana song "Lithium" and I realized the notes and chords he used in the song don't fit in any scale! so does that mean its OK to do that? chords he used are : E5, G#5,C#5,A5,C,D,B
You can use anything you like, as long as it sounds good to you. You can use many scales or not use any; you can use chords from some scale or use chords outside that scale.
Just experiment with the theory knowledge you have and you'll see that the rules are made to be broken. You can use them when you want (or need) to, but when you are writing a song like Lithium, you can use anything, and I mean anything you like, no matter how theoretically 'wrong' it seems.
As the other answers have correctly pointed out, you can do what sounds good to you. But this might leave you with a feeling of not knowing where to start. That's why I would like to let you know the little trick used in Lithium and zillions of other songs: you can mix the chords from the major scale and its parallel minor. In the case of Lithium you have chords from E major (E, G#m, C#m, A, B) plus some chords from E minor (C and D). The chords from the parallel minor key give some cool minor/bluesy/... vibe to a song in major. As soon as you understand this principle you'll realize that it's used a lot in almost any style.
Theory is not a set of rules to be followed or broken. Theory is a set of explanations for why things sound the way they do. As a composer you use theory to help inform your choices, but it never dictates anything.
First, you play a toy xylophone, and it has 1 octave of the notes in a major scale. And you can play a lot of nursery rhymes.
Then you pick up a recorder, and there are some more notes in between the ones you used to play, and now you can do more - sometimes it sounds dissonant when you pick a random note or miss the one you meant to play, but you are now able to explore minor keys. And you can span more than an octave; you could hit the wrong D, but you also have the freedom to go higher or lower.
You learn some guitar or piano, and at first there seem to be so many wrong notes that finding a combination that sounds good is hard, and it seems like people know secret chords and combinations. Slowly you find out how chords work, and how you can transpose a 1-4-5 sequence into a different key, or change a major tune into a minor one, and you have the breadth to fit chords and melody together. It can still go wrong, but the is so much more you can do now, your skill and your opportunity have risen together.
And you learn 1-4-5, and 2-5-1 and minor thirds and harmonies and it's great, but pop songs and soundtracks seem somehow more complex, cooler.
And then one night you see Jools Holland on his music program and he plays some boogie woogie and it's fun and jazzy and all that and then he's sort of mashing the keys, pressing four or five keys together in a row, like you would if you were 5, but amazingly it sounds good, it sounds great! What is this? It's not carefully found combinations of notes in particular combinations, it's not grace notes or suspensions or something you have a name for, not even those augmented chords that don't quite sound right, he's just hitting all those keys together and it's working and he's not even reading it off a score and how can you make up things like that?
And then you're sitting alone in front of a piano and you look at the book of jazz that you bought and it's a mass of dotted notes and sharps and flats and you realise that this stuff would be almost impossible to compose in some traditional way by harmonising over chord sequences and ABABC patterns and it's like you were back at the beginning and there's some secret power that you don't have.
But you read somewhere about learning the jazz scale, and learning it in all the keys, and you learn a little 12-bar thing, and when no-one's looking you try some extra notes, or a new little melody while you keep the bassline going and it slowly becomes less like a secret you don't know, it's more like a landscape you can see. You play mostly the notes you used to play, and a lot of the chords, but you see them as the root of something else, a platform to build on, a boat you can jump off at any time to idle chromatically through the keyboard or the string, hearing little momentary harmonies that fizz and sing from your fingers or your voice even though you never tried to make them.
And then it's 1 in the morning and your headphones have superheated your head and your flatmate is asking you to maybe go to bed because the furious clattering of the keys of your keyboard is keeping him awake and you realise that you weren't even there trying to play, you were kind of watching your fingers, surprised at what they were doing and the speed of their dance.
There is no real boundary, no set of things that makes all music. There is always more complexity of harmony, rhythm, mood, timbre. You build it all up, and then you knock it all back; you can be writing complex jazz harmonies and then delight in playing a child's xylophone. You can write a symphony and then a solo voice with acoustic guitar.
TLDR: Necessary for what? Music is an exploration, this is one of the thresholds of a new valley.
The beauty of music is that you can essentially do whatever you want provided you think it sounds good!
Having said that..
In my opinon, it's advisable to know the rules before you break them. Some might argue this will contain you and force you into an uncreative box, but I argue that you can truly break the rules of music theory once you learn them. So if you don't know much already, then search around the internet for Music Theory information, tutorials, lessons, etc and then once you begin to understand it, you can choose to throw the book out and write whatever and however you want.
Before music theory existed (in a form that was shared with others) there was music. When humans first began making music with their voice or with instruments, some of it sounded pleasing - some not so much.
At some point, the scholarly among the early dwellers of the planet, decided to try to figure out why certain notes sounded good when played in succession as part of a melody and later - why certain notes blended together to make a pleasant sounding chord (even though the word chord was not yet coined). Out of this type of analysis, music theory evolved.
So music started out being composed in the absence of theory, and intuitively it was shaped into arrangements that sounded pleasing or evoked the desired emotion. Theory came later, to explain why a certain arrangement evoked a certain emotion, and thus also provide a framework to help future composers replicate similar compositions.
But in the beginning there were no rules. Today there still are no rules if one chooses to exercise their own creativity in innovative and new ways. Of course when someone does that and it works, someone else will add a new chapter to the theory book to explain it.
If you seek to compose your own music, and you wish to be inventive and create something truly unique, original and completely outside the bounds of current music theory, feel free to be the author of the next chapter in music theory. Don't feel constrained by what others have done before you.
Absolutely not. Music is an art form. You can do whatever you want (and whatever you can imagine). The theory is there only to help understanding what you are doing; as a guide, if you need one. It must never limit your creativity. If still in doubt, ask someone who plays jazz :)
No, obviously, as you can hear in the example you stated, but generally when you transition between two notes that don't belong in the same scale, most people can "hear" the transition, (as you can also hear in the example you provided). It's the moments when it feels like the whole song shifted up or down (even if it's just for the duration of one or two notes).
Also, it's not necessary to use any "official" notes at all, example: any dubstep - many of the wobbling basses and more complicated lead sounds have hand-tweaked soundwave frequencies, which means they often end up not being any "standard notes" and you can still create harmonies even when working with pure frequencies outside of any "note" bounds. It's just usually hard, and many of those harmonies sound strange and exotic, or even disharmonic (because once you go outside of what human brain is accustomed to as normal, all bets are of and good/bad becomes purely subjective).
To expand on the comment by 200_success, no, it's not necessary to restrict yourself to the notes of one scale when composing. Often when you see the sharp, flat or natural 'accidental' symbols (♯, ♭ or ♮) in music written on a stave with a key signature, a note that is not part of the scale of the key is being used. This is an extremely common technique.