There are a number of bands, Genesis, The Police, Crack the Sky, Gentle Giant, that have all incorporated unusual time signatures as their core sound - some entire "pop"songs, "Ordinary World - Duran Duran; Money- Pink Floyd; Texas Flood (intro)=- SRV, have even made it an integral component. What is the theory behind not playing in the usual 4/4 time, because I would like to be able to write a song based on a different time signature..
While I really wouldn't consider Texas Flood as having an odd time signature (12/8 is really just a representation of a triplet swing feel over 4/4), many of the songs/bands you mentioned do use bizarre time signatures for effect.
Let's take Money by Pink Floyd as an example. It is in 7/4 which is nicely outlined by a strong bass line. The rhythm for the bass line is approximated with:
| 1 2 3 4 5 6 7 | | Q E E Q Q Q Q Q |
Now, the easiest way (I think), to break this down is into two chunks: one 3/4 and one 4/4. Now, the order isn't as important when composing because 4/4 + 3/4 is effectively the same in the end as 3/4 + 4/4. But for this tune, it works better like so:
| 1 2 3 | 1 2 3 4 | | Q E E Q | Q Q Q Q |
So we have a little snappier 3/4 section, with a 4/4 section that walks back up to the tonic which gives it a more stable feel than most strange times signatures. As with most oddly timed tunes, Money is very much riff based. Take a look at Take Five by Dave Brubeck for another good example of riffing during an odd time signature.
Basically, when composing your own tune, try to put together two shorter odd time signatures first to compose an interesting riff that feels stable and sounds interesting. Two odd meters can also combine to create a normalish meter as well (5/4 + 3/4 = 8/4 or 2 4/4).
I recommend listening to Dave Brubeck's Time Out. That whole record is an exploration of odd time signatures and unusual ways of subdividing rhythms, but in a way that still swings. Three of the more famous tracks, for example, are:
Anyway. It's a great record and a wonderful collection of odd time experiments.
Things are getting way off-topic here (although it's a fascinating discussion). The initial question was: "What is the theory behind not playing in the usual 4/4 time"?
Who says 4/4 time is "usual"? Most musical phrases and thus time signatures are ultimately based on dance rhythms. Surely you've heard of the waltz? it's in 3/4 time. The minuet is in 6/8 time. There are many folk dances of many cultures that are not in 4/4 time. Folk dance rhythms and drumming rhythms in African cultures are in multiple different time signatures simultaneously; these are called "polyrhythms".
On the other hand, the most widely recognized ancestor of all the melodies in Western culture are the Gregorian chants. These have no rhythm and no time signature at all.
The use of odd-time signatures, and frequently changing time signatures from measure to measure, came into popular Western music in the late 1800s and early 1900s through classical music pieces written by composers including Bela Bartok, Igor Stravinsky, and others.
I've looked into it, and it's my conclusion that complex changing time signatures got into rock when rock musicians in bands like The Nice, Emerson, Lake and Palmer, Genesis and Yes were influenced by composers like Stravinsky, Bartok and all those that followed them. Rock musicians listened to serious classical music and they took some of those ideas and applied them to different forms. It did not necessarily require that the rock musicians involved actually read sheet music or study it in a classroom; they heard classical music on recordings and on radio broadcasts and they decided to use these ideas when writing their own original music.
I'm a big, big fan of odd time signature. The thing is, there is not really any theory that I know of, it's all notation and counting/feeling, and notation is convention. Basically, the idea is you have a X/Y time signature, meaning you have X elements of value Y (generally 1/4 or 1/8 notes). That's about it.
While Bryan's answer is perfectly valid, my approach is slightly different : I tend to think of them as patterns in themselves, or relating to other patterns, but not thinking of them as a composition of patterns. Let me explain.
If you have a 7/8 pattern, chances are the accentuations, the general groove, is related to either a 4/4 pattern minus a 1/8, creating a syncopation. Or it could be a 6/8 beat with an added 1/8.
Sometimes the pulse is at odds with the basic element : the notation introduces the denominator as a basic element, which may or may not relate to the pulse of the music. In the case of a 4/4 "mangled" into 7/8, you probably will have a strong feeling of a 1/4 pulse with an abrupt ending at the end of the bar. This would in fact have to be written 3,5/4 except 1) there is no such notation, 2) if we were to invent it, it would only be confusing.
Sometimes, the pulse will depend on the instrument : you frequently have a drum beat in, say, 7/4, over two bars of 7/8 on the guitar, and I would not call that a polyrhythm.
Also, you can have a 4/4 measure which is in fact two odd beats put together (off the top of my head, I'd say the end of Holy Water rom Soundgarden's Badmotorfinger : 3/4 + 5/4 = two bars of 4/4), as Bryan pointed out. Once again, it's far easier to write and read this way.
As cliché as it sounds, it's all about the music. Try playing written patterns, try to find the pulse on a strange riff and write it down, play with others and I guarantee you'll have fun. I know I did and still do.
Time signatures are all about counting beats between accents. We count to a number, with the ONE having the strongest accent (this is oversimplifying a bit -- for example some reggae feels emphasise the third beat of a 4/4 rhythm more than the first).
Humans like rhythms based on an even number of beats, because we have two legs and two arms. If you listen to a piece in 2/2 or 4/4, while walking or running, you'll find yourself easily falling into step with the music, and when a bar finishes, you're back on the leg you started with.
12/8 shouldn't be considered "odd". It is almost always played such that you can count a 4/4 beat over the top of it, with the 12/8 beats being triplets.
It sounds as if 3/4 would be "peculiar" by my "two legs" logic above, and to a small extent it is. But even in 3/4, conventional composers tend to create phrases using 2, 4, 8, 16 bars. People tend to play/perceive 3/4 bars in pairs ("ONE two three TWO two three"), and again, if you walk to to this rhythm, you find yourself back on the original foot at the end of that pattern.
Now to the question, "what is the theory behind not playing in 4/4 time?"
It is simply a matter of counting to 5, 7, 9, 11, or whatever your chosen number of beats is, then starting again from 1, ensuring that you keep the beats even as you would with 4/4.
At first, try just tapping with your fingers. Thumb for the down beat, a finger for the others. At first you are likely to find it difficult. Most people, I believe, attempting to tap out a 5/4 rhythm for the first time, will feel a strong urge to pause for a beat after the 5, bringing it to a more conventional 6 beats.
It will help, of course, to count along to a recording of an existing piece, such as Take 5 (5/4), Money (7/8, 4/4 for the middle section), Radiohead's 15 step.
Watch out, also, for compositions which throw in the occasional bar in a different signature to the rest (15 Step might be an example).
Once you've got the hang of that, it's "simply" a matter of creating your own riffs and melodies that fit with one of those time signatures. The songs you've mentioned pull off the clever trick of sounding natural; people tend to be able to move their shoulders to Money without being thrown off by the time signature.
As with many things, the best way to get good at it is to try it, do a crappy job, then try it again, and keep going until you get a result you're pleased with.
I would argue that 4/4 time is not the "usual" time for music, it's just that your musical influence tends that way. The music and composer dictate the time and the feel. Listen to a seminal rock band like Rush. They didn't sit down and say "let's make a complex time piece of music to really stimulate theorists." They chose a melody and a feel and built from there.
If your compositions are pop-focused and you are specifically looking to write "hits" then there are formulae and workshops to learn that.
If your compositions are about getting music from your head into the world then go for it. Create the music that feels right. The right melodies, harmonies, and rhythms. Then you may or may not have complex time.
Depends on the genre IMO. Swing, for example, is mostly 12/8, and I can't think of a boogie woogie title that isn't. Rock music became increasingly 4/4 after the 50's, but there were lots of 50's titles that are in 12/8 (Blueberry Hill and That'll be the Day are a couple of well-known examples.) Most blues music is in 12/8 as well, especially earlier blues.