I know that classical, or Western artistic music mainly uses traditional chords e.g. major, minor, dominant seventh, augmented, suspended and diminished. But, Jazz goes way more into it..Major seventh, minor seventh add2/9, add6/13, suspended chords, etc. Classical and Jazz musicians would argue over the nomenclature of a chord of C, E, G, and A as being an inversion of A, C, E, and G. Even eliminating certain notes in the right hand and adding them to the bass can give it unique textures. I know that theorists sometimes talk about passing tones, non-chord tones, auxiliary notes, and borrowed chords, but I don't know if any of those relate to improvising. So, do Jazz players take piano lessons just like a classical player would? Or, do they learn more about blues, and practise improvisation exercises? What things can I do to increase my improvisation ability with what I already know about chords and bass?
I'm not 100 % sure I understand what is being asked, but the subject is close to my interests. I don't get my living from music, I don't call myself a pianist, let alone jazz pianist, but I think I have several of the skills the OP probably means with this question. (Not completely sure though.) Many "properly trained" musicians, both classical and pop musicians, have said they admire and would want to do what I can do, but they have never been taught how, and it remains a little mysterious to them. For some, liberally changing and toying around with any aspect of a song like that feels even a bit scary. A few people have at first even reacted a bit like "Is that possible? How can you do that? Are you allowed to do things like that?? There must be a rule or law you're breaking, because you're not playing the notes and chords that are written here!". (Knowing many professional musicians as well as music teachers, and having followed the "proper" training of family members and friends, I have a guess as to why that is. But that's a different story)
There are actual trained musicians on this forum, so feel free to totally disagree with everything I say. ;) And sorry for the too-long-won't-read text.
The most important thing is, you have to be able to very fluently play songs by ear (or rather "from your heart" if that makes sense) in terms of melody, chords and rhythm. Or actually, chord roles. I'd go even further: when playing anything, you always need to know what you're playing in terms of melody, chord roles and rhythm. As a warning example, there are players who can't follow what's happening in terms of chords, so they play scales, which is usually just awful. Never play scales, unless it's some sort of modal jazz (which I know very little about), or if the whole scale works as a chord, like if you sound all the notes at the same time.
You need to know your melody, chords and rhythm not only like, "this is what's written", but you need to be intimately familiar with the chord roles, so for any chord you instantly see several alternatives and variations. If you intend to improvise, you never play directly what is written, as in "from eyes to fingers". Instead, you read (or listen to) the song and understand what you read (or heard), in terms of melody, harmony and rhythm. And then you play from that understanding. If you're reading music, it has to be from eyes to heart, from heart to fingers. Even when reading, you do not just pronounce it, you re-tell the story in your own words. You might accept the written words as such, if you can't think of anything better, but you need to understand the story so that you'd be able to paraphrase it, if needed.
For example, when you have a C7, you know you could play Gm7, Gb9, Edim7 or one of many other possible substitutes - if the C7 is in a role that just leads to an F chord or something. You have to have an idea of the role or possible functions of each chord. It doesn't have to be "the correct idea", it's just an idea. Just pick one. There is no one "correct" interpretation, because harmony really consists of one or more sounding notes and expectations made possible (i.e. implied) by that bunch of sounding notes, coming from that particular harmonic context (what happened before that point). So, forget about "correct". Not as in "politically correct" or "socially acceptable", but as in "the one and only correct answer in an exam or otherwise the teacher punishes you". Music is not a school exam. You might offend people by playing or implying the wrong chords for a well-known song, but that question belongs to the social acceptance domain.
You must also control the "superstructure" of the song, so you could turn the whole chord progression around in several ways, even with (temporary) key changes, if you like. For example, for a melody that consists of several bars of only the note C, you should be able to create any number of alternative chord progressions to accompany it. For example: C - Bb11 - Am7 - Abmaj7 - G11 - Gb9b5 - F9 - Dbmaj7 - C.
I do not improvise chord progressions like that in terms of chord symbols, I see them on a piano keyboard. My quick-and-simple recipe for things like that goes like this:
- (1) Keep the melody note in place, usually as the topmost note or at least a well-pronounced note.
- (2) Start with any chord you like that fits that note and tonality.
- (3) Select a harmony note to start moving, the easiest choice is the bass note (as in the example above).
- (4) Move the selected note step-by-step to some direction (step size is up to you)
- (5) Fill the rest of the space with as many notes you think are needed to do sensible chords with, and that create the effect you want. Note that you don't have to explicate all the notes you thought of. It's sexier if you leave something to imagination and don't show everything! ;) For maximum jazzification, the chords can be from other keys - there are many possible modulations to choose from, one of the most common ones being three semitones up or down, i.e. switch between major/minor. In the example above, I hope you can feel how it's temporarily changing keys.
So. First you understand what you're playing. Not in terms of micro-level things like exact notes, but in terms of larger abstractions and macro-level things. To you, the song must not look like just a lot of trees, it must look like it consists of forests, or a hundred truckloads of logs or something.
The best way to get this first skill is to play all kinds of songs, lots of songs, by ear. Transcribe things. Read music. Then play it by ear in different keys, which is supposed to translate and replace, in your mind, the absolute pitches and chords with chord (and note) degrees relative to the tonal center.
After you have a higher-level understanding of what you're playing, then you start changing and toying around with the macro-level things. I call these changes "tricks". You always apply every trick you learn to a number of (applicable) songs you know, in different keys. Of course, you must have a taste and sense of what the tricks do, and if they feel sensible. There are lots of nonsensical tricks you could do, and to some degree it's a matter of taste, but if you can't develop a taste, then no amount of logical rules can teach you that.
Improvisation can be slow or fast. Composing and arranging could be called very slow improvisation. To improvise chords fast enough to be totally live, you have to practice it a lot. For myself, playing feels more meaningful if I imagine it's a gig - or if it's an actual gig. ;)
So, last but not least, and this applies to learning all kinds of music skills: make your playing serve a purpose. Make stuff that matters. You won't learn much, if you don't think it matters. Get as many gigs as you can! Make recordings. Publish your stuff on Youtube or Soundcloud. (The more difficult part may be getting anyone to actually listen to it, but everyone has that problem - on a gig they'll have to listen... or leave or kick you out, or both.) Dance band, wedding band, entertainment, church, your family meetings, anything. Myself, I've done countless hours of accompanying songs in church and similar situations, and also some but not as much, on various entertainment gigs. (Luckily my income doesn't depend on that, and that was a career choice. "Don't quit your day job".) Usually, nobody cares what chords you play, as long as it supports the situation. So if you know a trick, apply it! If you want to learn to improvise chords, don't contend with playing what's written. There has to be a bit of rock star ego involved, so you're like "look what I can do, can you do better??". Don't be shy.
What comes to the "unpredictability" aspect ... I'm still not entirely sure what that means, but I guess it's the thing that makes you go "I love the way these notes make me feel, but I cannot explain how". So in other words, that's a holistic music experience - when you cannot yet see the forest from the trees and it feels mystical. I think that what's happening is not magical at all - there are just more tricks applied simultaneously than you can digest. For a simple example, play a C13 chord with your left hand around middle-C, and then play some notes an octave or two higher with your right hand as a melody. Then leave out all other notes from your left hand, leaving only Bb, E and A, and play the melody notes again. It still works, even though you don't even have the bass note anywhere - and this might baffle someone. Is this something like the "unpredictability"?
Edit. "Unpredictability" in the question's title was edited to "make it sound more complicated than it actually is". That's a very good description of what I meant with applying multiple tricks simultaneously. You as the player (i.e. live arranger) keep your leading idea of what's happening, but you obfuscate the most obvious signs behind layers of tricks. And then it sounds "jazzy". The plot of your story is still there, but it's less clearly explicated. More is left for the listener to figure out or imagine.