I want a theoretical framework to know WHY or HOW these chords "work"
This question is often asked, and the answers vary. They might be something like, "because they have 2 notes in common" or "it borrows from such and such key or scale" or "it creates dissonance or consonance" or "it is expected/unexpected" or "there's a leading tone" or "it's a tritone substitution" or "it comes from the diminished scale" or "because blah blah on the circle of fifths" or "look at the steps in the voice leading", "the fifth is omitted", Etc. Check out Barry Harris, he talks about scales and/or chords having a love child, producing offspring... it gets a bit over my head. There are many possible perspectives and observations you can make, but usually none of them is the ultimate "why" answer for all intents and purposes. Several perspectives can be equally valid at the same time, depending on how you're used to looking at things.
If you can locate and reproduce a chord in relation to the then-prevailing tonic, then you know a lot already. If you can reproduce the same chord in other keys, you know even more. Is that not enough? Maybe not.
Why a chord "works" is not the right thing to ask. You should ask, what does it do and what other chords could you use to accomplish essentially the same things in that context. The answer to the "what does a chord do" question comes in the form of multiple perspective-dependent parameters or dimensions.
- What does it do to my perception of other plausible intervals around the tonic at the moment?
- Does it perhaps seem to even move the tonic?
- To which role in terms of V (left) - I (center) - IV (right) would you reduce the chord's effect, if you had to substitute it with the most basic triads? Would that even be possible, or would something essential be lost in such a reduction?
- If there's a V - I sort of motion going on, where would the target "I" be?
- Where is the bass now?
- What is the highest note?
- What's the essence of the chord, does it feel like some inversion of a basic triad, and if so, is it a major, minor or diminished triad, and which inversion is it?
- How's the voicing, the spacing of notes - is it open or closed, and what could you turn it to by moving one voice by one step in some direction?
- Do I know this same pattern from somewhere else?
- Could I turn this into a well-known progression with a few chord exchange operations, for example Am - Em - Dm - E --> Am - G - F - E (Hit the Road Jack).
Demanding all of that from a beginner feels very hard or impossible. Don't demand too much from yourself immediately, you have to be able to get on, get more experience. Learn more tunes, play them in different keys, even though you don't know all possible aspects and perspectives of analysis. Nobody knows all aspects. Somehow this reminds me of the question of an adult violin student complaining about the teacher not demanding all possible aspects of playing to be perfect before moving to the next piece. You just have to move on and accumulate more music. You'll remember something of each new chord and note. Maybe that there was a major chord on the major tonic's leading tone and it felt nice going to the tonic from there. Hey, this sounds like some Elvis Presley tune! Easy to recognize, easy to remember, easy to reproduce on the guitar and on the piano. I don't feel any need for anything more analytical to say about that chord.
The thing with harmony is, it's like the game of Go, which is often compared to Chess. In chess, each piece has its own particular abilities and ways of moving, and you use these differences for classifying each situation. But in Go - so I've heard - the role of a stone i.e. piece on the table can be fairly ambiguous. Is it an offensive or a defensive stone? Can it "do" anything at all? It depends on its position, and where other stones are relative to it. Maybe you recognize the pattern in advance, and then you can classify what's going on. In harmony it is the same. Sounding notes are like stones in Go: the plot of the game is built from the combination of notes and their relations with each other. There can be many simultaneously valid "explanations". For example, a B half-diminished chord can be seen as Bm7-5 or Dm6/B, and it is both at the same time. A G7 chord can be seen as almost a Bdim, and C9 can be seen as having a Gm triad, and an E half-diminished. You don't have to see all possible re-interpretations, or any of them. You know what notes there are in each chord, what notes you expected in your "current scale" and if the chord was against the expectations, maybe making a chromatic alteration, modal mixture or even a modulation. You don't have to get it "right". It's enough to be able to reason about the notes even on some very simple level. Your understanding of the game gets stronger by playing.
Instead of a theoretical framework, I offer you a practical one. Learn to play tunes as melody and backing chords, in different keys, and make improvised changes to both, to get a hands-on feel for what's happening. You need to play the game, not get stuck on some fancy theory about strategies.
For example, let's take this chord progression.
Am - Dm - G - Am - F - Dm - E - Am.
Play an improvised melody over that - even simply running scales or arpeggios up and down. Grope around, trying to feel where the walls are. Which scale notes seem to fit over each chord? On the first Am, can you play the so-called natural, harmonic and/or melodic minors? All of them or just one? How about on the Dm? On the G? Etc. How about on the E chord, does the E chord demand either an F or F# note in the melody, or could you freely choose either?
Let's take another one:
Am - Dm - G - Am - F - Dm - B7 - E7.
Can you feel what notes would fit over the B7 chord? How about if we make some changes:
Am - Bb - E7 - Am - F - Dm - B7 - E7 - Am6.
What happens to the scale possibilities over the Bb chord? How about the Am6, how does it change the scale?
Can you play the same chord sequences in other keys, relative to other tonics?
In this question's title you're asking for something less scale-centric, partly because of my complaining about harmonic minor. My complaint wasn't about thinking about scales as such, the problem with the harmonic minor thing was, you tried to fit it as a rigid unchangeable structure over entire chord progressions or songs, as if it was a mode. Thinking about a scale is OK, and it can be even essential, but the question should be, what could the scale be now. A single E major chord doesn't say anything about the F question: is it F or F#... or F##.