I am trying to self-learn how to mix music and while searching I have found that a lot of people talk about mud and resonance in the sound, why do they mean by that and how do I identify those. I realize it is not a trivial process and cannot be answered here completely but I am just looking for some guidance on how to get started.
I'm not sure if it's a correct board to post this question, but anyway...
"Mud" usually refers to too much low frequencies and/or too scarce high frequencies. You should correct it per instrument + sometimes in whole mix with equalizer. Resonances are hums on certain frequencies than can be caused by different reasons. For example when you record acoustic guitar you can have resonanses in some frequencies because your instrument and/or room amplifies too much certain frequency. You can remove resonances with equalizer by turning volume down near the resonating frequency.
When mixing you should always listen to all the mixer channels individually (solo mode on track) to focus on the specific instruments. Add graphic equalizer to the effects stack in your DAW. Good equalizer should show you which frequencies are played. In FL Studio the default EQ looks like this:
Next step is use one of the equalizer steps, give it high selectivity (so you will boost only small range of frequencies) and swipe through the frequencies boosting them. Carefully listen where sound is especially unpleasant - if you find such places these are resonances. You can use equalizer to turn the volume down of them. (You first boost to identify the problem and then set the amplitude down on that frequency range to fix it).
Muddy sound is usually caused when too many tracks occupy low frequencies. Sometimes when you record you will record unneeded frequencies (super-low hums from the room etc.). Listen to each track on solo and refer to frequency chart (for example: HERE) to know which frequencies you should cut out from the mix for the given instrument. For example, as you see on the chart, Trumpet does not generate any sounds lower than ~150Hz so you can use high pass to cut everything below that frequency. (The same goes with very high frequencies: you can use low pass for Trumpet track on frequencies over 10kHz). When you will do this for every track in the mix on solo and then switch back to full mix, it should suddenly become less muddy and more clear. You will certainly need to finetune a bit ;-).
Mud is almost always around 250 Hz. Widely it might be anywhere between 100 Hz and 500 Hz, but usually there's good content between 90 - 150 Hz and 300 - 600 Hz. For some reason, 250 Hz is one of the only magic frequencies where cutting it a little bit tends to clean up the sound on almost every mix.
Finding resonance is different, and usually each instrument has its own "resonant" frequencies. By "resonant" here, we're not really talking about the physical principle of resonance, we're just talking about frequencies where the track is naturally emphasized.
Here's how you find the "resonances" or "where the track lives": Put an EQ on it, solo the track, and set up a band pass filter with a fairly tight Q (greater than 2, I'd say). Boost it by 9 to 15 dB and start slowly sweeping it up and down. There should be some places where the extreme boost really stands out for the instrument. That's an important frequency for that instrument. If you repeat the process, you may find between one and four different frequencies for one instrument. A lot of times, when you find a frequency you might find interesting effects at double or half the frequency also, or at four times the frequency also.
When you've found these frequencies, what you can do with them is boost them to help the instrument come forward or cut them to send it backwards. At lot of times, when working on a dense mix, you need to give each track it's own "place to live". Slightly boosting the key frequencies of each track can kind of give each track it's own place in the mix, as long as each different instrument has different frequencies.
Another thing to do is find the key frequencies for the lead vocalist and give a gentle boost to those, and then slightly cut those frequencies across the rest of the mix. That helps the vocals sit on top.