Matthew has provided a great answer. I want to expand a bit and look at different aspects of your question to help with some misconceptions that might be causing trouble.
I would take a professionally produced song with a similar set of instruments that my song uses (or genre) and compare the frequencies through the voxengo span vst... ...Even though the frequency graphs are very close, the qualities of the sounds are way different.
While it makes some sense to try this analytical approach to improving your mixes, it's just not going to work at all. First, the instrumentation is such a tiny piece of how a mix sounds that from a mix perspective, there's no reason to expect there to be any useful similarities between the material you are working on and the material you are comparing it two.
Second, no spectrum analyzer in the world has even one one-hundredth the resolution, detail, and analysis capabilities of even an untrained human ear. Basically, a spectrum analyzer is almost useless for making a mix sound right. That's why professional engineers spend tens of thousands of dollars on speakers and not much at all on spectrum analyzers. The only really effective analysis tools are on the sides of your head. The best thing is, they are free.
I would use eq on my finally bounced mix
This is good. EQ is one of the most important tools at all parts of the music production process, especially mastering. Only learning to EQ effectively will do more to improve your mixes than any other single thing you could learn.
bounced after master buss fx things are done, like the EQ, Analoguer, widener, compressor, expander, limiter etc
This is not so good. The analoguer, widener, and expander are almost certainly going to make your mixes sound worse and not better. Especially if you are mixing music. Most of these tools are meant to solve problems, but in solving those problems they create other problems. Usually the trade-off is worth it since the original problem is usually bad enough that the new problem is acceptable in exchange. It's highly unlikely that your mixes have the problems that are solved by "analoguers", wideners, and expanders, and if your mixes do have those problems, you need to fix them in another way (before they get to the mix buss). If your problem is mud, then cut these three out of your signal chain and never use them again.
Compressors and limiters are useful tools, but keep in mind that along with EQ, all of these tools do create problems when you use them. That means you want to use them as little as possible. If your mix doesn't sound full and clear when you throw the faders up with no plug-ins at all, then your problem is your recording process. If it does, then you're just looking for that extra 5% - 10% of improvement that you can get from careful and experienced use of EQ and compression (and reverb and delay). And everything you do to get that extra 5% - 10%, can take away from that initial fullness and clearness, so you have to be very gentle and back off if it doesn't actually sound better overall.
Compressors are the second most important tool in the engineer's toolbox (after EQ). Compression and EQ often require years of practice before an engineer knows how to use them effectively. That practice is going to involve EQing and compressing things gently and listening to the change in the sound. Only by listening critically can you develop your ears and create better sounding mixes. If you're looking at things while you're mixing, you're only distracting your brain from the important work it needs to do.
I do not understand how I am generating the mud in the low-mid frequencies though I try to get my kicks and basses separated and low-cut other things. And my highs do not sound high enough even when the frequencies above 5k-8k is boosted.
This probably won't make you feel any better, but everyone deals with exactly these kinds of problems when recording, mixing, and mastering. At each part of the production process, it's not too hard to take away something that's there and sounds bad, but it's usually much harder to add something that's good that's not there.
If clear highs are not coming out of the guitar amps and off the cymbals and drums, well you're not going to pick them up with a mic. If they are in the room but the mic is set up wrong, you won't get them recorded. If you get them recorded but you end up cutting them or compressing them out in the mix (yes, compressors mess with your highs and lows, that's part of the trade-off), then they won't be there any more when you want them at mastering. And if they aren't there when you're mastering, you pretty much can't add them back. Mastering cannot add very much at all. It can enhance what is already there and it can take away. Take away too much, and you can ruin things, so mastering is a very fine art that can be extremely challenging.
It's better to focus on making sure the instruments sound bright and not muddy in the room, then make sure you're capturing that sound in the recording. Then enhance and don't take away from that sound when mixing, and then when you get to mastering it will be pretty much good already.
In terms of cutting mud, if you can't go back to recording, you can do a little to cut the mud (since it's cutting, not adding). First, once again, you have to use your ears to tell you when you've cut enough but not too much. No plug-in has a "mud light" that lights up when there's mud and then turns off when the mud is gone, but your ears hear it right away and they will know when you've taken it out.
Mud usually somewhere between 150 and 300 Hz. It can be as high as 400 Hz or so but usually you want to start right smack dab at 250 Hz and try cutting that gently.
Here's the trick, though. You can cut all the 250 Hz right out of the mix and you definitely won't have any mud. The problem is you might not have any warmth or thump either. So you have to start cutting and listen for the change, and then slowly cut more and more until the mud is gone, and then listen and see if you still have thump and warmth. If you've still got them, then you're done and you got rid of the mud. If they are gone or not pleasing you any more, well ideally you'd go back to recording and mixing and see if you can get rid of mud as early as possible. Otherwise you have to spend a long time playing around with frequencies and different amounts of cuts to see if you can find a balance between mud, thump, and warmth.
Now you're thinking "Hey why can't I just boost the thump and warmth while I cut the mud?" If it's not already clear by now, trying to boost thump and warmth will pretty much end up boosting the mud back in again, and cutting and boosting the same thing just makes a huge mess. When mastering, it's much more common to cut than it is to boost.
Pro mastering engineers make most of their adjustments at a level of only +/- .5 dB, sometimes going as high as +/- 1.5 dB. From the standpoint of a mix engineer, those are almost inaudible changes, but in the master, it's a lot more sensitive. If you have to cut 3 dB or 6 dB to get rid of the mud, that probably means your mix is bad.
What is that I am doing wrong?
- "Fix" as much as possible when recording, then try to do as little damage in the mix as possible, and your mastering will get a lot easier.
- Use your ears and brain to analyze music, don't use your eyes.
- Use hardly any plugins. Make the sounds sound good right away, and then don't mess up the goodness coming right from the instruments. A little bit of EQ here and there, a bit of compression on the bass and vocals, and some reverb to spice things up. Get really good at just those three effects and you'll stop wanting any other plugins (ok maybe delay). Your favorite mixes probably have none of the crazy plugins that plug-in makers put in their "mastering suites". Some of those people are just trying to get your money. Others are trying to give you a lot of good tools just in case you need them, expecting that you'll hardly ever use half of it, which is why they threw it in with the EQ and compression anyway - because they'd never sell it on its own.
- Expect to spend years practicing and mixing and hating your mixes before you start to sound good. Pro engineers have decades of experience. The guy who mastered Taylor Swift's 1989 had already been mastering for ten years when Taylor Swift was born in 1989. They're not pros because someone gave them really awesome tools, they were given awesome tools because they are pros. They are pros because they have tons of experience and training from other pros.
Response to comment:
However, is an AKG K44 something I should let my ears trust?
No.
Unfortunately, you can't trust any pair of speakers or headphones. Not 100%. Every system has problems, and even if you magically had perfect speakers (BTW, speakers are generally better than headphones for monitoring for many reasons), your room would cause problems with the sound coming out of the speakers and so you will always be hearing something that is a little off.
So what do you do about that? Well once again it takes practice and experience to learn how your monitors are lying to you. Once you understand the limitations of your monitors, then you can compensate. The way you learn is by listening very carefully to your monitors and then compare them with all kinds of other speakers.
For example, I once had a set of monitors where I would mix the snare to a certain level, and then when I listened to my mixes in my car, on headphones, on my regular stereo, etc., the snare always seemed way too loud. So basically my monitors were lacking in an important snare frequency. So I had to make the snare sound "too quiet" on my monitors so it would sound right everywhere else.
Finding "good" monitors is not as important as learning how your monitors actually sound compared to where you want your mixes to be. Again, this takes a lot of time and practice. The more you read, research, and especially practice, the better your mixes will get.
