It's all about practice and experience. You have to train your ears, you have to know your tools. Producing is not the same as mixing or mastering, some skills overlap but they need mostly different skill sets.
There are tools and techniques that will help you judge the frequency balance of a mix. You might eventually not need them at all, though some of the best mixing engineers keep them in their daily arsenal.
- Listen to reference songs. Pick a set of professionally produced songs that are similar to whatever you are mixing. Look for songs that have a mix that you really like. Play your mix and the reference song separately and compare. If you are too high on lows, mids, highs, some instrument is too loud or too soft or whatever, do adjustments.
Use visual aids. This is specially useful if you don't trust your listening environment (ears, speakers, room, etc). This is most likely the case if you are not an experienced mixer with expensive monitors and acoustically treated room. You can use an spectrum analyzer to see what's going on in your mix. If you don't know what shapes and proportions to look for, use the spectrum analyzer in reference tracks to check roughly which forms you should aim for. Logic and Live have built-in spectrum analyzers in their native equalizers, if your DAW doesn't have one you can get FabFilter Pro-Q 2. There are many other visualizing plugins out there, so you might want to try many before making a choice.
Listen your mix through many different systems. Every part of your listening environment (room, speakers, equipment, etc) will color the sound (accentuate the mids, attenuate the highs, or whatever), so you want to listen your mix in different rooms, with different speakers. Try stereo, car, earbuds, etc. Find a good middle ground where your mix sounds good everywhere (at the same time understand the limitations of each system, lows will be lacking in small speakers and that's perfectly normal and fine). Reference tracks are again useful here, to give you an idea how each system should sound like.
These can be of great help, even to experienced mixing engineers, but are specially useful if you are just starting.
They often tell me that my mix sounds weird and "muddy", but I myself really can't tell what is wrong with it.
If you can't tell what's wrong with it, maybe it's because different tastes, or maybe you are not very sensible to the low frequencies (ear damage of some kind), or maybe you don't have a subwoofer and the other person does so you can't even hear muddiness he is referring to, maybe his room accentuates the lows. Who knows?
Again, your problem here is lack of experience. The solution is getting experience, so you are on the right track.
Mixing is both a science and an art. If you want to get philosophical about it, there's no such thing as a weird, bad, or good mix. The ideal balance is subjective. But if you are mixing for the masses, you want to take their tastes into consideration too.
I understand that to make a proper mix: You need to "reserve" frequencies on the band to make that instrument sound clean. But what If I have two instruments that lay on the same frequencies (And that happens a lot in my mixes since I produce Hardstyle)? How do I go about that?
There's nothing inherently wrong with having two elements in the same place and time (violin ensembles can sound beautiful), but if they are not working as a unity you usually want to separate them, otherwise it can turn into something similar to many people talking at the same time.
This has little to do with mixing and more to do with production, composition, and instrumentation. The art of when, where, what, and how to put stuff doesn't have a blanket answer for "how do I get about that?". Maybe you want to consider separating elements in time (so they don't sound at the same time) or in frequency (they use different octaves) or tonally (use another instrument, or tweak the timbre of the current one).
There's little you can do in the mix with instruments that clash in time and frequency. You can try some separation with panning or eq and filters, but the results are limited. If you think two elements clash too hard, you want to take your mixing hat off and put your composer hat on. Maybe that idea doesn't work as well as it did in your head, and needs a change in the composition level, not at the mixing level.
Can anybody recommend any good source to really learn this? I have several books on this, and I've read them throughout. But I still can't differentiate between a bad mix and a professional mix with my ears.
Reading is invaluable, but will do nothing for you if you don't practice using your ears.
- First of all, pick an ear training program. I recommend you the Golden Ears program. Train daily.
- Mix, mix, and mix. Mix at the very least one song daily.
- Don't stop studying! There are great resources out there.
In short
There are many subtleness in sound dynamics, and training your ears to recognize them takes time and practice. Accurately judging the scenario takes knowledge. Practice, practice, and practice.
If you want to get serious about mixing, you want to get an audiometry. Test your ears and see if everything is ok so you can be sure any misinterpretation is because of lack of practice and not because something like ear damage.