like everybody here I love listening to music, I'm also a guitarist who's been playing the instrument for over 10 years and I'm not bad at it. But sometimes I struggle to isolate and follow instruments in my head when they overlap. Most of the time I can hear a specific instrument I'm focusing my attention to and it's main rhythm but I struggle with nuances and I lose some notes when more instruments are playing at the same time (e.g guitar with distortion, drums, bass and voice). The most difficult part to isolate for me is bass, it's really hard to pick it up except some parts (for example Police songs or very rhythmic and bass driven songs). The weird thing is that I have a pretty good hear for solos and vocals because they are always sitting on top of the mix. I transcribed and played Teen Town by Jaco Pastorius without big problems. That said I have a decent equipment (studio monitor Yamaha HS7 and ATHM50x headphones coupled with a scarlett 2i2) and I always listen to lossless music. How can I improve the instruments separation in my head??
But for analysis, you can take the track and apply a low-pass/high-cut filter to it such that the frequencies above the typical bass-guitar-note range are eliminated. You will be able to pick out the bass line (and some drums) much more easily.
I have a pretty good ear for solos and vocals because they are always sitting on top of the mix
You just put your finger on it right there. When instruments are mixed together into a stereo recording, information is lost. Some of that information is what we need to hear to separate instruments from each other in our brains.
When a track is emphasized in a mix, it's usually to make sure we can still hear it separately, so that information is preserved for that track (by making it louder, using EQ, etc.).
I listened to a specific rock record the other day from a band that I was totally obsessed with in High School. I hadn't listened to that band in a long time-like almost a decade! It was fun being taken back to those moments in my life. I noticed something interesting, though. I was hearing things in those songs that I hadn't noticed before. I KNOW those songs. I have listened to them a lot. I know every note in the solos, but still, I was hearing new bits from the other instruments.
Here is my point: It takes repeated, intentional practice over a long period of time to develop a good ear. It's the same as developing your skill as a player, or anything else that is worth learning. I have grown a lot musically speaking since High School. As a musician, my hours of experience have increased exponentially since then, and my ability has improved as a result of that.
This may sound like an obvious suggestion, but give it a try. Always be a learner. The moment you decide that you have "made it" is the moment that you stop growing. It's actually even worse than that You'll get worse! You have to keep pushing yourself, or your skill will begin to atrophy.
Edit: I just read my post, and decided that it was too big picture. Here is a practical suggestion to give balance. Give a look at an app called Capo. It's available on Mac and iOS. It's a great practice tool. It lets you change playback speed, key, and filter out frequency ranges. I use it all the time for learning parts of songs more quickly! I love the selective looping functions.
Maybe spending a little time playing bass (since you already know guitar, this shouldn't be so hard) will help bring that part of the range into clearer focus. Also, going over some "how to" books on bass could help by providing more common patterns and logic that will help sharpen the process by giving you more things to listen for.