Sound processing - get one track per instrument/voice automatically

Question

Given a recording of a song, is there any (automatic or not) way to process the recording so that I get one track per each instrument/voice separately?

Does related research exist? If yes, how well do the existing methods work?

Answer 1

It can surely be done. In fact, humans are quite good at it. We have no trouble extracting the lyrics from a song and reproducing them without music. We can reproduce a melody even if we've only heard it mixed in with other parts. We can, without confusion, converse with people while music plays in the background and simultaneously isolate miscellaneous sounds like a door bell, even in a crowded room with many other people talking. Finally, though more difficult, musicians can listen to a song and transcribe all the different parts. Our brains manage to separate these audible signals, despite none of them being available as individual tracks.

Evidently, it is not impossible, but this is a task at which humans still outperform computers by far, so you'll find it difficult automating the process.

If a stereo recording is available, we can decompose it perfectly into at most two separate mono tracks. If we know how the individual parts were panned, or can make an educated guess, this may be a satisfactory way of isolating some of them. This technique is commonly used to remove vocals; it assumes the vocals are panned center and the rest is not. For all but the simplest cases though, the result will be an badly sounding mix (remaining parts that were panned further to the sides will be louder) with artifacts where the vocals used to be.

Another method relies on individual parts being in different frequency ranges. The parts can then be isolated with a simple bandpass filter. The results can be excellent if there's little overlap in the frequency domain, but even if the instruments' pitches are far apart, their harmonics may still overlap. Other parts will then bleed through and sound unnatural due to what is essentially an equaliser pushed to extreme values.

These are simple tricks, but they only work well in ideal cases. More akin to how the human mind tackles this problem, is the subject of ongoing research called 'blind signal separation'. It is not restricted to audio, but about the more general problem of unmixing two mixed signals with no information on either source. There are many different approaches to this problem. A common theme is imposing constraints on the resulting tracks. For example: if you know what a saxophone sounds like and that it can only play one note at a time, a track with two such notes sounding simultaneously has not been properly unmixed. Another common theme is finding a decomposition so that the individual tracks are maximally unlike one another, e.g. some measure of similarity between two tracks will be larger if the same violin part can be heard on both, than if it had been properly unmixed.

In conclusion, you can unbake a cake (a little). Unfortunately, to the best of my knowledge, state of the art is nowhere near advanced enough for many musical applications.

Answer 2

I have previously referred to this as "The holy grail of the misinformed".

It cannot be done. You cannot un-bake a cake.

You may have some small success at removing/isolating anything in the centre by phase-reversing the two sides... but then, of course, you have the sides out of phase, which is to say the least, uncomfortable to listen to.

See Sound Design SE - Getting Multitracks from a file for legitimate [& otherwise, with caveats] methods of acquiring multitracks of existing songs.

Answer 3

I'd have thought not, given that there may be several tracks which are panned the same as each other. Even eliminating the voice, when panned centrally, doesn't do the job completely, so isolating each track - going back to the original separate tracks - needs just that: getting the original tracks.

Answer 4

The best way to get the individual tracks is to contact the artist, the artist's agent, or the publisher, to see if they can be released to you. You will need to do this anyway if you intend to create a derivative work (which must be done under license).

If you wish to "hack" the tracks out of a recording without permission, there are a couple limited things you can do.

1. Some tracks might be isolatable due to the way they are panned. For example, lead vocals are often panned center, so you can remove them by putting the two channels 180 degrees out of phase, causing an effect known as phase cancellation. You can isolate the vocals by subtracting the resulting track from the master mix. This is how software like Audacity works. You do end up with some artifacting, since vocals are usually subjected to stereo effects which make the phase-cancellation approach imperfect.

2. You can hire a musician or composer who can transcribe the song and write a score, then hire musicians to perform the tracks individually, and an engineer to record them. This is how knockoffs are made, and you don't need to get permission to do it from the artist-- you can get what is known as a compulsory mechanical license.

3. If you just need a limited portion of a track (e.g. if you are trying to make a sample of a short riff which you intend to loop), you might be able to get satisfactory results using a band pass filter that brings out the frequency ranges used by that individual track. You could also use a parametric equalizer with an aggressive Q value. These approaches often don't work very well if the track is mixed to blend in with other tracks, since they will share frequency ranges.

Currently there is no software available that will automatically tease all the tracks out for an entire song, since this would be somewhat akin to unbaking a cake. Maybe someday there will be AI that can do it, but nothing exists in 2017.