4

This question is inspired by comments to this related question:

I would be surprised unless you could find a mobile app or at least some Apple/PC SW that filters out a given instrument from the orchestra sound.

Is this possible at all?

3
  • 1
    Tuomo should prepare to be surprised, then. – Kilian Foth Oct 16 '19 at 6:42
  • 2
    It is possible to some extent, but not like in the C.S.I. TV show. Melodyne can separate individual voices, though not necessarily instruments. It's not perfect and not like in the movies though, but kind of sort of a little bit like a step in the general direction of being able to separate instruments. – piiperi Reinstate Monica Oct 16 '19 at 10:26
  • 1
    The issue with Melodyne could be the model or sample data it uses for separating voices. The filters have to have some tuning parameters. I'd think if its filters were upgraded to have realistic instrument attack + sustain + decay curves it might have a chance. – ggcg Feb 3 at 21:57
6

No it is not possible given the current state of art. Therefore recording engineers distribute microphones to have separate tracks for the instruments or instrument groups (so they can change the balance) and therefore special recordings as music minus one are required with filtered soloist voice to produce training material with full tutti voices and nothing else.

Artifical Intelligence may provide that possibility in the future, but I would not hold breath.

Note that splitting to all components is even more difficult, than to extract one instrument.

4
  • Interesting. I intuitively thought that if the input was e.g. a CD quality sound, somehow combining the following: 1) The instruments aren't perfectly tuned, i.e. some differences in the "nominal frequency". 2) Frequency profile, like amplitudes of the harmonics and normal frequency modes of the instruments 3) The fact that the members don't [regardless of from which position one "listens"] play exactly in the same rhythm 4) Cross-correlating [filtered] R and L channels to finds which frequency components, impacts etc come from the same direction. But, I guess I was too optimistic ;-( – Tuomo Oct 16 '19 at 11:53
  • Not even close to possible, since trying to pull out the overtones belonging to each instrument simply can't happen. – Carl Witthoft Oct 16 '19 at 13:24
  • @CarlWitthoft, what do you think makes pulling overtones for each instrument something that simply cannot happen? Is it 100% impossible or only possible with some corruption of data due to missing degrees of freedom? I am just curious. – ggcg Feb 3 at 19:46
  • @CarlWitthoft, I see we have similar backgrounds so I'd be really interesting in your take on this, and on my proposed answer. I don't say impossible but I say marginal at best with a distorted mess as a result. – ggcg Feb 3 at 19:47
4

No, at the present state-of-the-art, 'un-baking the cake' isn't practically achievable except in special cases like a stereo recording with ONLY one instrument being panned centre etc.

However, I would be very surprised if the military aren't interested in software that can isolate one voice from a babble. If that is successful, stand by for a spin-off.

[later]

Try this! If it can recognize a vocal...

https://vocalremover.org/

5
  • @"Maybe the brother of Max": while I guess we would never had ended up to the point where someone is walking on the moon without having a sufficiently challenging mind, I do love your cake unbaking symbolism [sorry for my non-native English!] – Tuomo Oct 19 '19 at 13:43
  • I think voice recognition in a recording is possible at present. Maybe I'm wrong on that. – ggcg Feb 3 at 19:43
  • 1
    A voice buried in "babble" is an easier task that separating out numerous coherent signals since you care about all of them, rather than just one. You can destroy the rest of the signal in the process of extracting the one and through it out. – ggcg Feb 3 at 19:44
  • Well, you've still got the original version to 'destroy' when extracting the NEXT instrument! – Laurence Payne Feb 3 at 21:22
  • 1
    @LaurencePayne, I agree but then the process needs to be done in a loop and each time different choices are made, so double checking and refining could be tricky. This is not as big of a problem if you assume the background is "noise". – ggcg Feb 3 at 21:59
1

There are machine learning models that attempt to separate recordings into several instrument tracks- notably, Deezer's "Spleeter", MIT's PixelPlayer, and Izotope's (very expensive) RX8. Researchers "teach" their machine learning program by feeding it large amounts of real multitracks. After training the model, it can be given any recording, and it will attempt to create multitracks that "look" like the ones they learned from.

It's far from perfect, and each instrument track will have some extra noise and significant pieces missing. The software can't always tell the difference between, for example, a hi-hat or a singer making a "t" sound. The result ends up sounding like a very low quality mp3. Still, it's an interesting proof of concept and will almost surely improve over time.

Right now, Spleeter can be installed for free, but it only comes with pretrained models for pop/rock-like music (Drums, Bass, Vocals, Piano, Other). If you want to separate an orchestra, you'd have to find yourself lots of orchestral multitracks to train it on.

MIT's demonstration of their model:

Demonstration of the Spleeter model:

0

That depends on how clean you expect it to be and how automated you expect it to be. Technology exists to identify specific voices in a mono recording of several people in a noisy place so I can't believe that it is impossible. But you would have a lot of heavy lifting to do and some manual labor to decide if what the filters are catching is real or an artifact.

There exists s/w, and homegrown versions are not hard to make, that can ID the separate notes present in a recording. But the next step would be to attribute them to a specific instrument. To do this one would need to ID that a specific instrument exists from among the collection of sounds. This is not impossible as we can take recorded versions of impulse responses for each instrument, or use simulated data from models and convolve this with the recording. I disagree with the comments that it's impossible, or not even close. But there are numerous pitfalls and possible ambiguity. It would be an easier task if rather than trying to prove that there is a violin in the mix of 100 instruments you were to provide a library of instruments that are known to be in the recording. Then you've reduced the effort required and the number of errors that could be made.

Even so, you will have overlapping sound from many instruments mixed together and there will be possible ambiguities and misfires w/r to identifying which note was played by which instrument. Even if you could do a crude separation that got the notes correctly ID'd and mapped to the instruments the actual sounds could be a jumbled mess as you would likely have gotten the amplitude and phase distribution for each instrument wrong. Say a violin and Sax are playing the same note and you correctly discovered both instruments based on detailed harmonic profiles. How do you determine if they are both contributing 50% to the overall sound? Also, if you found the correct peaks in a signal filter but didn't get the phase and subtracted out what you thought was the Sax the "violin" might be distorted as a result of some phase ambiguity. You would need to correct this by then synthesizing the original track using the separated signals, and modulate the unknown variables in each spectrum until you succeeded in match the original noise free. And even then you may have a whole space of possible mixtures that correctly synthesize to the original.

Again, I assert, not impossible but very costly to get a dirty result. In the end it may not be worth anything.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.