-2

From a comment on Hacker News:

[...] physical playing has been replaced by software quite a while ago. [...] almost nobody can distinguish a computer rendering from the real thing on a recording.

Can software really convert sheet music into an audio recording as well as a skilled human with a physical instrument and a microphone (as judged by the audience)?

1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – Richard
    Aug 31 at 20:22
4
+50

The question was "Can software render music as well as a skilled human with a physical instrument and a microphone?" and before you answer no, (which seems to be most people's answers here) you should check out NotePerformer which performs music scores (Sibelius, Dorico, Finale) using AI. I'm only mentioning it because no-one else has.

Sure you can tell it's not live musicians, but it's far better than the sample or GM playback that music typesetting programs generally provide out of the box.

It sounds like an actual performance because of the way it renders legato and dynamics changes using AI. Check out some of the sample audio snippets on the site.

2
  • 2
    Being aware of NotePerformer, I only didn’t mention it because it still doesn’t replace humans Aug 31 at 13:45
  • Wow! although samples can be misleading, if the same pieces were part of their training dataset (then the trained neural network would essentially copy the training data)
    – MWB
    Aug 31 at 17:11
6

Can software really convert sheet music into an audio recording as well as a skilled human with a physical instrument and a microphone (as judged by the audience)?

"Converting sheet music into an audio recording" consists of two steps: recreating instrument sound and recreating artist expression.

Recreating instrument sound

Virtual instruments do exist and are widely in use. You tagged your question as but while modeling sounds of real instruments was explored in the past, presently most instruments emulating sounds of real acoustic instruments base on sampling. Recording technique is developed since over 100 years, and today it works just perfect – you record a note played on an instrument, and then you play it back. It can't go wrong.

Typically an instrument consists of several or even several tens of GB of audio samples. The challenge, besides the obvious one of getting a consistent and high quality recording of all notes of the instrument, is to collect samples representing various dynamic levels and articulations. Another difficulty is to represent various nuances, like moving from one note to another, repeating the same note, ending a note, additional mechanical sounds made by the instrument...

Imitation is the most successful in music styles that don't rely too heavily on subtle instrument articulation changes, though the instrument producers keep pushing these boundaries farther.

Recreating artist expression

This is more difficult, as it's not well defined in technical terms. Sheet music represents certain idea, but the composer must rely on the musician's ability to interpret it. A musician will play each note differently. They will emphasize metric accents and phrases using capabilities of their instrument. A crescendo mark represents idea of increasing the volume, but in practice a simple gradual change may not create appropriate artistic effect. And "appropriate artistic effect" is a highly subjective term.

Various instrument offer various expressive capabilities. The most universal ones are timing and changes of tempo. Most instruments also can be played with various dynamics and may also provide various articulations. These nuances typically are not, or even cannot be precisely notated in a sheet music meant for humans. A music score fed directly into a virtual instrument, and thus devoid of these nuances will sound rather mechanical and not interesting.

Much better results can be obtained by converting sheet music to a MIDI sequence with manual adjustment of dynamics, timing and articulation of each note. It is a tedious process, and most importantly: manual. While some software tools can aid in it, human involvement is necessary. And recreating the expression of a skilled musician this way is still very challenging.

In practice MIDI sequences which are played by virtual instruments are often recorded by human musicians on MIDI controllers that capture their expression. This can yield very good results, but as a musician is involved, this doesn't fulfill the requirement of "converting sheet music into an audio recording by software".

Use of programmed music sequences emulating the real instruments is therefore the most successful in styles and situations that don't rely heavily on artist expression and interpretation, that have fixed tempo, or where said instrument is in the background. But as soon as we focus on artistic interpretation, a machine cannot replace a skilled musician. At least not yet.

3
  • The "fun" part: MIDIs recorded by human musicians on MIDI controllers often come with horrific tuplets, off-beat 64th notes, and readability issues. (At least that's my experience seeing others' MIDIs on Musescore and Finale Notepad.)
    – Dekkadeci
    Aug 31 at 10:34
  • MIDI file is not a notation format, but sequence of events (note on, note off, control change). If the musician doesn't play precisely on the beat, naively converting event timestamps to notation will produce horrible results.
    – ojs
    Aug 31 at 11:09
  • 1
    @ojs that's exactly the point of my answer. A good MIDI sequence doesn't have a simple correspondence to sheet music notation, and conversion between the two is not trivial. I made an edit to make it more clear. Aug 31 at 13:12
4

No, software that could take any given piece of sheet music and render it with the inflections and expressive devices that would be expected of a human performer (or group) is not yet generally available. A musical score isn't generally intended to be rendered precisely, but in a stylistically-appropriate way (see What does it mean to "play what is not written"? ; What does play with feeling mean?) and you need your performer to add that stylistic 'flavour'.

There's no reason that it would be impossible to create such software, and a google search for "render sheet music expressively" brings up links to a number of research projects - but we're certainly not at the point where "physical playing has been replaced by software quite a while ago" is a sensible statement.

Even if you involve a human in playing or programming in a digital performance for a synthesizer to render, it's arguable that synthesizer technology itself isn't quite at the point where it can reproduce all physical instruments in a way that would satisfy critical ears - so the statement "physical playing has been replaced by software quite a while ago" seems to me to fail on that count as well.

It seems rather similar to making a statement like "physical acting has been replaced by software quite a while ago". Clearly computers are able to help humans generate some very realistic artificial imagery, but you can't yet feed a script into a computer and render a photo-realistic movie.

-1

The word "well" is subjective. If by "well," you mean "precisely," then 100%, computers can do infinitely better than any person can. Scales at 1000 BPM, all notes played at exactly the same volume? Check!

If by "well," you mean "played with dynamics that give human beings deep and complex emotional reactions," then YES. . . but not yet.

As long as you can hook up an AI system to something real (like a brain), and you can correlate that to something abstract (like subjective reports of feelings), computers can absolutely learn to do that, infinitely better than even the best people.

Give it about 10-20 years before computers start playing "feedback music" that is dynamically interact to YOUR brain at a given MOMENT, to maximize mood.

17
  • 2
    Any sources to back up the claim would be nice.
    – ojs
    Aug 31 at 3:26
  • 1
    All of them, of course. But let's start with the 10-20 years. It sounds suspiciously close to the time when fusion power will be available.
    – ojs
    Aug 31 at 9:55
  • 1
    I'm also interested what "hooking up an AI system to a brain" means
    – ojs
    Aug 31 at 9:56
  • 2
    Sorry if this sounds arrogant, but do you have any actual neuroscience background?
    – ojs
    Aug 31 at 10:32
  • 1
    The people featured on thedailywtf are professional programmers too. Anyway, do you have any sources?
    – ojs
    Sep 1 at 5:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.