12

When does audio latency (physical action to heard sound) matter and/or not matter?

iOS touch-to-sound latency is reported to be on the order of 50 milliSeconds, yet some musicians manage to play iOS music apps without too much complaint.

I've also heard reports of complaints about even 5 mS of latency, yet musicians used to stand well over 10 feet (diagonal, ground level to head level) from their stage monitor speakers which adds over 10 mS to any audio system latency.

So, what are the situations where a few milliSeconds of latency really matters? And what are the situations where even dozens of milliSeconds of latency might still allow acceptable (virtual or MIDI) musical (instrument) practice or performance?

  • I don't feel that I have quite enough knowledge to provide an actual answer but my understanding is that the average human can hear that two repetitions of the same sound happened at different times when they are >30ms. In this context, I imagine the latency would become most noticeable around 30ms. This tends to matter a lot more when recording overdubs, where too much latency is actually noticeable, consistent and generally you intend to hear that recording repeatedly, so small issues that might go unnoticed in live performance will stand out more. – Basstickler Feb 27 '15 at 21:25
  • 1
    @Basstickler - I too hesitated to put up an actual answer. You can 'feel' latency at around 5 - 7 ms, though not strictly 'hear' it [phase discrepancy aside]. Yet for a long time, early synths/midi keyboards had 10 - 25ms latency even when just playing the keyboard attached; musos just 'got used to it' & the world changed because of that. [I still can't manufacture an actual answer from that info] – Tetsujin Feb 27 '15 at 21:30
  • @Tetsujin - I have basically gathered that some hear/feel it more than others. I'm also wondering to what extent the 30ms rule would actually apply to two different sounds. Typically the human condition leads us to inconsistencies in rhythm, so a little latency isn't much different than that, other than the fact that it is all set back by whatever the latency is, making things that are behind the beat further behind and those ahead either less ahead, on time or behind. As such, I would imagine that the difference of the overall performance could be felt. – Basstickler Feb 27 '15 at 21:46
  • I second that 'some hear/feel it more than others', although I think technically anyone could be trained to notice the differences. It is really a matter of what you are used to, what you're expecting, and what your performance requirements are. To make a tight, studio quality recording and to really lock yourself in 'the zone' I posit that anything above 5 or 6 milliseconds can really start to cause problems. If you are in a live performance, put that up to 20 and nobody'll be the wiser. If you're playing something with a soft, slow attack like synth pads I daresay even 500 ms can be ok. – Darren Ringer Feb 27 '15 at 22:46
  • 2
    Oh, you know who had to deal with latency in the pre-electricity era? Organists who played organs with pneumatic transmission. Read it up. – Some Dude On The Interwebs Feb 28 '15 at 9:45
10

Apparently the AES wondered the same at some point and did this study: http://lsbaudio.com/publications/AES_Latency.pdf

They had a sample comprising of different instrumentalists give a subjective grade to a monitoring system configured with a certain amount of latency,

While the title says "in live sound monitoring" if you read it you'll see that the in-ear-monitor tests answer your question perfectly even in a studio setting (the wedge ones are still interesting if less relevant).

You might also find this table - from a reputable author - useful: http://www.soundonsound.com/sos/jan05/articles/pcmusician.htm#7

It boils down to, roughly, optimal values of around:

  • vocals < 3ms;
  • drums < 6ms;
  • guitars < 12ms;
  • piano < 10ms;
  • keyboards (pads, etc) < 20ms

Note how, according to the author,

"The speed of sound in air is roughly a thousand feet per second, each millisecond of delay is equivalent to listening to the sound from a point one foot further away. So if you can play an electric guitar 12 feet from your amp, you can easily cope with a 12ms latency."

and

"Vocals: This is the most difficult example, because anyone listening to their vocals in 'real time' will have headphones on, and therefore have the sounds 'inside their head'. A latency of even 3ms can be disconcerting in these conditions."

Above those values, I guess that it's all a matter of how much it acceptable to you personally (how picky you are) and to your work.

If you are doing 100+ vocal overdubs (or layered percussion), a value close to 0 is a good value -- you just don't want the delay on the consonants to pile up.

Ditto if you are Mariah Carey and you just can afford the best.

If you are doing a quick demo in a home studio - whatever -- you'll move the "s"'s around with the mouse, right?

  • 1
    Thanks for the links. Great job. I mentioned your contributions in my answer. – Rockin Cowboy Feb 28 '15 at 4:09
  • @RockinCowboy: you are welcome :) – Some Dude On The Interwebs Feb 28 '15 at 9:40
  • "If you are doing 100+ vocal overdubs (or layered percussion), a value close to 0 is a good value -- you just don't want the delay on the consonants to pile up." I don't think that's right--any recording software can automatically shift the overdubs to the right position, all you need is round trip latency that's known and predictable. The absolute value of the latency matters more if you need to monitor yourself through the software (for example, if you want your DAW to apply some effects to what you hear live. If that doesn't matter, zero-latency hardware monitoring is usually easy.) – Bruce Fields Mar 30 '17 at 2:23
4

I would like to share my thoughts as they relate to your question about iOS touch to sound latency and "(virtual or MIDI) musical (instrument) practice or performance?"

I must assume that your primary interest is - latency as it relates to iOS musical instruments and devices or other "virtual and digitally produced musical sounds (MIDI) during "practice or performance" on these type devices or "instruments".

For the TL;DR skip to the end. To understand how I arrived at the conclusions at the end - continue reading.

The studies cited in Some Dude's answer and some of the anecdotal information cited in comments - relate to musicians playing real (as opposed to virtual) instruments. But what we learn from these studies and observations will lead us to some logical conclusions as they relate to your specific question.

First let me add to the anecdotal evidence from my perspective as a performing guitarist and vocalist who has also played some keyboard and piano. I often perform in a totally acoustic setting with un-amplified acoustic guitar and no microphone. In a case like that, the latency is very low - particularly on vocals. When I perform plugged in and sing through a mic, I generally use floor wedge monitors fairly close. None of these scenarios pose any problems for me.

When I perform in larger venues with a house PA and monitors that may be suspended from the ceiling, I am quite uncomfortable with the delay between the time I pluck a string on my guitar and the time I hear it through the monitor - and I have a difficult time performing under those conditions. I am sure that if I did that all the time as a matter of practice, I might eventually get used to it and adapt.

So let's look at what we can extrapolate from the AES study shared by Some Dude. Then we can make some conclusions about what the findings might suggest for virtual instruments.

In that study we learn that vocalist and saxophone players have the least amount of tolerance for latency and keyboardist and drummers have the most tolerance and guitar players are in between. The findings of the study clearly show that the tolerance levels are directly related to the instrument and NOT the individuals. The same individual playing different instruments would have latency tolerance levels consistent with other results on the given instrument.

So let's consider why that might be true. Whenever you talk or sing, your brain expects an immediate zero latency response to sound coming out of your mouth at the exact same time your diaphragm and lungs push air over your vocal chords. It has been that way from the time you entered the world screaming and hearing the sound of your own voice for the first time. So it's easy to understand why a vocalist, would have a low tolerance for latency. The brain is used to immediate feedback when vocalizing.

With the saxophone player, the brain has developed a similar expectation. Making a sound with a horn has one very significant cause and effect commonality with making a sound with your voice. The act of blowing air - produces an immediate sound. Whether singing or blowing a horn, there is a very conscious physiologic muscular action that produces a simultaneous and instantaneous reaction. Horn players are used to blowing through their instrument and immediately hearing a sound.

EDIT: Community member Supercat made and excellent point in comments and I felt compelled to include it in this answer. To sing a note with voice or to play a note with many types of horns - the facial muscles and vocal muscles actually control the sound as it's being created and minor adjustments must be made during the creation process to reach the desired pitch. So a singer for example can hear in the monitor if he/she is a bit sharp or flat and immediately make the necessary adjustments to the facial/vocal muscles to correct the pitch. Another good explanation for lower tolerance of latency for a singer or horn player.

Now let's consider why keyboardist have such a high tolerance for latency. Most keyboards including acoustic pianos have a slight delay between the time you press a key and the time you hear a sound. On acoustic piano, pushing the key triggers a mechanical action - which eventually causes a hammer inside the piano to strike the strings. But it is not an immediate "hit it and hear" it effect. Synthesizers and digital and electric piano's have always had some degree of latency. So keyboardist have had to learn to adapt to the unavoidable latency from the moment they began playing. So to a keyboardist, latency is normal and expected - therefore well tolerated. It's what they learned to expect from the very beginning.

With a drummer, it is in fact a hit it and hear it response. So how do we explain a drummer's tolerance to latency in the AES study? The study done in 2007, did not mention the use of electronic drums for testing the drummer's tolerance to latency so I must assume that the drummers in the test, used an acoustic kit.

A drummer hitting an acoustic drum with a drumstick, quite frankly does not need a monitor to hear the sound of his drum. A drummer uses a monitor NOT to hear himself (some drummers even wear earplugs because they can hear their drums TOO well) but to hear the other musicians (and the other musicians turn up their monitors so they can hear themselves over top of the drummer).

So a logical conclusion that we can make based on the results of the AES study - is that tolerance to latency is **learned.** The brain learns all kinds of useful information that keeps us sane. For example, any time your eyes are open, you see your nose, but the brain has learned to disregard that image, because your brain knows it's not important.

The brain has learned to expect immediate feedback when we exert the necessary muscle contractions to blow air - either through a horn or our vocal chords. If we learn to play a piano or other keyboard instrument - the brain LEARNED from the beginning to adjust to the built in latency of the keyboard.

So my expectation would be, that we can easily learn to tolerate latency with most virtual instruments, because we have not been pre-conditioned to expect otherwise. If we never learn to expect an immediate response from virtual instruments, latency should not present the same problem it would for a vocalist, or sax player or guitarist.

I would also expect that with iOS drums played on a touchscreen, latency might very well present some timing issues. If you tap the screen and don't hear an immediate response to the drum, it could make playing in time very difficult. Just the opposite of what live drummers on real acoustic kits experienced in the latency tolerance test.

In reality, there is no tolerance for latency for a drummer. It's impossible to create or simulate latency for a drummer on a live acoustic kit. But on an iOS drum, you could introduce latency - and I would suspect that this would be the one case - where a few milliseconds of latency would matter.

Recording and monitoring vocals is not included in the scope of this answer because you did not mention vocals or even recording (just playing and practicing - instruments) in the question. So that is a completely different discussion.

TL;DR - The bottom line answer to your questions:

  1. Question: "What are the situations where a few milliSeconds of latency really matters?" Answer: When playing iOS or MIDI drums on a silent pad or touchscreen if playing with other music where timing is important.

  2. Question: "What are the situations where even dozens of milliSeconds of latency might still allow acceptable (virtual or MIDI) musical (instrument) practice or performance?"

Answer: When practicing or performing on any iOS or MIDI or virtual instrument other than drums.

  • So, given your experience, it seems that not only drums, but simulated stringed instruments on any iPhone or iPad should also feel unacceptable to an experienced musician, assuming that the reports of around a 50 mS minimum latency from physical touch to audio output are correct. Same with "blown" virtual wind instruments, given around 15 to 17 mS of latency from mic to audio, (not counting DSP "blow" detection) minimum. – hotpaw2 Feb 28 '15 at 8:38
  • @RockinCowboy Good one :) – Some Dude On The Interwebs Feb 28 '15 at 9:41
  • @hotpaw2 a simulated guitar on an iPhone is a different instrument than a real guitar. It lacks the tactile feel of plucking a string, feeling a resulting vibration, etc. It is going to be more akin to playing a keyboard - even if the "keyboard" is made to look and sound like a guitar. But for serious in the pocket playing with other instruments, I think I would prefer as little latency as possible. I have no personal experience with wind blown MIDI horns. But reading online reviews and forums, it appears that they do require an "adjustment in thought process" and "a learning curve". – Rockin Cowboy Feb 28 '15 at 14:47
  • 1
    I suspect another critical factor with the latency tolerance on keyboard and drums is that all of the actions to initiate a note must be completed before anything is audible, whereas singing or playing a saxophone requires that one adjust one's facial/vocal muscles in response to what the voice or instrument is doing. – supercat Mar 2 '15 at 20:12
  • @supercat Excellent point. In other words, the facial muscles and vocal muscles actually control the sound as it's being created and minor adjustments must be made during the creation process. – Rockin Cowboy Mar 2 '15 at 21:01
2

The effects of latency are a sliding scale ranging through:

  • zero
  • imperceivable
  • perceivable as "feel" (like a drummer who's pulling slightly)
  • perceivable as "something a bit wrong"
  • perceivable as "something completely wrong"; making playing completely impossible

The effect also depend on the kind of sound you're creating. If you're playing something with a gentle attack, like synth washes, then the exact timing doesn't really matter. If you're playing something rhythmic and staccato like drums or a rhythm guitar part, then timing is crucial.

Even a lead guitar part can be quite tolerant of timing shifts, compared to the rhythm parts which need to be tight.

The 50ms delay you mention is around the "feel" or "something a bit wrong" level, and wouldn't completely ruin music, or make it hard to play.

However, latency accumulates. 50ms from touchscreen to sound being generated might be OK. 50ms for the sound to travel from speaker to ear might be OK. Add them together and you've got 100ms, which is not OK.

A 5ms latency in an effects pedal might seem insignificant -- but some guitarists like to chain 10 or more effects, so every millisecond counts.

There are all kinds of situations where latency accumulates -- for example when overdubbing multiple tracks, and you (and/or software authors) have to keep this in mind when engineering.

1

One example of when audio latency barely matters:

You're recording clean vocals into your PC your audio interface, singing along with the tracks you've already laid down. There will be a latency effect both on the way in to the PC, and on the way out. However, the computer knows how big the audio buffer is on the way out, and plays 'early' to compensate; likewise, the DAW knows that what you played was actually played a bit earlier in time, and pulls it forward in time by the buffer length. So although there are two stages of latency in this system, it's almost all dealt with; you barely notice it (if everything's working, of course)

I say 'almost' dealt with because there are some delays in the system that aren't so easily quantifiable. But in practice it seems to get very close.

  • That is, if you are not monitoring through the PC :) – Some Dude On The Interwebs Feb 27 '15 at 22:55
  • Yes - in my scenario I was imagining we were not monitoring at all (just hearing yourself sing). But true, if you want to monitor and be in the scenario I described, monitoring needs to be from a point before the signal gets to the digital world of the PC. – topo morto Feb 27 '15 at 23:03
1

When the (ISA)Turtle Beach Pinnacle was the king of PC soundcards, with 750 ms of audio latency, we somehow managed to make perfectly-synchronized multi-track recordings by arranging direct monitoring of the input, and pre-fetching the already ecorded tracks.

But if you want to play a software sampler in real time from a MIDI keyboard, or to hear a processed (by the computer) version of your audio input, you need to strive for low latency.

Anything in the region of 10ms or lower is generally considered acceptable. Low single-figure values are quite easily achieved. Anyone content with 50ms is either being very undemanding or is deluded by iPad worship :-)

When "zero latency monitoring" is advertised, it's misleading. Zero is unattainable. They're talking about looping an input signal straight out again, without going through the audio processing system on the computer. It isn't zero latency, it's just avoiding latency completely. (Like we did in the first paragraph of this answer.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.