When does audio latency matter and not matter?

Question

When does audio latency (physical action to heard sound) matter and/or not matter?

iOS touch-to-sound latency is reported to be on the order of 50 milliSeconds, yet some musicians manage to play iOS music apps without too much complaint.

I've also heard reports of complaints about even 5 mS of latency, yet musicians used to stand well over 10 feet (diagonal, ground level to head level) from their stage monitor speakers which adds over 10 mS to any audio system latency.

So, what are the situations where a few milliSeconds of latency really matters? And what are the situations where even dozens of milliSeconds of latency might still allow acceptable (virtual or MIDI) musical (instrument) practice or performance?

Answer 1

Apparently the AES wondered the same at some point and did this study: http://lsbaudio.com/publications/AES_Latency.pdf

They had a sample comprising of different instrumentalists give a subjective grade to a monitoring system configured with a certain amount of latency,

While the title says "in live sound monitoring" if you read it you'll see that the in-ear-monitor tests answer your question perfectly even in a studio setting (the wedge ones are still interesting if less relevant).

You might also find this table - from a reputable author - useful: http://www.soundonsound.com/sos/jan05/articles/pcmusician.htm#7

It boils down to, roughly, optimal values of around:

• vocals < 3ms;
• drums < 6ms;
• guitars < 12ms;
• piano < 10ms;
• keyboards (pads, etc) < 20ms

Note how, according to the author,

"The speed of sound in air is roughly a thousand feet per second, each millisecond of delay is equivalent to listening to the sound from a point one foot further away. So if you can play an electric guitar 12 feet from your amp, you can easily cope with a 12ms latency."

and

"Vocals: This is the most difficult example, because anyone listening to their vocals in 'real time' will have headphones on, and therefore have the sounds 'inside their head'. A latency of even 3ms can be disconcerting in these conditions."

Above those values, I guess that it's all a matter of how much it acceptable to you personally (how picky you are) and to your work.

If you are doing 100+ vocal overdubs (or layered percussion), a value close to 0 is a good value -- you just don't want the delay on the consonants to pile up.

Ditto if you are Mariah Carey and you just can afford the best.

If you are doing a quick demo in a home studio - whatever -- you'll move the "s"'s around with the mouse, right?

Answer 2

I can only speak from my own experience as an electronic live musician.

The ms values are what I read in my computer software. I don't know how accurate it is, but they are replicated across multiple devices, and software packs. So I tend to use these values.

I've noticed that anything higher than 100ms is enough lag to be bothered by it, to the point of calling it 'unplayable'. Our brain can compensate for only so much.

If for instance I have 100ms latency, I will only hear my note at least 120ms after I played it with the instrument. Not only will I need to play my notes earlier than when I actually need to play them, not to sound off, but also, if I play a wrong note, I won't hear until 120ms later, process it, and adjust, and that whole latency can make a 0.5 sec round trip very audible to the audience!

For live music equipment, 100ms is the absolute upper medium, and comparable to having instrument monitors set up on the other side of a football field. You're already playing a new note, by the time you hear the first note being played. Is like reading subtitles and seeing the character on a tv speak one word after the word you're actually hearing through the speakers. Very annoying!

I try to aim for 32 or 24ms. With modern laptops, values as low as 18 to 12ms are possible. Anything below ~12-10ms is indistinguishable from the input. For instance, I can play a note and at the same time as the instrument gives the clicking noise from playing the note, hear the note being played through the monitors (without delay).

For that reason, as a live instrument player, I try to set the latency as low as possible, without overburdening the CPU. If I set it to 10ms, my CPU may start crackling noises, and CPU core usage may be above 100% on one core!

Once I find a happy medium, for instance as example below (CPU utilization meaning the percentage of CPU used depending on how many threads the program is actually made to use):

128ms = 2% CPU utilization
 64ms = 2.5% CPU utilization
 32ms = 3% CPU utilization
 18ms = 5% CPU utilization
 15ms = 10% CPU utilization
 12ms = 22% CPU utilization
 10ms = 85% CPU utilization

I may decide to set the values between 15 and 18ms. As a live instrument player, I can not afford having even one crackle or pop due to buffer underruns.

Answer 3

I would like to share my thoughts as they relate to your question about iOS touch to sound latency and "(virtual or MIDI) musical (instrument) practice or performance?"

I must assume that your primary interest is - latency as it relates to iOS musical instruments and devices or other "virtual and digitally produced musical sounds (MIDI) during "practice or performance" on these type devices or "instruments".

For the TL;DR skip to the end. To understand how I arrived at the conclusions at the end - continue reading.

The studies cited in Some Dude's answer and some of the anecdotal information cited in comments - relate to musicians playing real (as opposed to virtual) instruments. But what we learn from these studies and observations will lead us to some logical conclusions as they relate to your specific question.

First let me add to the anecdotal evidence from my perspective as a performing guitarist and vocalist who has also played some keyboard and piano. I often perform in a totally acoustic setting with un-amplified acoustic guitar and no microphone. In a case like that, the latency is very low - particularly on vocals. When I perform plugged in and sing through a mic, I generally use floor wedge monitors fairly close. None of these scenarios pose any problems for me.

When I perform in larger venues with a house PA and monitors that may be suspended from the ceiling, I am quite uncomfortable with the delay between the time I pluck a string on my guitar and the time I hear it through the monitor - and I have a difficult time performing under those conditions. I am sure that if I did that all the time as a matter of practice, I might eventually get used to it and adapt.

So let's look at what we can extrapolate from the AES study shared by Some Dude. Then we can make some conclusions about what the findings might suggest for virtual instruments.

In that study we learn that vocalist and saxophone players have the least amount of tolerance for latency and keyboardist and drummers have the most tolerance and guitar players are in between. The findings of the study clearly show that the tolerance levels are directly related to the instrument and NOT the individuals. The same individual playing different instruments would have latency tolerance levels consistent with other results on the given instrument.

So let's consider why that might be true. Whenever you talk or sing, your brain expects an immediate zero latency response to sound coming out of your mouth at the exact same time your diaphragm and lungs push air over your vocal chords. It has been that way from the time you entered the world screaming and hearing the sound of your own voice for the first time. So it's easy to understand why a vocalist, would have a low tolerance for latency. The brain is used to immediate feedback when vocalizing.

With the saxophone player, the brain has developed a similar expectation. Making a sound with a horn has one very significant cause and effect commonality with making a sound with your voice. The act of blowing air - produces an immediate sound. Whether singing or blowing a horn, there is a very conscious physiologic muscular action that produces a simultaneous and instantaneous reaction. Horn players are used to blowing through their instrument and immediately hearing a sound.

EDIT: Community member Supercat made and excellent point in comments and I felt compelled to include it in this answer. To sing a note with voice or to play a note with many types of horns - the facial muscles and vocal muscles actually control the sound as it's being created and minor adjustments must be made during the creation process to reach the desired pitch. So a singer for example can hear in the monitor if he/she is a bit sharp or flat and immediately make the necessary adjustments to the facial/vocal muscles to correct the pitch. Another good explanation for lower tolerance of latency for a singer or horn player.

Now let's consider why keyboardist have such a high tolerance for latency. Most keyboards including acoustic pianos have a slight delay between the time you press a key and the time you hear a sound. On acoustic piano, pushing the key triggers a mechanical action - which eventually causes a hammer inside the piano to strike the strings. But it is not an immediate "hit it and hear" it effect. Synthesizers and digital and electric piano's have always had some degree of latency. So keyboardist have had to learn to adapt to the unavoidable latency from the moment they began playing. So to a keyboardist, latency is normal and expected - therefore well tolerated. It's what they learned to expect from the very beginning.

With a drummer, it is in fact a hit it and hear it response. So how do we explain a drummer's tolerance to latency in the AES study? The study done in 2007, did not mention the use of electronic drums for testing the drummer's tolerance to latency so I must assume that the drummers in the test, used an acoustic kit.

A drummer hitting an acoustic drum with a drumstick, quite frankly does not need a monitor to hear the sound of his drum. A drummer uses a monitor NOT to hear himself (some drummers even wear earplugs because they can hear their drums TOO well) but to hear the other musicians (and the other musicians turn up their monitors so they can hear themselves over top of the drummer).

So a logical conclusion that we can make based on the results of the AES study - is that tolerance to latency is **learned.** The brain learns all kinds of useful information that keeps us sane. For example, any time your eyes are open, you see your nose, but the brain has learned to disregard that image, because your brain knows it's not important.

The brain has learned to expect immediate feedback when we exert the necessary muscle contractions to blow air - either through a horn or our vocal chords. If we learn to play a piano or other keyboard instrument - the brain LEARNED from the beginning to adjust to the built in latency of the keyboard.

So my expectation would be, that we can easily learn to tolerate latency with most virtual instruments, because we have not been pre-conditioned to expect otherwise. If we never learn to expect an immediate response from virtual instruments, latency should not present the same problem it would for a vocalist, or sax player or guitarist.

I would also expect that with iOS drums played on a touchscreen, latency might very well present some timing issues. If you tap the screen and don't hear an immediate response to the drum, it could make playing in time very difficult. Just the opposite of what live drummers on real acoustic kits experienced in the latency tolerance test.

In reality, there is no tolerance for latency for a drummer. It's impossible to create or simulate latency for a drummer on a live acoustic kit. But on an iOS drum, you could introduce latency - and I would suspect that this would be the one case - where a few milliseconds of latency would matter.

Recording and monitoring vocals is not included in the scope of this answer because you did not mention vocals or even recording (just playing and practicing - instruments) in the question. So that is a completely different discussion.

TL;DR - The bottom line answer to your questions:

1. Question: "What are the situations where a few milliSeconds of latency really matters?" Answer: When playing iOS or MIDI drums on a silent pad or touchscreen if playing with other music where timing is important.

2. Question: "What are the situations where even dozens of milliSeconds of latency might still allow acceptable (virtual or MIDI) musical (instrument) practice or performance?"

Answer: When practicing or performing on any iOS or MIDI or virtual instrument other than drums.

Answer 4

The effects of latency are a sliding scale ranging through:

• zero
• imperceivable
• perceivable as "feel" (like a drummer who's pulling slightly)
• perceivable as "something a bit wrong"
• perceivable as "something completely wrong"; making playing completely impossible

The effect also depend on the kind of sound you're creating. If you're playing something with a gentle attack, like synth washes, then the exact timing doesn't really matter. If you're playing something rhythmic and staccato like drums or a rhythm guitar part, then timing is crucial.

Even a lead guitar part can be quite tolerant of timing shifts, compared to the rhythm parts which need to be tight.

The 50ms delay you mention is around the "feel" or "something a bit wrong" level, and wouldn't completely ruin music, or make it hard to play.

However, latency accumulates. 50ms from touchscreen to sound being generated might be OK. 50ms for the sound to travel from speaker to ear might be OK. Add them together and you've got 100ms, which is not OK.

A 5ms latency in an effects pedal might seem insignificant -- but some guitarists like to chain 10 or more effects, so every millisecond counts.

There are all kinds of situations where latency accumulates -- for example when overdubbing multiple tracks, and you (and/or software authors) have to keep this in mind when engineering.

Answer 5

When the (ISA)Turtle Beach Pinnacle was the king of PC soundcards, with 750 ms of audio latency, we somehow managed to make perfectly-synchronized multi-track recordings by arranging direct monitoring of the input, and pre-fetching the already ecorded tracks.

But if you want to play a software sampler in real time from a MIDI keyboard, or to hear a processed (by the computer) version of your audio input, you need to strive for low latency.

Anything in the region of 10ms or lower is generally considered acceptable. Low single-figure values are quite easily achieved. Anyone content with 50ms is either being very undemanding or is deluded by iPad worship :-)

When "zero latency monitoring" is advertised, it's misleading. Zero is unattainable. They're talking about looping an input signal straight out again, without going through the audio processing system on the computer. It isn't zero latency, it's just avoiding latency completely. (Like we did in the first paragraph of this answer.)

Answer 6

One example of when audio latency barely matters:

You're recording clean vocals into your PC your audio interface, singing along with the tracks you've already laid down. There will be a latency effect both on the way in to the PC, and on the way out. However, the computer knows how big the audio buffer is on the way out, and plays 'early' to compensate; likewise, the DAW knows that what you played was actually played a bit earlier in time, and pulls it forward in time by the buffer length. So although there are two stages of latency in this system, it's almost all dealt with; you barely notice it (if everything's working, of course)

I say 'almost' dealt with because there are some delays in the system that aren't so easily quantifiable. But in practice it seems to get very close.