It's very subjective to 'the moment' and also dependent on the genre of music in question. What is considered emotive, exciting, phrasing in one piece may be perceived as sloppy or late execution in another.
That said, when I was working in a fancy studio with high budgets and little room for error my secondary specialism (primary being tuning parts with melodyne etc.) was editing drums. Ideally no editing was the objective but if a band or specifically drummer wasn't getting locked in to the degree desired then editing was needed, generally this was with smaller bands on a lower budget but it happens with very experienced bands too.
When required to make a drum edit I would of course take stylistic considerations into the task, but to answer the question on what's percievable I would tend to work toward either a 10ms or 3ms edit, the latter taking quite a lot more time of course. Other engineers at the studio would send me a part to edit while I was on downtime and they would carry on tracking (of course, tracking to a sub optimal part was not ideal but time is money in studios). The 3ms / 10ms choice would essentially be how I would set the 'nudge' settings in protools, the resolution of each edit I made. 10ms is about enough to take a sloppy part and make it sound 'in time'. This is heavily dependant on the tempo of course but I've found percentage deviation of the position of a transient is less important than absolute deviation. Though this relationship MAY be non linear as using varispeed, which is (generally) speeding up a recording in post does tend to make all timing errors sound less egregious, but it doesn't effect the feel a rhythmic player is imparting.
I should be clear this is with rock/pop recordings where tempo is strict and room for performer time manipulation for the purposes of feel very limited.
So within about 10ms, ±5ms about the objective ideal, a part would sound in time and probably not perceptible as 'bad' to most listeners.
If I were doing a 3ms edit then I would spend time on each snare (for example) pushing and pulling it to get the feel required. 3ms deviation is very hard to recognise, even when I was doing this 10 hours a day every day for many (misspent perhaps?) years of my life! But once you learn to hear it it becomes the difference between something sounding amazingly locked in, ala Steely Dan or similar, and just, well, 'in time'. I want to be clear I was not aiming towards aligning with a grid of the tempo, editing in the way I describe is all done by ear and with an understanding that the most 'locked in' a drum part may sound may be with all the snares (for example, rarely the kicks) behind the beat, or ahead of the beat for that matter.
So I would say a good evaluator of performance at the highest professional level would work at around the 3ms level being the smallest possible margin of error thats perceivable and editable. That sounds small but I think most accomplished and experienced musicians would hear the difference between a drum loop where snares were all exactly on beat compared to where they were all dragging by 3ms. This is probably a skill that only has need to develop in the western popular music field, where metronomic accuracy, and deviation from it to create pushy or pully feels, is an important but relatively unique concept.
That said, a further understanding of perception of delay can be investigated when recording and dealing with the latency of native audio systems (that is, the audio processing is happening on a general purpose computer, not a specialised low latency device). A critical drawback of earlier audio interfaces was inherent latency in the system, so much so that insanely expensive hardware solutions were the norm (think early Protools). A latency of 5ms - 10ms in recording is definitely noticeable, though not perhaps to the average listener. A performer used to perceiving audio feedback to the tune of the speed of sound (which is probably the limiting factor given the nature of the audio reproduction and perception systems we use) is used to near instant response when sitting next to their drum or guitar amp, and a processing delay of even 5ms was considered 'not that good' for audio interfaces, suggesting some kind of perceived difference down to this level.
Anecdotally, I perceive when playing electric bass (not talking of editing now) about 2ms of latency as feeling odd, and this will potentially impact a recording I make, though it's very subjective at this speed and I would hope I could do a good job at much higher latency. This is specifically on headphones and electric bass, where you get used to the instant feel of finger>string>pickup>electric-signal. It’s more a situation of being very slightly off-put by not hearing what your fingertips are feeling, rather than anything being ‘late’.
That said, you could be way less accurate than this and still be considered playing perfectly 'in time', especially if you are not playing a style that values these quite silly levels of precision (which is most music, to be honest, although these silly levels of precision have their place for a certain sound!).
EDIT - More Detail on the editing process
It seems like lots of people have read this post so I thought I should add a bit more context in case anyone reads and feels it sounds like an impossibly large task to edit in the way I mention, it's not really that bad, ProTools makes this sort of editing very easy.
Using tab-to-transient + shift you can select all of one drum/note, up to the start of the next, then you can split with b then nudge around with m and n to your nudge resolution. Once you reached the end of the section you want to edit you can use an inbuilt feature called 'fill gaps and crossfade' that deals with any gaps caused by delaying rushy notes.
3ms was the resolution, not the accuracy for every beat. 3ms is almost imperceptible, JUST imperceptible. Thats a good threshold for a tool, now a double tap of 'n' is clearly noticeable, a triple tap and you can fix a flaw. If you had time you could spend a bit more time with single taps to get the feel just right. If the part was bad enough to be pressing nudge 6 times I'd scrap 3ms and go to 10.
With practice you could do it touch-type speed (almost) so its a mad process to hear happening.
Also, the aim here is NOT to hear a decent eg. drum track then try to doctor its feel. Generally you would be editing a section where the groove is sloppy or mistakes were made, once mistakes are fixed it can all sound a bit plastic, with the luxury of time you may then be able to go in with the 3ms tool and push it around until it sounds great. Occasionally the snare might be generally too lazy in eg. the chorus, but click, tab, shift+tab, b, n, n per snare and say 32 - 64 snares in a chorus is not that bad all in all.
For this sort of editing you need 100% isolation, or you will get an impossibly phasey mess. You also almost always edit all the drum tracks grouped together as one for the same reasons, your cymbals start sounding very very odd if you leave the overheads out of the drum group by mistake...
And lastly, the need to edit is not always because someone did a bad job. A common problem is bands that get signed often work really hard and are very good musicians but they haven't studied music deeply for years, or had training and a sound practice ethic to be become world class performers, besides that would probably knock all the edge off your average band anyway! But pressure is high, budgets strained and sometimes aspirations of label/producer/band not matching reality ('We want this one to sound really like Dreamtheater!.....').
