Long story, and I don't expect magic answers. But I'm always striving to improve and learn, and some of this might elicit good advice. Read if you have the time.
I've been playing a long time, well over 40 years. You may or may not have heard of me as the "internet Peter Pan" guy. Some of my music is here on reverb nation, and it is indeed "fairy tale themed" rock. http://www.reverbnation.com/peterpanrandyconstan I have some jazz stuff elsewhere, though there is one up-tempo jazz tune on that reverb nation page at the bottom of the song list too (Zing). You may or may not like any of this stuff, but its always good to share what you're doing when you approach anyone discussing technique, so they have an idea where you're coming from musically.
Now the guitar has always been a struggle for me, and most teachers have only been able to help my technical problems so much. Truth be known, the lion's-share of all I have gained technique-wise over the years has come by my violating many of the so-called "right ways" in favor of finding something that at least works. Here is some history of my problem, which has always been 90% a right hand issue, and what I've done so far to push my barriers.
My practice, at least the technical part, has always involved the standard protocol... various scales, arpeggios, chords, inversions, rythems, and rhythmic accent combinations, obviously always practiced with a metronome at progressively higher tempos. But sooner or later I'll get to that point where my picking hand just doesn't want to go faster. There are some reasonable gains by putting in more hours of practice, trying different picking angles, widely different techniques, wrist angles, different amounts of pick motion from the fingers holding the pick, wrist motion (at least a dozen wrist muscle choices), and even elbow and arm motion blended in. But still, sooner or later for some of us it becomes obvious that where some guitarists become better and better, we are stuck in a rut. So what happens then? many players give up, or submit to the fact that they just will never have the chops to play the things they would like. And lets face it, there are only so many time you can revisit the road of starting over, with some teacher telling you the "right" way to pick... the right way to focus or relax tension. All good advise usually, but as the years tick on its obvious you have to either give up or find a way to break through. Most teachers are GOOD, meaning they didn't have these problems, and so can only understand these ruts to a limited degree.
Now my approach was a little different. Call it a terrible choice, but after years of frustration as a teenage player not getting anywhere, I finally took the first tempo on that metronome that was giving me grief and I FORCED my hand to pick it. Lo and behold after doing so for a while and then resting, that tempo was now comfortable. So...I'd play a while more at that tempo, rest, and then move up to the next tempo. Force some more, Rinse, repeat, and with enough hours in the day, this would get me up to about 1/16 notes at 120 (or 1/8 notes depending on whether you call 120 the 1/4 note or the 1/2 note. But anyway, that's pretty good for most rock 'n' roll, even if its not shredding tempos. Interestingly, at some point around that tempo, it felt like I was driving at high speed in first gear, and so I began exploring technique variations. I found that some muscle combinations were better for the speeds beyond about 112, others better for tempos up to that amount. And over time I've learned many more technique variations that seem to help me better focus my energy, make the best use of consecutive vs alternate picking, and tons of other little solutions I'm sure you're heard enough times to get tired of it. All totaled, I can now play at pretty much any tempo as long as I'm willing to put in non-stop practice.
So OK, so what if I had to force? If it works it works, right? Well every good player I know still advises against such forcing, so I'm still always pressing for better solutions. For one thing, my muscle memory was never a strong point for me. At least not to extent I'd hoped. To this day, for me to be at my best, I still have to start relatively slow (maybe 1/16th notes in the mid 90s), gradually move up to my find my problem zone, and then start climbing the force ladder again. It takes hours... hours of time dedicated to what I'd call relatively non-musical things. I'm encouraged when players I respect tell me that if I want good muscle memory, I have to play at least 4 hours a day. That's encouraging because it reminds me that even the best players don't skimp when it comes to practice time. But I don't have that kind of time anymore. So I'm continually looking for shortcuts or advice that might help.
Now over the past couple of years i have in fact re-visited all aspects of my right hand technique, and it has yielded some good time saving results. Even if I have to force, for example, it is helpful for me to spend a little time in the beginning of my practice at a very slow tempo, so that I can channel that force into making my motions crisp, short, and with just the right twist or bounce so that cross string alternate picking will work without colliding. And here's another... I've learned that playing small groups of notes at problem tempos yielded a lot better retained accuracy, and somewhat less forcing. Seems that its not the speed that requires the force as much as the non stop playing up and down the scales. Playing groups of 5, then 9, then a few more notes at a time with rests in-between seems to be easing the problem. But still, forcing has always seemed to be the only constant that always yields results, even if I'd prefer that was not the case.
So this is the reason I'm asking this question... to see how many others are in a similar bind, and what they are doing about it. I'm sure there are many players who could chime in and say things like... "playing at any tempo should be effortless, and require zero force". Yeah, and we should all have John McLaughlin's chops on an acoustic guitar with .013 strings too, but this is more a discussion for the "rest" of us, that have struggled some, but have refused to give up, and have gained ground as a result.
Anyway, thanks for listening.
One recent discovery, which I'm adding quite a long time after writing the above. An interesting thing I've added to me practice is that when I get to a troublesome tempo, I play short groups of notes for a while at that tempo instead of whole scales, and gradually work my way up. It feels as though my 'speed" muscles are executing the shorter runs just fine, but my slower "endurance" muscles soon take over, and I end up tensing up, followed by forcing. By doing, say, 5 notes of a scale, so that the first and last end dead on the tick, the metronome is still keeping me honest, but I'm now mainly exercising the speed muscles cell, and resting in between. The hope was to slowly increase the endurance of the speed muscles, rather than increase the speed of the endurance muscles. And after almost 1/2 year of doing this, I can say that it works! It doesn't work miracles, but it does work, and I'm gradually moving to greater accuracy, and less forcing. the practice time unfortunately takes longer, because the key is coming to a complete rest between these note groups. But soon I can get to groups of 9 (2 bars of 1/6th notes), and soon 2 octaves up two octaves down, and soon 2 octaves up and down followed by the rest. Sometimes on moves that have given me grief (often cross string combinations), I'll just focus on 3 notes, resting , repeat.
I'm absolutely sure I'm on to something here! It comes down to this. They say you can make a sprinter into a long distance runner, but not the other way around. But now methods of speed training for athletes have shown that this technique, using short bursts of speed followed by resting, works. You may not be able to make a long distance runner into a world class sprinter... but you definitely can make him into a BETTER sprinter than he was, by exercising in manner that targets the muscle cells most in need of work.
I won't say this is a magic bullet... I've not totally eliminated the need to force certain things. But it definitely helps. And as we all know, especially if you've been playing a long time, anything that actually does help is gravy!