I mostly play bass and guitar (and a little drums), and I've done a number of martial arts over the years - mostly Brazilian jiu-jitsu, but also boxing, kickboxing and MMA. My personal experience, and what I've gathered from other people in the gym:
Grappling arts like judo or BJJ are great for improving for grip strength. I didn't experience negative effects on speed or sensitivity. However, finger injuries are not uncommon - I personally never had any, but many of my training partners have injured their fingers in one way or another.
Boxing, when done right, also strengthens the wrist muscles, bones and tendons. Do make sure to wrap your hand and wrist thoroughly and to keep your wrist straight when hitting things. Hand injuries can occur, in particular broken metacarpals and sprained wrists, but mostly if you hit hard objects (skulls) really hard - shouldn't be a problem for a careful hobbyist. Boxing also emphasizes staying loose and relaxed in the upper body - that should actually be helpful for drumming.
I've never done karate. From what I read, some karate styles have a focus on hand conditioning - i.e., hitting hard things with the bare hand. I would not recomment that for musicians.
Generally, I don't see a reason that developing strength is automatically bad for dexterity, speed or anything else, and anecdotal evidence also indicates that's not the case (have you seen John Petrucci's biceps?). But check out a couple of styles and gyms, talk to the instructors about your worries, and see what they say.