It sounds like you agree with him: "I tell him I am not sure their timing is off, but if it is, they have great backup musicians who can deal with it.".
So the problem as you tell it to him is that you aren't great enough to deal with tempo variations of the lead. The classic "it's not you, it's me" tantra that translates into "I don't expect you to change but maybe I'll go elsewhere".
If your imperfection would indeed be the actual problem, practice would be the obvious answer.
So why are you arguing with your own incompetency rather than with any inherent problem of his lack of timekeeping?
Is the result ugly or is it wrong? "Wrong" is not a useful category to judge entertainment with, and ugliness is in the mind of the beholder.
So how about recording your performances and listening to them? Stuff feels different to performers and listeners even when the same person.
The main challenge here is not to take the joy out of what the two of you are doing.
If all of this sounds rather openended, it's because it is. I do a lot of solo play from scores I haven't actually heard performed, and take a whole lot of liberties with the phrasing, execution and tempo depending on the expressiveness of the instrument. When I'm playing Tango Nuevo on an acoustic accordion, it would be quite challenging to transcribe the rhythm from my playing: it's more kind of a general flow. When I am playing on a Midi accordion hooked up to a Grand Organ simulator with a cathedral of reverbation, my timing tends to be much more precisely following the score, probably because the acoustics of simulated instrument and room cannot follow tempo variations as organically as a continuously controlled instrument like voice, a wind or even a string instrument can.
When working expressively with music, its rhythm may be pliable like the meter of poetry: it doesn't gain from mechanical rigidity and yet its presence is what makes a difference.
So the question is how cohesive, organic and deliberate his variations are and whether they, taken alone, improve the performance to the listener or detract from it. If it is indeed the former (and that's where recording and asking opinions of yourself and others without leading questions may help), it makes sense to try how much can actually be accommodated in a duo without things falling apart or becoming inconsistent. That's something you need to work out based on a metric entirely different from a "right/wrong" perspective, and then you need to see how much fun this leaves either of you with ultimately.