...in the last years of his life he said what to me is a very
revealing statement: he said that a great deal of music remains
to be written in the key of C major. But that means to me two
things: first of all it means that he was a man of some
catholicity of taste—he didn't automatically assume that
because he had espoused a certain system of composing music
which came to be called, not too accurately, the twelve-tone
technique, that all music not composed in that system or not
harmonious with it or not in some way acquiescent to it was
false or wrong...That's one thing it represents.
I think this is probably pretty easy to understand. Basically, Schoenberg and others had put forward a new strategy for composing music. Some composers who had come up with a system like that would be very dogmatic about it and insist that everyone who didn't follow them was on the wrong track (which was an attitude that some later serialists did take). Gould is suggesting that Schoenberg himself wasn't so dogmatic as that, but rather that he saw his approach as one reasonable possibility among many.
EDIT 2: If you're wondering what his "not too accurately" comment is about, my guess is it's because serialism came to be understood as a much more general idea than simply relating to the 12 tones of the traditional chromatic scale—so general that you could even apply it to other art forms aside from music. Read on for more. :P
The other thing that it represents, I think, is the feeling
that comes very strongly from Schoenberg's compositions and
from his writing about those compositions in his last years,
and that is that he begins to see that the very highly
dissonant forms of twelve-tone music, of serial music...aren't
necessarily a concomitant of this music. That one could apply
some of the quasi-mathematical (never more than
quasi-mathematical) formulas that he worked out and apply these
to sound formations that were really quite consonant, that were
triadic, that were in fact precisely the sound formations that
every composer from the Renaissance onward had worked with. So
that what you find in the late years of Schoenberg is this
extraordinary coalescence of ideas that were extremely radical,
in the sense that he was saying, "I do not believe the
arithmetical components of music as it's been practiced in the
generation or two before my own. I do not believe that these
are any longer serviceable," and at the same time extremely
conciliatory in that he was saying, "Look, there is a way to
take very old sound formations, very consonant ones, totaly
undissonant ones if you like, and organize them in a
mathematical relationship that gives them maximum expressivity
and minimum debate against each other," which was the formation
of tonality. So I see Schoenberg more as a man affecting a
conciliation than as a man overturning things.
Obviously this is a little more technical and I wouldn't be surprised if this is more of a sticking point for you, so I'll get into this in more detail.
First: serial music is really not so complicated, essentially speaking. The basic idea is that you take a set of possible musical actions and arrange all of them in some order. Then when you're composing, you use those actions in that order. So, if the musical actions are the 12 tones of the traditional Western chromatic scale, you would arrange all 12 of them in some order (called a "series"), and then every time you wrote a note you would follow it with the next note from the series. Like, if A was followed by C#, then whenever you wrote an A in a certain voice, you would make sure C# came next in that voice. They don't have to have the same note value and they can even be in different octaves and such, but just in terms of pitch class, they should follow the series.
The idea behind this is to make sure that you use all 12 tones equally much, instead of giving prominence to any one of them. Before Schoenberg devised this technique, he was writing "freely atonal" music, which had no key signature but also no overarching formal principle akin to tonality or the like (this was his first piece in that idiom, specifically the fourth movement). Music written in this manner can still end up kind of tonal in one way or another, so the 12-tone technique is a more rigorous approach to atonality.
However, that technique, in and of itself, says nothing about consonance or dissonance, or even passing moments of tonality. Note that I never said, "Make sure you can't form a major triad by lining your series up against itself a certain way," or something like that. However, many following serialist composers became very adamant about avoiding even the faintest whisper of tonality in their music, and even moved to serialize other aspects of music like rhythm and dynamics (see Boulez's Polyphonie X for an extreme example of this). The result was music that was very suspicious of consonance, rhythmic pulse, repetition of any idea smaller in scope that the series, etc.—a much more extreme break with the musical past than plainly-stated serialism requires. These composers generally felt that they were being true to the spirit of serialism, but of course that's a matter of perspective.
Schoenberg himself did not really share that perspective, and in fact he became rather unfashionable in mid-late-century serialist circles in favor of his pupil Webern, who pioneered this more "pure" or "severe" (depending on your outlook) approach to serialism. This isn't surprising considering how Schoenberg arrived at atonality to begin with: he started out as a well-regarded late Romantic composer, and said he moved to atonality because total chromaticism was the only place left to go after composers like Wagner and Mahler pushed the boundaries of tonality so far. He wasn't necessarily trying to be an iconoclast, nor was he driven by some sort of all-encompassing abstract ideology about music.
Gould is making the case that he was even more like this in his later years than in the decades prior. As an example of the sort of music of his Gould might be referring to, you could check out Ode to Napoleon Buonaparte, which has a pronounced tonal character despite its serialism, even ending on an Eb major chord that's approached in a manner recalling his late Romantic period. There's a lot of classicism in it, not just in terms of its harmony but also some of its rhythmic figures and so on. That might be the sort of thing Gould is getting at, that he found ways to use this novel and math-ish (lots of terminology from set theory in serialism) composing strategy to get at very traditional ideas in Western music.
In terms of his general attitude around that time, he even returned to that late Romantic style with stuff like his Chamber Symphony No. 2, which he had started writing a couple years before his first atonal pieces. Also, he was a music professor in his later life and wrote some very good textbooks covering the styles of the common practice period—at least of what I've read of his output, he spent far more words discussing the techniques of composers like Mozart and Beethoven than he did describing how to write serial music.
So, in a way, he's not such an enfant terrible as he's made out to be. That reputation might be more reasonable to confer on some of his successors, like Babbit, Boulez, Stockhausen, and so on, who were more determined to totally upset the Romantic apple cart.
EDIT: As a postscript, I figure Gould says "quasi-mathematical" because serialism is not truly mathematical, in the sense that it lacks axioms and proofs and what have you. It has quantitative procedures for constructing and manipulating series, but there's nothing unusual about Western music theory having a quantitative flavor even going back hundreds of years. what maybe distinguishes serialism is that it also lifted a fair amount of terminology and a handful of ideas from set theory, since the elements used to construct a series (like the steps of the chromatic scale) can be thought of as a set, and a series as an ordering of the set. That still doesn't make it math, though, just "math-inspired". :P