The tonal center of a piece cannot be independent of its melody, and rhythm alone cannot determine a tonal center. Drum cadences are made purely of rhythmic unpitched drum parts, and they are, predictably, atonal.
However, a monophonic, single track of a capella raw vocals with no accompaniment can - and often will - still have a tonal center. Take the first few measures of a melody-only rendition of "Joy to the World" that starts with C, for example - it starts with a descending C major scale with C notes lengthened, and then only notes in the C major scale appear afterwards. It's very hard to say that "Joy to the World" that starts with C is not in C major.
"Joy to the World" is arguably an abnormal case, but the single melody track will often narrow down the tonal center to a few possibilities at most. The tonal center must not clash with the notes in the melody, especially the most strongly emphasized ones. (The second part of my last sentence is important for those melodies with all 12 notes in the chromatic scale, such as the melody of "You Will Know Our Names" from Xenoblade Chronicles.) Even a single-note drone will yield that its most likely tonal center is that note.
Note that I implied above that the tonal center of a melody can often be ambiguous, but be restricted to a few possibilities. Multiple harmony tracks can indeed be added to help narrow down to one unambiguous tonal center. For example, the normally D-major regimented melody near the start of Edward Elgar's Pomp and Circumstance March No. 1 in D Major is reharmonized into B minor in "In the Whirlpool" from Dance Dance Revolution: Mario Mix. I bet you anything that reharmonizing that melody into D sharp minor or A flat major without changing any of the notes is going to be extremely hard - this is because those keys prominently use notes not in that melody such as D sharp and A flat.
You can compose a melody made of notes strictly from the C major scale and have the tonal center be B flat. In order for that tonal center to be convincing instead of giving listeners the impression that you're "playing outside" the chord progression, your choice of notes from the C major scale will probably be somewhat restricted. You'll probably avoid E and B (except possibly as non-accented passing tones and neighbour tones) because those notes are neither in the B flat major scale nor the B flat minor scales.
My example "melody made of notes strictly from the C major scale with a B flat tonal center" is F-G-F F-G-F F-G-F D-C-D, which can be harmonized with a chord progression of B♭ B♭ F7 B♭. Take notice that every note in my example melody can be found in both C major and B flat major, though.
It is still true, though, that a chord progression that a piece strictly uses throughout may not be sufficient to predict the scale - especially its tonic. Depending on where the chord progression starts and ends, the tonal center can be different. It's easier to interpret a chord progression of looped Am F C G as being in A minor and one of looped C G Am F as being in C major instead, especially if the former progression ends with an Am chord and the latter with a C chord. Do note that A minor and C major scales share most of, if not all, their notes.
A monophonic melody can easily have a key that includes all the notes in that melody, especially its most strongly emphasized (longest, often repeated, stressed at cadences) notes, and its tonal center will fit in that key. Accenting on particular syllables can easily change which notes are the most strongly emphasized and therefore which notes fix down the tonal center. The same applies for a chord progression - it can easily have a key that includes all the notes in all its chords, especially root notes of chords and contents of chords that end cadences, and its tonal center will also fit that. However, it is very hard to perhaps impossible to compose a piece where the melody has notes from a particular scale, but the tonal center is made to be a note not belonging to the scale, because it is very hard to impossible to find the appropriate bass and chords that both switch the tonal center to a note that is not in any major or minor scale with all the notes of the melody in it and do not give off the impression that you are "playing outside" and refusing to fit the chord progression. ("Playing outside" often, if not always, involves dissonant intervals between melody and chord notes.)