In most cases (if not all), a piece of music without sharps or flats in the key signature, would be considered in the key of C major or A minor. This question assumes the context of Western Music since that is the only type I know much about.
The key signature defines which notes (in the absence of accidentals) are available to include in the composition. So obviously if the key signature has no sharps or flats, any note written on the staff will be one of C D E F G A or B (all natural), which are the notes in the key of C major and also its relative minor (A minor).
But if we change the tonal center from C to another note, without changing the available notes, then we are in a different mode. For example if we start on F and play a scale using all the notes in C major (resolving back to F), I believe that would be considered the F Lydian Mode (even though the notes are all common to the C major scale and derived from the key of C major).
So does that mean that F Lydian Mode actually belongs to the key of C? Similarly does each mode in the chart below (all comprised of the notes found in the key of C major/A minor) belong to the key of either C major or A minor? Certainly the key signature would indicate as much.
I know that the Aeolian Mode is equivalent to the A natural minor scale. This makes me believe that modes must be defined as being derived from either a major or minor parent key.