I would assume that many would agree that if we have a piece of music and there are no sharps or flats in the key signature that in most cases (if not all) the piece would be considered in the key of C major or A minor. This question assumes the context of Western Music since that is the only type I know much about.
The key signature defines which notes (in the absence of accidentals) are available to include in the composition. So obviously if there are no sharps or flats in the key signature - without accidentals any note written on the staff will be one of C D E F G A or B (all natural) which are the notes in the key of C major and also the relative minor - A minor.
But if we change the tonal center from C to another note, without changing the available notes, then we are in a different mode. For example if we start on F and play a scale using all the notes in C major (resolving back to F), I believe that would be considered the F Lydian Mode (even though the notes are all common to the C major scale and derived from the key of C major).
So does that mean that F Lydian Mode actually belongs to the KEY - of C? Similarly does each mode in the chart below which are all comprised of the notes found in the key of C major/ A minor, belong to either the key of either C major or A minor? Certainly the key signature would indicate as much.
I know that the Aeolian Mode is equivalent to the A natural minor (A minor) scale - so that makes me believe that modes must be defined as being derived from either a major or minor parent key.