Keep in mind you could have a piece of music with sections that switch between modal and major/minor harmony. But let's assume the music is consistently one way or the other.
The seventh scale degree is critical to understanding modal versus major/minor. Let's dig into that idea starting with major/minor music.
The ending of a piece of music is traditionally where you will get confirmation of a major/minor work's key. If the music ends with a traditional cadence, it's using the major/minor system and not modal.
You can continue examining cadences within the music. Pieces in major/minor will have various internal cadences marking the end of sections. Again, if you find traditional cadences, it isn't modal.
The defining feature of "traditional" cadences is the appearance of the leading tone scale degree. That is the seventh degree of the scale which is one half step below the tonic. The leading tone will appear in cadences as either
I to V with the leading tone going up to the tonic, or as a phrase ending on
V where the leading tone will be present in the
When we look at all the modes except Lydian - Dorian, Phrygian, Mixolydian, Aeolian, and Locrian - the seventh scale degree is a whole step below the tonic. By the definition above these seventh scales degrees are not leading tones. Instead we can call them subtonics. Let's skip over Lydian for the rest of this, because it complicates things in a way that I don't think will help you right now.
When looking at modal music we should see the subtonic instead of the leading tone in melody or harmony.
For example, we might see melodies ending
C to D in
D Dorian rather than
C# to D which would be used in
Instead of seeing chords like
V to i we might see
(minor triad) v to i in Aeolian or
(minor triad) v to I in Mixolydian. In major/minor the chord of the leading tone - a diminished triad - might be seen like
viio6 to i. But in modal style the chord of the seventh degree is not a diminished chord. It often will be a major triad like
VII to i in Dorian or Aeolian.
I just used a bunch of musical terms that may be unfamiliar to you. You will eventually learn them as you study the modes. But one important thing to look for, one idea to simplify the topic, is checking the seventh scale degree. In modal music it is usually a whole step below the tonic.
Regarding scales like double harmonic or freygish those should be pretty straight forward to identify, because the tones in such scales are not diatonic. In such exotic scales the seventh degree may be a half step below the tonic - that would look like the leading tone described above - but the non-diatonic nature of the scales will probably involve accompaniment with chords different than the major/minor system. It's hard to make a simple generalization. But the exotic scale should stand out as not diatonic and therefore not belonging to the major/minor system.