(at the bottom of this answer, there are youtube example videos with sound)
Definition of modes
Different modes produce different harmonic feelings. These differences are caused by the notes of the scale having different distances to the home note than in the "normal" major or minor scales you've used to hearing. Because of these differences, the set of chords existing in the mode is different from what you're used to. The modal feeling has two ingredients that have to be exactly right for each mode: (1) the set of chords/notes, and (2) the home note i.e. tonic.
- the chords/notes of the C major scale + tonic at D --> D dorian mode
- the chords/notes of the C major scale + tonic at F --> F lydian mode
- the chords/notes of the C major scale + tonic at G --> G mixolydian mode
How to break the mode
The set of chords and the home note are very specific in each mode. If you play the wrong chord, or if you mess up the feeling of where the home note is, you're not in the same mode anymore. You broke the modal feeling. So, here are the two ways to break the mode:
- Wrong chord: using a chord that has a note that doesn't belong to the mode's scale. If you play a chord that doesn't belong to the mode, you might stay within the same key, as in, the tonic stays the same, but the modal feeling is changed. For example if you're in A dorian mode that has Am and D chords, but if you play a Dm (or F) chord, you break the dorian feeling. There is no F note in the A dorian scale, and no Dm chord in the A dorian harmonic feeling. It might still sound perfectly fine and beautiful, and the tonic probably doesn't move, and it may be some other mode, just not A dorian. In regular major/minor tonalities in pop music, it doesn't matter if you play D major or D minor every now and then, you can still say that you're in the key of A minor. But if you play a D minor (or F) chord, you're not in A dorian mode anymore, at least for the duration of that "wrong" chord. You can get back to the modal mood by continuing to play the right chords (and notes) for the mode.
- Wrong tonic : if you play something that leads the listener to think that the home note is not where it must be in that mode, then you're not in the original mode anymore. For example, if you try to play G mixolydian which has the chords G, Am, Bdim, C, Dm, Em, F ... if you put the bass in C and move the melody to C as well, chances are that it might sound like the tonic is actually C, and so you're not in G mixolydian anymore, even though the set of chords and scale notes remains exactly the same. In G mixolydian, the tonic has to be G. If the tonic moves, it might be perfectly beautiful music, and it might be some other mode, just not G mixolydian.
In your unsuccessful experiments with Band-in-a-Box, I suspect that if you just typed in chords from a list, it also meant that BIAB played the corresponding bass notes for each chord, making it much more difficult to retain a modal feeling. Try using bass inversions for creating a pedal tone instead, by having the same bass note for all chords, e.g. Dm - G/D - F/D - Em/D - C/D - Am/D - G/D - Dm. This progression should create a D dorian sound, as long as BIAB doesn't add any out-of-scale notes.
Changing or creating a modal feeling deliberately
In the key of C minor you normally have Cm and Fm chords, and C being the "tonic" i.e. home note, where a melody feels to be at rest, and a natural ending for a song. C minor has three flats in its key signature and the notes of the scale are C, D, Eb, F, G, Ab, Bb. In C dorian, C is also the home note, but instead of F minor, you would have e.g. F major or D minor chords, and for melody lines, A natural instead of Ab. C dorian has the notes C, D, Eb, F, G, A, Bb. The A natural is the "dorian note" for C.
You can basically change a "normal" C minor song to C dorian modal feeling by changing all Ab notes to A naturals and all F minor chords to F major chords, provided that the original melody only uses notes from the C natural minor scale. For example, if the song uses a G major or G7 chord somewhere - with a B natural note - then at least for the duration of that G chord the song steps away from a possibility for a C dorian feel. The same goes for any other "out-of-scale" chord like D7, which contains an F# note but is common in songs in the key of C minor. Some other modes could have an F# note, but C dorian doesn't.
The whole song doesn't have to use the same mode. Different sections of a song can use different modal moods for short periods of time, even as short as a single bar or half a bar. If you have a song that's in C minor, and if it has a section of just Cm chord and with no Ab in the melody during that section, you can deliberately create a dorian feeling to it by e.g. alternating between Cm and F chords, or Cm and Dm chords, instead of just the Cm. Tricks like this are used by guitarists and other players. If the chords and melody only use the six other notes of the minor scale, it's a chance for a creative player to pour some dorian sauce on it.
Another example. If the song is in A minor, and the written melody and chords only use the six notes A, B, C, D, E, G ... then you can throw in some quick D major or B minor chords, because the "F natural or F sharp" decision is left open and for you to play with.
The challenge is, you have to know (or hear) what notes and when the melody and any other players are going to use, and if they're going to clash with your modal endeavors. For example if the song is in A minor and you'd like to twist it into A dorian, if there are chords like Dm, F, B7, C7, G7, those are going to more or less step on your dorian mode's toes.
Dorian mode is probably one of the easiest examples, and it's very widely used. Another mode that's quite easy to do for a chord-player guitarist is mixolydian. For example in a basic pop/rock song that's in A major, to give it mixolydian feeling, throw in an A7 or G major chord every now and then, except in places where there's an E major chord, or any of the out-of-scale ones like B7. A mixolydian has the G natural note, instead of the G# that A major songs normally use.
For introducing these modal feelings, here are a few guitar chord shape examples.
To twist a song that's originally in the key of A minor into A dorian modal feel, alternate between Am and Bm/A chords when there's originally an Am chord:
In regular A minor key, a corresponding alternating chord pattern would be Am - Dm6/A (or Bdim/A), but in the A dorian mode it is Am - Bm/A. The F# note played with the 4th fret of the D string is the "dorian note", when in regular A minor you would have F natural.
For a slightly different feeling, alternate between Am7 and this other chord which I won't try to name. (well... Dadd2/A maybe?)
To twist a song that's originally in the key of A major into A mixolydian modal feel, alternate between A and Em/A chords when there's originally an A major chord:
In regular A major, a corresponding alternating chord pattern would be A - E/A, but in A mixolydian it is A - Em/A. The open G string is the "mixolydian note" here, but in regular A major you would have a G#.
To twist a song that's originally in the key of A major into A lydian modal feel, alternate between A and B/A chords when there's originally an A major chord:
In regular A major, a corresponding alternating chord pattern would be A - Bm/A, but in A lydian it is A - B/A. The D# note played with the 4th fret on the B string is the "lydian note", when in the regular A major scale you have a D natural.
Here is a small etude in A lydian, constructed entirely with chords, and with the open A string as a pedal tone. To play a lydian mode solo you could basically use these chord grips and pick just one or two of the notes at a time.
You might notice that all of the chords are ones that are often used in songs in E major. If we use the chords and notes of E major, but so that the tonic i.e. home note is A, it creates an A lydian sound. The key signature, four sharps, is what you'd normally find in E major (or C#m minor).
If we take the same chords, but move the pedal tone (the bass note that stays the same all the time) from A down to F#, we get a dorian mode feeling. YMMV, but the pedal tone is meant to make it feel like that's the tonic i.e. home note.
(Theory enthusiasts might want to explain modes a bit more pedantically, but in this answer I tried to introduce modal feelings from a chord-playing perspective to someone who's familiar with guitar chord accompaniment, and perhaps not so much into soloing and playing individual notes.)