You can't really differentiate between them. 4/4 is most often emphasized like this:
ONE two three four
6/8 like this:
ONE two three four five six
If you play triplets in 4/4 you get the exact same pattern as in 6/8. Similarly if you play dotted eighths in 6/8 you get the same pattern as 4/4.
In general you should pick a time signature for the effect it gives you when writing "naturally" in that time. What I mean by this is that "swaying six eight feel" you describe is related to having three notes per beat and more emphasis on the first beat per bar. 6/8 has three notes per beat by default, so you should use it if you want that effect rather than using triplets in 4/4, even though both have more emphasis on the first beat per bar. If you want two notes per beat, use 4/4. If you're going to have some mixture, use whatever makes it easier for you to write.
As noted in another answer, time signatures are arbitrary. Trying to differentiate between time signatures is futile because of how different people can perceive them, among other reasons. The change in perception you mention is different for everyone and many people won't even be aware of the time.
Your goal should be to produce the particular effect you desire, not to have a clear time signature recognizable to the listener. Time signatures are only a tool to make writing and reading music easier.