I'm writing a piece of music in 6/8 time signature. Is it correct to have a bar with 3 quarter notes in this time signature?
This is technically not allowed, but it's one of the common exceptions to the rules that is widely accepted, as long as it's exactly this rhythm.
Here's one way to explain where notes are allowed to be. There are multiple equivalent ways to state the rule, so you may have seen something else and that's okay. It's easiest to describe in simple meters like 4/4, so I'll start there:
First, consider the natural subdivision of notes for the time signature:
Terminology: The whole note is the "parent" of the half notes (who are "children" of the whole note). All of the notes pictured are in their "natural" positions and are of course allowed to be there.
A non-dotted note can also be in a position which is offset by one of its (direct) children, but still within the bounds of its naturally-placed parent. Consider these four examples:
In the first example, the quarter note is offset by an eighth, but within the bounds of its parent half note (in its natural position), so this is correct. In the second example, the note is offset by an eighth, but breaking the bounds of its parent, so this is not allowed, and it should be written as the third example. The fourth is another example of a commonly accepted exception to the rule, as long as it's exactly this rhythm (replacing the first quarter with two eighths or anything else would make it invalid). Offsetting a quarter note by a sixteenth or smaller is always wrong.
For note durations that are not part of the natural subdivision of the time signature, they may only sit in a position fully within the next longest natural duration, and shoved all the way up against one end of it. Examples:
The first example shows the two ways that a dotter quarter note can validly exist in a 4/4 measure--occupying either the first or last three-fourths of a naturally positioned half note. The second example is technically invalid but very commonly used, and whether or not it should be tolerated is contentious. I personally hate it but many people have used it to the point of ubiquitousness. It is properly written as the third example. The fourth is also a contentious potential exception.
In 6/8, the natural breakdown looks like this:
Quarter notes are not one of the naturally occurring durations, so technically they can only fill the first or last two-thirds of a dotted quarter.
The first example is the technically correct way to write this rhythm, but the second is an accepted exception.
It's not incorrect, but it should be understood that you're temporarily subverting the normal grouping of notes. Think about this: you could give a 3/4 piece a 6/8 time signature, and that would be incorrect. There would be the right number of 8th notes in each measure, but they would be grouped in 3 sets of two notes ("simple triple" meter: "triple" because there are three beats, and "simple" because each beat is divided in two). 6/8 is normally understood to be "compound duple" meter, having two beats divided into three subdivisions. But composers often play with its ability to "go both ways"; Tim mentioned "America," which basically alternates between "1 2 3 1 2 3 | 1 2 1 2 1 2 |" throughout the song. In "Mercury" from The Planets, Holst alternates in 6/8 between measures of three quarter notes and six eighth notes (and, just to complicate things, sets it against a backdrop of 2/4) (https://scholarworks.calstate.edu/downloads/41687m324, p 49 ff).
The Grade 5 theory examination answer is that three quarters in a 6/8 bar should be grouped like this.
In modern practice, we have the choice. To emphasise the cross-rhythm against a continuing 6/8 pulse, you might still do it that way. But if you'd been tempted to change to 3/4 for that bar but decided (quite reasonably) not to clutter the score with continual time signature changes, three quarter notes would be fine. And would probably now be considered the norm.
Here's the classic example:
Strictly speaking, it's naughty! But read on... The whole point in 6/8 time signature existing is to differentiate from 3/4. Both contain the same value as far as notes are concerned, but the more than subtle difference is the grouping, making the feel of each very different. 1 - 2 - 3 - 1 - 2 - 3 -is the feel of 3/4, (or 1 & 2 & 3 &) whereas 1 - 2 - 3 - 4 - 5 - 6 - 1 - 2 - 3 - 4 - 5 - 6 is the feel of 6/8. (Also counted as 1 & & 2 & & ) This splits the bar into two equal halves, which is not what 3/4 is about.
So, if a piece has its basic feel as one or the other, that's the time signature to use. However - since 6/8 is essentially two groups of three quavers, writing the three crotchets is best done using 'crotchet, quaver-tied-to-quaver, crotchet'. That then keeps the division within a bar 'correct'. Although most readers will be able to understand what's going on with a simple 3 crotchets, as is used in several pieces already mentioned.