Assume that the whole note time equals 8. Then a note which time equals 3 will be written as 1/4 note and dot. How you write down a note that time is equal 5/8 of a whole note?
I think I've figured this out
So to write such a note one would have to use a tie
Write a 1/2 note and 1/8 note and tie them together
As you have said, you have to use a tie for this. You can tie any combination of notes that make up 5/8ths.
However, it's also true that you can (and should) give a visual indicator of how you want the notes grouped metrically by which ties you use for the longer values. For example, if you accent the first and third 8th notes (very common in 5/8 time), use a quarter tied to a dotted quarter. If you accent the first and fourth 8th notes, use a dotted quarter tied to a quarter. And so on.
There's no standard way to notate this duration without ties (as OP suggests). George Crumb uses the notation of a half note with a dot on both sides of the note for the duration (see: ).
You could also of course write a whole note with the tuplet [8:5] above it (any rational number duration can be written as a single note with the appropriate tuplet).
It is possible to write with standard notation a rest worth 5/8ths of a whole note: just write a whole rest in a measure of 5/8. :-)
You were on the right track, but because 5/8 is an asymmetrical meter, you have to group the tie by smaller groups of the meter. In an asymmetrical meter the notes in a measure are grouped in smaller groups of two or three. The possible grouping for 5/8 are 2 eighth notes + 3 eighth notes and 3 eighth notes + 2 eighth notes. When you tie notes together, you have to abide by the groupings so the tie would be expressed as a quarter note tied to a dotted quarter note or a dotted quarter note tied to a quarter note depending on how the song groups the eighth notes.