Math Alert! Also, I will be very much discussing what is theoretically possible, not necessarily what is convenient for the poor musician.
The notation for musical rhythm is more or less equivalent to writing a fractional number in binary (e.g. using a radix point). Each note type represents a different place value. For example:
- Whole note = 1.02
- Half note = 0.12
- Quarter note = 0.012
- Eighth note = 0.0012
- and so forth...
Since you can (theoretically) add an arbitrary number of flags to a note head, you can carry this out as long as you please.
Dotted notes (single or double) represent adding two (or three) adjacent places:
- Dotted half note = 0.102 + 0.012 = 0.112
Ties allow you to add any place values, regardless of if they are adjacent:
- Half note tied to eighth note = 0.1002 + 0.0012 = 0.1012
Since, given enough place values, any numerical value can be represented in binary (i.e. a sum of the value of each place), the answer to your question is technically, no, there are no values that cannot be represented. However, this is not practical, since many numbers would require an infinite series of digits to represent. These can be broken into two classes of numbers (as you allude to in your question) -- rational and irrational.
Tuplets can be used to handle the rational case. If ties represent addition, then an n-tuplet represents division of a time unit into n equally-sized portions. Any rational number can be written as a fraction, m/n, which has the same value as 1/n added to itself m times. This number can be represented musically by tying m copies of an n-tuplet. For example:
- 31/87 can be represented by tying together 31 notes that are 87-lets.
However, in the irrational case, the value cannot be represented as a ratio or fraction, so tuplets don't work. This leaves you back to writing out an infinite series of tied notes, which cannot be improved upon because irrational numbers require an infinite, non-repeating representation.
The final question that must be considered is the precision required. While theoretically, you would require an infinite number of notes, with an increasing number of flags on the staff, the human ear is limited in how precisely it can detect rhythms. At a certain point, rounding off becomes inevitable. For example, even NASA requires only 15 digits of pi (in base ten) in order to calculate the positions of interplanetary probes. Digits beyond this are insignificant enough as to not matter. Similarly, any number inside a computer must be rounded off to some finite approximation, simply because of limited memory.
So if you limit yourself to a basic unit of precision (perhaps a 128th note?), then you can represent any multiple of that basic unit (using ties), as well as any rational number (using tuplets).