As suggested in a comment, there is an acoustic aspect to consonance and dissonance:
Harmonic partials
In (perhaps overly-) simple terms, the sound of a single note is composed of many individual frequencies. Collectively, these are called "partials". For most pitched musical instruments, the partials are (at least approximately) "harmonic", meaning they occur at frequencies that are whole-number multiples of the fundamental frequency. (For example, an A2 contains 110 Hz, 220 Hz, 330 Hz, 440 Hz, etc.) The partials generally decrease in amplitude with increasing frequency.†
Consonance
When two notes are played together, one aspect of the perceived consonance between them arises from the alignment of the partials in one note with those of the other.
Fewer partials align in a perfect fourth than in a perfect fifth.
Further details
The intervals analyzed for consonance below are based on just intonation (JI), which uses whole number ratios between pitches. The equal temperament (ET) system in wide use now is a compromise based on the uniform, irrational ratio of 21/12 between semitones. The ET equivalents are close, but the JI ratios make for neater presentation of the numbers.‡
An octave corresponds to a frequency ratio of 2/1. The perfect fourth corresponds to a ratio of 4/3 and a perfect fifth is 3/2.
Consider the fundamental of the root note to be 1. (It doesn't make a difference which basis frequency we use, so this just makes the numbers easier to read.)
The root note contains partials at these relative frequencies:
1, 2, 3, 4, 5, 6, ...
A note one octave higher contains these relative frequencies (all partials of the root note, each multiplied by 2):
2, 4, 6, 8, 10, 12, ...
You can see from the above that every partial of the octave corresponds exactly with a partial of the root.
A note a perfect fifth above the root contains these (each multiplied by 3/2):
3/2, 3, 9/2, 6, 15/2, 9, ...
A note a perfect fourth above the root contains these (each multiplied by 4/3):
4/3, 8/3, 4, 16/3, 20/3, 8, ...
I've bolded the partials of the perfect fourth and fifth that align with partials of the root.
Every second partial of the perfect fifth aligns with a partial of the root, while only every third partial of the perfect fourth aligns. The perfect fifth has more partials that align and sounds more consonant as a result.
For comparison, let's also look at another consonant interval, the major third (with a ratio of 5/4) and its less-consonant inverse, the minor sixth (with a ratio of 8/5):
A major third contains these (multiplied by 5/4):
5/4, 5/2, 15/2, 5, 25/4, 15/2, 35/4, 10, ... (Every fourth partial aligns.)
A minor sixth contains these (multiplied by 8/5):
8/5, 16/5, 24/5, 32/5, 8, ... (Every fifth partial aligns.)
In addition to the quantity of partials in alignment, consider that the higher-order partials are generally weaker in amplitude and contribute less to the perception of consonance or dissonance. The more-consonant intervals contain alignments of lower-order partials.
Notes
† Some instruments (e.g. pitched percussion) may have significant partials that are substantially inharmonic. Some instruments (e.g. piano) have partials that are slightly inharmonic. The relative amplitude of the partials varies over time and some may be entirely absent. The composition is a large part of the specific timbre of each instrument
‡ An ET perfect fourth has a ratio of about 1.3348, a bit sharp of JI. An ET perfect fifth has a ratio of about 1.4983, a bit flat of JI. The third and sixth are further out-of-tune in ET.