My answer builds on the answer contributed by DR6.
Based on your reaction to other very good answers posted here already, your question seems to boil down to: "Why do humans innately feel that certain intervals are consonant". And so much so that they are willing to call them "perfect". Before getting to that question, let's look at why Western culture might consider them "perfect". My answer to your question will be rather freeform because the truth of the matter is there is not really good answer to your question outside the music theory-based explanations given above.
The modern Western music system has been inherited from some of the groundwork set by Pythagoras. It has been heavily modified to the point now that the modern 12-tone equal temperament we use now has the spirit of the original ideas from Pythagoras even if it differs greatly in many other ways. To Pythagoras, and possibly many Greeks at the time, certain intervals sounded very pleasing to the ear. Mathematically, these intervals are superparticular ratios [(n + 1)/n) or multiples [(x*n)/n]. For example, 4/3 is a superparticular ratio and 3/1 is a multiple. In other words, when the two frequencies resonate together and the ratio of the frequencies comes out in either of these forms many people in Western culture would agree they are pleasing. The perfect ratios display this quality in the best sense: 2/1 is an octave, 3/2 is a perfect fifth, and 4/3 is a perfect fourth. There is the least amount of conflict in the frequencies between the notes allowing for more complete symmetrical intersection between the waveforms. This is probably why Pythagoras liked these intervals - the Pythagoreans loved this kind of mathematical perfection. He liked it so much he tried to develop a tuning system out of it (Pythagorean Tuning) which ended being impossible without introducing a tuning error (the Pythagorean Comma).
I am not too clear on how Pythagoras's discoveries exactly carried over through time but his ideas were often used and cited by other musicologists through time. One example is Ptolemy who created scales based of Pythagorean tuning that included other less consonant intervals (thirds). What I am getting at here is that our assumption of the "perfect" intervals derives from the fact that the system's originator (and possibly his culture) deemed them to be perfect. It's hard to say why the name persisted through time but needless to say, thousands of tunings systems were developed after Pythagoras, most of which tried to preserve the perfect fifth, fourth, and the octave while allowing wiggle room for other intervals to fit together in the scales (I'm oversimplifying but that's the idea).
But is it pleasing to humans in general? That depends. Many cultures developed other systems that don't necessarily have this obsession with the perfect intervals or used many others equally. Other cultures (Persian music) have divided the octave into 53-tones, 24-tones (some forms of Indian music), and other divisions. One response to this is that the majority of non-Western cultures tended to develop music systems that were melodically complex: complex scales over a single droning note, but not harmonically complex like Western music. So perhaps they never needed to develop the notions of "perfect" in the first place. There is also the fact that in the modern era we have become increasingly attracted to dissonant or unusual forms of harmony. There is widespread interest in rock/metal which emphasizes distorting the sound wave to emphasis dissonant overtones (even if the intervals actually played are quite consonant). Dubstep is not exactly harmonically pleasing either but it is popular. Modern Jazz uses some complex and dissonant forms of harmony. A lot of 20th century classical music is also very dissonant. The question comes down to if it's a matter of taste, the unexpected (things that surprise us make things interesting, a change from regularity), culture/social norms, or if it's innate. There's also a difference between enjoying dissonant music and actually finding it pleasing. I love dissonant music but I don't really find it more "pleasing" than consonant music - I like it because it is jarring.
Music psychology and cognitive neuroscience has not come to a firm conclusion on this question. There have been a lot of studies on this topic but none are quite conclusive. One simple explanation is that evolutionarily, the human brain learned to find patterns and structure to apply semantic meaning. This means that we seek things that have regularity and predictability and attempt to assign meaning to things to help them to fit within these frameworks. Dissonant music deliberately goes outside predictable frequency ratios that line up, producing uneven sounds. Perhaps the aversion to these sounds is a by-product of the general manner in which the brain functions in the world.
But this is a post hoc explanation. Cognitive neuroscience has been asking these questions for a long time and modern advances in computational neuroscience may soon provide an answer. A simple look at this question can be found in this Nature article.
To summarize: We probably call it "perfect" because of Pythagoras and musicologists that came after him. We probably think it's "perfect" for cultural and social reasons. If it is really "perfect" to us innately is to be determined.