I've always wondered what makes these particular intervals so difficult.
What makes these intervals so difficult to sing is that they are very dissonant. Dissonance is used all the time in music to set-up tension that will later be "released" by a consonant interval. Augmented fourths and diminished fifths (which are also enharmonic tones) are the two most dissonant intervals.
They aren't hard to sing at all. Sing the opening of 'Maria' from West Side Story. Tonic, sharpened 4th, 5th. (No need to argue over whether it's a #4 or b5 for now.) Easy, if you know the song. There's not nearly so much 'hard' in music as merely 'unfamiliar'. Ask a singer to perform an exercise based on octave jumps. They might make it sound difficult. Then ask for the first two notes of 'Over the Rainbow'. Easy, for the most elementary singer.