Okay, I hit google and did some research.
The book you want to read is The Saxophone by Stephen Cottrell. It has a chapter, eight, "The Saxophone as Symbol and Icon".
As I thought, long before the sax was associated with love, it was associated with sex, and it was substantially sexualized because of its use in early jazz, and thus it got tangled up in American and European racism. Cottrell's book is not wholly readable for free online, but just what's available through Google is pretty informative. Here's some chunks.
BTW, you are not wrong to wonder about the role of its timbre...
The saxophone was developed to address specific problems that Adolphe Sax had identified among low wind instruments, and his solution to these was made possible in part because of nineteenth-century advances in engineering [...] From its inception, therefore, the instrument has been identified with modernity, innovation, and a sense of exploration and enquiry, and this reputation has in many ways remained with it since. These characteristics were explicitly promoted in the nineteenth century by Jullien and others in their popular concerts in Paris, London, New York and beyond, in which musical novelty was combined with astute showmanship for commercial gain. Here the instrument's unusual morphology and its unfamiliar timbre were harnessed to great effect, imprinting the saxophone in the minds of the audience as something new and exotic. [...]
But then things took an interesting turn.
The saxophone's close association with dance music and jazz in the first half of the twentieth century has been without question the single most important reason for its present widespread familiarity. But for much of the 1920s and 30s particularly, and for many years thereafter, the instrument suffered because of this. The arrival of these new styles was accompanied by an outraged response from the moral majority (as is so often the case in the development of Western popular music -- consider also rock and roll, punk, hip-hop, etc.), who saw in them evidence of every kind of youthful dissolution. The opposition to what was construed as 'jazz' was especially fierce. By way of example, in 1925 the editor of Etude magazine observed that 'we know that in its sinister aspects, jazz is doing a vast amount of harm to young minds and bodies not yet developed to resist evil temptations. perhaps this is the explanation of America's enormous crime rate at the present'. Such sentiments were widely expressed in musical publications of the 1920s through to the 1940s, and more occasionally beyond.
What were taken at the time to be somewhat lascivious dance styles or, in the case of jazz clubs, the identification of nascent counter culture seemingly involving drug consumption, promiscuity and other ills, inevitably resulted in a backlash against the musicians promulgating such music and the instruments they played. The saxophone suffered particularly in this respect. Whereas the trumpet, trombone, double bass or piano were all established, 'legitimate' instruments with a strong classical background, and could thus perhaps be viewed as only temporarily errant when found in more degenerate contexts, the saxophone had no such heritage to call upon. And since the unfamiliar sound and shape of this relatively new instrument took a dominant roll in many of the musical proceedings, it could clearly be identified as the principal miscreant, at whose door much of the blam for this decadent musical behavior could be laid.
Consequently, a series of profoundly negative views of the saxophone arose, which can be loosely divided into two related categories. In the first was a conservative musical establishment [...]. The second category comprised of those who appropriated for themselves the role of moral guardian, and were thus less concerned with the implications for Western musical culture than for the welfare fo society as a whole. For this group it was what they perceived as the essentially sinful or evil nature of dance music and jazz that was most troubling, and in their view the principle architect of such depravity was again the saxophone.
Skipping ahead (page not available in preview, alas)
The religious establishment was still suspicious of the isntrument as late as 1948 in Worcester, England, when a performance of Vaughn Williams's Job has to be given without the movement containing the prominent saxophone solo, because the church authorities refused to allow the profane saxophone inside the church where the performance was due to take place. And inevitably, when moral turpitude looms, it is the honour of their young women that societies frequently deem to be most at risk. In his book on music censorship in America, Eric Nuzum notes that 'when the saxophone was popularized in the 1920s, critics called it the "devil's flute" and thought that its low, seductive tones would cause young girls to behave immorally'.
He quotes an editor of the London Times in May 1933 (and none of this is meant as a compliment):
[The saxophone] is at once penetrating and languorous, aggressive and cajoling. It mopes and yearns with all the reticence of a film-star. It is blatant and slimy; it is hone powdered with cayenne pepper [...]. It is as sentimental as a green suburban girl with a 'crush' on a mantinée idol, [...]
But of course, all of this also had to do with race.
The ethnic tensions that so characterised North American culture around the turn fo the twentieth century and beyond, particularly though not exclusively in the southern states, inevitably had an impact on the work of saxophonists and the manner in which the instrument itself was disseminated. [...]
[...] The exponents of these latter styles took to the instrument in increasing numbers, however, regardless of their ethnic background, and by the mid 1920s the saxophone was an essential component of nearly all bands working in the jazz tradition. And notwithstanding the extensive empirical evidence that both white and black jazz player used the instrument, over time, both inside but especially outside of the United States, the saxophone in general, and the tenor saxophone in particular, became closely identified with African-American jazz musicians.
The association of the saxophone with dark-skinned performers -- or whites masquerading as them -- considerably predates the jazz tradition.
He discusses examples of how the saxophone was presented, right from the beginning, as associated with "Turkish" and "exotic" dark-skinned people, as a deliberate attempt to cultivate its mystique in order to promote it.
He then explains blackface and "minstrelsy", and the role of the saxophone in it; unfortunately this is where the preview cuts out. When we can rejoin the story, it's in Germany, pre WWII.
Weiner goes on to illustrate how these tropes of 'Otherness' are embedded within a range of contemporary German literature, notably in Hermann Hesse's famous novel Steppenwolf (1927). Here the central character, Harry Haller, explores the liberating world of jazz-infused nightclubs, among which he meets Pablo, a saxophone player. This 'beautiful exotic demigod of love' of Creole descent devotes himself to hedonistic pursuits. At one point in the novel Pablo declares that 'when I take hold of my mouthpiece and play a lively shimmy [dance], it will give people pleasure. It gets into their legs and into their blood.' This conflation of signifiers -- the saxophone, the dark-skinned foreigner, the ascendance of physical pleasure over intellectual control -- exemplifies precisely those traits widely identifies as characterising black jazz musicians that contributed to the negative perception of the saxophone in Germany at this time.
Then, Nazis. I'm pleased to learn that the saxophone pissed Nazis off: on one hand, it was a glorious European invention they totally wanted to claim as their own; on the other hand, it had acquired, from their perspective, African cooties, and championing it would likely be misconstrued as their admitting there was something a "lesser" race was good for.
He then discusses the spread of the sax in the USSR, and then gender associations and the sax, and then, right when things are starting to get really explicit....
Thus the priapic Dionysian ecstasy of wind instruments was set against the Apollonian restraint represented by the strings. This distinction was continued through the Middle Ages and beyond when, for example, the bagpipes were often symbolically taken in art and literature to represent human depravation in general and sexual lust in particular. Such associations persist today, as
Pages 333 to 338 are not shown in this preview.
However, as with the caricatures of African Americans noted previously, this unconscious association of the saxophone with basic, instinctive human traits, particularly those pertaining to physicality and sensuality, were conceived as negative attributes in the minds of some in the 1920s. Among the early indignant opposition to jazz and dance music, the instrument's ability to moan was often perceived to be an unwelcome if widespread characteristic. The point was made explicitly by one commentator in 1921 who observed that 'those moaning saxophones [...] and the rest of the instruments with their broken, jerky rhythm make a purely sensual appeal [...]. Jazz is the very foundation of salacious dancing.' The conflation of quasi-human moaning and potentially destabilizing physicality could not be clearer.
As the Blues tradition became more widely recognised so the saxophone wail was invested with different meanings [....]
But then on a page that isn't part of the preview, we get back to the sex.
[...] In the film the moment when the stage actresses are first required to disrobe in rehearsals is again underscored by a prominent saxophone solo. Nor is this association between the moaning saxophone and the sexual act confined to Hollywood. In India, Bollywood films have also long borrowed this Hollywood cliché in their scores, and used a bluesy saxophone cue to accompany a female character who is deemed to be in some way unvirtuous. .
The association between sex and the saxophone goes beyond the latter's 'moaning' qualities, however and for some it is the very timbre of the instrument that connotes sexual engagement.
I am going to stop here, because (1) my arms are done and (2) this is a G-rated site.