I haven't found any comments on this question online from someone who actually knows a lot about amplifier design. I also have not been able to find any horror stories such as "I did this once and destroyed my amp/speaker", which I would expect to find if it were actually an issue. I have found many mentions of professional bass players using guitar amps and cabinets, such as this list from TalkBass:
The dude from High on Fire
The dude from Death From Above 1979
There are mainly two ways to damage a driver: over excursion and over heating. Overheating is caused by consistently sending too much power to the speaker over a long enough period of time. Over excursion is caused by sending too much instantaneous power to the speaker in a very short time.
Normally, an amplifier is designed to safely drive certain speaker cabinet designs. That means the continuous and peak power outputs of an amplifier should be designed to be below the continuous and peak power capacities of the speaker cabinet(s) connected to it. This is generally true with guitar amps also.
The question really is, can an unexpected type of signal (i.e., one with lowest frequency an octave lower than designed for) lead to a higher continuous or peak power output of an amplifier? I would think a well-designed amplifier would be designed to operate within specification regardless of the input it is given, and also should have a safety margin in case of unexpected behavior. Most guitar amps and speaker cabinet pairings seem to fit this criteria.
In the case of an amplifier and speaker pairing where the amp can damage the speaker, using a guitar alone with no effects would not be any safer than using a bass or low frequency effects. Hitting a guitar or unplugging the cable can produce very low frequency impulses and unplugging a cable going into a cranked amp is a problem no matter what the instrument, amp, or speaker cabinet are.
Based on that, I think this is an urban legend. If it were true, guitar amplifier manuals would have a warning not to plug in keyboards, basses or other low-frequency sources. Pitch-shift effect makers would have warnings against using extended low frequencies through guitar amps. And we would see anecdotes all over about people damaging their guitar amps or speakers because they used such-and-such through it and NEVER DO THAT and etc. Instead all we see are people writing "I think it could damage the speaker".
Edit - caveat:
Plugging a bass amplifier into a guitar speaker cabinet could be very bad for the guitar cabinet, depending on the amp. But that would be true even if you played a guitar though that rig. It's not the low end of the instrument that's a problem, it's the mis-matching of amp and cabinet. There is a Celestion video which confirms that hooking a bass amp to a guitar speaker cabinet is not a good idea.
From the Eminence web site (emphasis mine):
A transducer might be able to handle 1,000W from a thermal perspective, but would fail long before that level was reached from a mechanical issue such as the coil hitting the back plate, the coil coming out of the gap, the cone buckling from too much outward movement, or the spider bottoming on the top plate. The most common cause of such a failure would be asking the speaker to produce more low frequencies than it could mechanically produce at the rated power. Be sure to consider the suggested usable frequency range and the Xlim parameter in conjunction with the power rating to avoid such failures.
While that would seem to indicate putting lower than usual frequencies into the input of a guitar amp could be dangerous for the driver(s) being driven by that amp, most guitar amp designs have limitations on the output frequencies and power that in practice make it hard to damage a guitar speaker that is properly matched with a guitar amplifier. Again, plugging a guitar speaker cabinet into a bass amplifier would be unsafe for the exact reason quoted from the Eminence site. Note that all but the smallest of bass amps will put out more than 50 Watts, with most putting out more than 100 Watts up to thousands of Watts. A 1x12 guitar speaker enclosure would not normally be rated for more than a 50 Watt amp, so just on power output alone, it's better to not use a bass amp with guitar speakers.