For the 2000th question on guitars - brass nuts are well thought of to enhance sustain on guitars, but surely that will only be the case with the open strings. Is there a secret that we need to know about?
They are denser than bone, plastic, or other materials (copper is the second most dense common metal after lead). That increased mass leads to increased inertia at the top of the neck which increases the mechanical impedance at the nut which means energy is more slowly transferred from the string to the neck. Energy staying in the string longer means longer sustain.
In addition, the stiffness of brass is about three times the stiffness of Micarta (a common high-quality plastic nut material - e.g. Martin uses this on their high end guitars), and higher stiffness also leads to higher mechanical impedance. The influences of mass and stiffness on impedance are roughly equal.
I suspect the wood and thickness of the neck have much bigger effect than replacing something as small as a nut. This is because the mass and stiffness of the entire system are what is important. So increasing the stiffness or mass of one component does not necessarily increase the mass or stiffness of the whole system to a significant degree. Also, reducing transfer of energy to the neck means the vibrations of the neck are reduced so I would expect a minor tonal change.
Combined with the difficulty of cutting brass makes me think is a lot of trouble for not enough benefit. I'd go for a Fat Finger or other headstock mass device long before I'd go for a brass nut for sustain.
I don't think it has anything to do with sustain (to be fair, there are no replacement parts that miraculously improve sustain). The idea is supposed to be that by having a nut material similar to the fret material (ie. metal), you get more tonal consistency between fretted and open notes. Compare a string anchored on nickel and steel to one anchored on brass and steel. I guess you would get the same consistency by making the frets out of bone...
Oh and for sustain, use a stiff pick, hit the strings hard, use lots of compression and distortion, high volume, and stand close to the speakers.