Is it possible to have overlapping system delimiters (braces and brackets) in lilypond? I know how to do complex nested hierarchies, but they're always nested. I can't figure out how to start, for instance, a brace before ending a bracket. Getting something like:
I've been digging in the manuals a lot, and it seems there is no override for this. The only way to get a brace is to put the staves in a staff group. Perhaps you can have your program give a warning that this is not possible and give it a default way of notating it.
Someone who knows the internals very thoroughly may be able to tell you how to manually draw the staves, but it will be very ugly coding.