Is there a way to notate the repeat of a larger section that itself has repeats in it, or is it best to give up and just write it out again?
You can introduce symbols (most often the "dal segno" sign pictured below; rehearsal marks are also not uncommon) to indicate the start of a repeated section, and then use a "to [sign]" indication to denote the return point of a repeat sign. This is most clearly used in combination with volta braces, more commonly called first and second endings (and third, fourth, etc. if necessary).
The below example shows one of many ways complex repeats can be accomplished. The "roadmap" in this case is:
- Play mm. 1-7; Repeat to m. 5
- Play mm. 5-6, 8-11; Repeat to m. 2 (D.S. = "Dal segno" = "go to the sign" [lit. "from the sign"])
- Play mm. 2-6, 8-11 (because m. 7 indicates first time only, and this is the third time through this section); Repeat to m. 2 (because m. 12 indicates to play both the first and second times through); Repeat to m. 2
- Play mm. 2-6, 8-10, 12-end.
Obviously, this can get confusing quickly, so it's up to you as the transcriber to decide whether nested repeats or a full rewrite is more clear.
A song often has internal repeated sections and an overall 'next verse' repeat. This can be shown by a D.C. (or D.$) al fine (or al coda). Sometimes this is simplified to just a segno & symbol at the beginning and end of the 'big' repeat. Not quite textbook notation, but understandable.
The orchestral parts of e.g. a Strauss Waltz (think 'Blue Danube) can be an absolute maze of nested repeats and segno jumps. Far from 'correct' notation and hard to sight-read until you catch on, but you save a LOT of paper (and page-turns).
I'd be reasonably confident in writing something like this - particularly if it was hand-written and I could emphasize the outer repeats even more heavily, maybe even with a red pen!
Different rules apply for music that is to be sight-read and music that is for reference. For a recording session, write it out! For the transcription of a pop song, save paper. Likewise for a march that needs to fit on a 'band card' clipped to an instrument.
If this question is about how to do it in Sibelius, the answer is that nested repeats won't playback properly by default, but you can write them and apply a manual playback map. Just be careful if a live player is ever going to see the music. If Sibelius playback can't understand the repeat structure, it's a fair bet the player won't either.
If the "inner" repeat is relatively short, consider using the "Repeat the previous N measures" symbols. Here's the info, from (of course) my favorite site, Dolmetsch.