The answer by "guest" is perhaps a bit extreme, but it does have a bit of a point. Music needs to be composed such that it flows together in time, and I think the view of "thematic" sections vs. "transition" sections tends to create a false sense of prioritization, as if the "theme" sections are primary and the "transitions" just "fill in" between them or something.
But nothing could be further from the truth. The transitions in most pieces of music are structurally as important to creating a good piece of music as the themes are. In the same way that the appropriate uses of dissonance help to move music forward between consonant chords, transitions must be an integral part of the structure. One can't simply "sprinkle in" dissonant notes and expect a good result.
Another analogy: writing the thematic sections and then trying to fill in transitions is akin to writing an essay by writing all topical parts of an essay first, and then at the end trying to go back in and make the essay connect together and flow with transitional sentences, etc. While it may be possible to do that, it's often easier to only sketch out the main topics and then gradually connect them together as you write. If you fill out the whole argument ahead of time and try to add in transitional sentences connecting disparate ideas, you're likely to end up with something that reads awkward at best and perhaps just feels like a mash-up of disconnected ideas that are patched together.
That's likely why you're ending up with something that "sounds either too abrupt, or superficial (like a theme, then some harmony, then a new them), like just two themes glued together."
Transitions should usually be sketched and planned out as you're first composing. That's not to say you can't come up with thematic ideas and sketch out some melodies throughout the piece, but it's a good idea to start planning transitions early.
The next step is to realize what I stated that the outset: transitions must not be thought of as "secondary." They are essential to driving the piece forward. Some of the most dramatic moments in sonata forms tend to occur during so-called transitions. To me, these are the "active" parts of the piece, when you modulate, break down motives and develop them, create drama by building up toward something (often through things like repetition, pedal points, etc.). When I'm analyzing a sonata form, I actually care more about what's happening in the transitions (and development), because they're the most interesting parts. Once you're in the theme, it's usually boring ho-hum periods and sentence structures. Transitions are what makes the music alive.
Hence, my biggest piece of advice is to go study those sections in actual pieces of music that you'd like to imitate. If you like Bruckner, analyze what Bruckner does in transitions. Analyze how he structures developments.
The problem with textbooks is that they don't tend to focus on these passages as much, simply because they are less predictable and thus harder to theorize. But that doesn't mean there aren't patterns. For example, take the transition between the first and second themes in a classical sonata form movement. You typically want to modulate there (either I-V in major, or i-III in minor, for the most common options). But that's not the only goal of a transition. You want to set up a passage that tells you that you've moved beyond the first theme and also feels like it may introduce something new.
So how might you go about this? There are a number of standard strategies for this particular transition in sonata form. For example, you could begin with some simple repetition of something from the first theme. Maybe you "start over" with the first theme, but create a sort of "fake-out" where it rapidly introduces chromatic notes and leads you away from tonic. The first theme gets distorted and leads to something else. Or, you might begin by riffing on the final phrase or cadence of the first theme area, breaking it up into smaller motives but leading elsewhere. While not necessary, a connection to the first theme makes the transition more smooth before beginning something different.
Alternatively, one can just introduce a more "hard break" with the first theme, but then you should introduce something really new -- perhaps a new motive that has a clear and distinctive musical gesture to build the first part of your transition off of. Again, the transition shouldn't just be a "throwaway" section -- it can have its own motives and ideas. The point is to create contrast with the first theme, whether by starting by riffing on the first theme and then making it go askew, or by introducing a new theme/motive that moves away from the first theme harmonically and motivically.
The next thing to note about transitions is that they often (though not always) feel unstable. You're modulating, but that doesn't mean you just need to compose a single phrase going directly from one key to another. Transitions often have much more chromaticism than thematic sections. You might skip around among a few keys before ultimately landing at your destination. You might spend an entire four bars or eight bars or longer on diminished seventh harmonies, simply feeling "unstable" with the ultimate direction unknown. You might throw in sequences that begin to move you through various keys.
But then, toward the end of your transition, you want to signal the arrival of something new. Often the best way to do this is to make the music "less interesting" in terms of new ideas, new harmony, etc. Effectively, you're setting up perceptual salience for the new theme to enter. The end of transitions are often very repetitive for this reason. The most common strategy is a "dominant lock" before the new key, where you just keep hammering away at the dominant of the new key, either repeating the chord itself with figuration or by having repeated half cadences with a motive that gradually breaks down and shortens. The conclusion of a transition may sound like "high drama" in the sense that it may be loud or fast, but harmonically and motivically (and even rhythmically) it may be incredibly repetitive and completely static. That's setting up something "boring" (at least in some parameters) and getting a listener prepared for something new and interesting to arrive in the next theme, with its new melody, new key, new harmony, etc.
That's only one strategy for one particular type of transition (1st to 2nd theme in sonata form). But hopefully you get the idea. There are generally three phases to an extended transition:
- Connect to the previous theme (optional), but then break conclusively from it (through new motives, new harmonies/chromaticism, by breaking down thematic units into smaller fragmented motives, etc.).
- Create a sense of instability that allows you to move toward the parameters of the new section (e.g., accomplishing the modulation).
- Prepare for the entrance of the new theme, generally by scaling back harmonic and motivic activity. There may also be a break or shift in texture. Also set up the expectation for novelty, perhaps by also anticipating a feeling of resolution (e.g., through a prolonged dominant or repeated cadential moves toward the dominant). Some pieces will build up a very dramatic moment here, with a sudden break, and then an entrance of something new, while others will make the formal "resolution" into the new section more smooth.
Different composers accomplish these tasks in very different ways, and individual pieces will vary significantly. Hopefully this gives a sense of where to begin, but if you're really trying to imitate a specific compositional style (e.g., Bruckner), I'd start with looking at the particular strategies that composer uses.
FYI -- Bruckner himself spent quite a bit of time practicing writing transitions in standard small forms before he developed his more wide-ranging approach to modulations, harmonic surprises, etc. in his mature practice. Even when older, he devoted considerable time to studying the way Mozart, Beethoven, etc. organized their forms. Underlying Bruckner's chromaticism is often a high level of key organization and motivic development that guides the various moves through his transitions.
