Non-Standard Key Signatures allow you to create keys that are not part of the normal group of key signatures. This article will show an example of a linear key signature that mixes flats and sharps, in this case Bb and C#:
Choose the Key Signature Tool, and double-click the first measure to set the new key.
Choose Nonstandard Key.
Select Linear key, and click Next until it's Linear Key Format 2 (0 is predefined as major, 1 as minor).
Use the scrollbar above until the key signature there has the same number of symbols that you want for your piece (2 in this example). For the sake of example, we're clicking up to add two sharps.
Click on Accidental Order and Amount (AOrdAmt). Unit 1 is your first symbol, and in this example, change it to step 6 (step 0 is C, step 1 is D, etc.), and the amount to -1 (-1 is flat, 0 is natural, 1 is sharp).
Click Next to change Unit 2 (the second symbol). This is unnecessary for this example, as it's already C#.
Your Bb will be in the wrong octave; to adjust the octave placement for a Unit, click on Accidental Octave Placement (ClefOrd) and change Unit 1's octave from 1 to 0 (-1 would put it beneath the staff, etc.). Unit 2 is already correct.
Click OK, and OK again. Try entering a scale passage from D4 to D5, and click play. It should display the proper notation and play back correctly as well.