I've set up a configuration in Sibelius 8.5.1 with Sibelius player, Aria, Play, UVI and Kontakt. They all work except Kontakt5, which doesn't seem to be receiving any midi input information yet everything else works fine.The midi channel option shows slot 1 or slot 2 - but won't allow me to change midi channels.
Not sure if you've managed to work this out yet. Anyway, a few things need to be done first before Kontakt will work properly.
Make sure Kontakt 5 has the "Sound Set" set to "(none)" (As shown in your image already) then go to the "Manual Sound Sets" (the next tab) and check the box labelled "Use manual sound set" (see image below) and the No. channels to 16.
If the checkbox for "use manual sound sets" isn't checked and the number of channels is not set then you won't be able to change the slots in the mixer.
Now you can change the slots in the mixer from 1 to 16, but make sure Kontakt 5 is activate for that track. Let's say I change the Flute I to "slot 3" (this basically means Channel 3 of the active VST) using the small green up arrows:
Open Kontakt 5 using the cog icon on the track, load in an instrument and change the Kontakt tracks channel to Port A => 3 (slot 3 as set in step #2) or whatever slot you used in the Sibelius mixer. The port should always be Port A.
If these are not set, the slots don't match the channel or no instrument is loaded, then they won't be any sound or midi data passing through.