Seems like there are a couple of parts to your 'musing' here - one aspect is "why focus on particular notes rather than scale degrees?" Another is "why use dots on a stave rather than numbers?"
I'm not a psychology expert, but think the reason to have a 'graphical' representation of notes is simply that graphs and similar visualisations tend to be faster to read than streams of numbers - the same reason you put graphs in your powerpoint to impress the boss, rather than just the raw numbers! Musical dots also have a very clever and efficient way of representing rhythms, which we'd have to reinvent if we got rid of the dots altogether.
Because isn't the actual key not really important. it's the relation to the scale that's the important part no?
In most specific cases, I don't think that's true. Often, transposing a work a few semitones down might make the notes' timbres mesh differently together and stop the harmony working as well; on a more practical level, many parts might only be playable on some instruments in a limited range of keys (or even only one) - for example, something in E or A is often much easier on a guitar than F or Bb.
Even so, people have recognised that it is valuable to have scale-agnostic ways of notating music - the roman numerals used in Roman numeral analysis are one example, and the idea is also used with Arabic numbers in the Nashville number system. You'll notice that in both cases, the focus is on giving numbers to chords, rather than notating every note with a numeral - notating every note as a number would probably be too dense a stream of information (going back to the idea of why graphs are better).
Both these systems do sometimes hit a practical problem - what do you do when the piece modulates, or when the key of the piece is unclear? That's going to be a problem with any system that explicitly works in relation to a key - in fact, it's already something of a problem with standard notation.
...when the notes are explicitly written as in sheet music then it makes it harder to transpose.
There is some truth in that, although you could argue that when the notes aren't explicitly written, that makes it harder to just play the piece straightforwardly without transposing... ultimately, every system people have come up with for notating music so far involves some compromises.