The written notes actually miss a lot of information. If you would write down every little tempo change, every little accent, small and large scale rubato, microdynamics, articulations, etc. etc., the score would be impossible to read, and it would probably still miss something. What the composer writes down is just a skeleton of the piece and the most important details. The rest is filled in by the performer(s).
Now every performer has a different background on which they base their interpretation. Thus the piece will sound different when played by different people. This is a problem when many people play together since the piece would not sound logical. So, one job for the conductor is to unify the interpretations (or rather, probably force his own). Of course the conductors also have different backgrounds so the results will again be different, even if the orchestra doesn't change.