If fingering is written down, what stops you from disregarding it if you want to sight read ?
I think your teacher's approach is good and common, and even invoked by Jean Fassina, who spread the Polish school teachings (Chopin, Liszt, Paderewsky). There are few videos about the benefits of this careful fingering:
Michel Beroff about Jean Fassina
Jacques Rouvier about Jean Fassina
Chopin believed that each finger has its own personality, and indeed there are medical and anatomical arguments to support that.
Therefore the fingering can be very important to be able to reach a better sound, and higher ease.
There is another argument for writing down the fingering: stability of movement ie., you use the same fingering each time. This increases your learning speed, and this helps you memorize a good movement, and this helps you to go back to what you were doing if you come back to the piece a long time later ie., enough to have you forget your fingering.
In Godowsky études, you'll see that the author wrote down at least one fingering for almost each note. This is part of Godowsky's way to play the piano, it's also very helpful to find faster the right movement, because indeed it is not obvious at all to do the "right" or the most optimized movement.
In Liszt's études transcendantes (eg., Wiener universal edition), you'll be able to see Liszt's fingering, but he did not write every finger on every note, rather when there is a need, maybe a requirement (eg., 42/42/24/24/42/42 in Mazeppa's theme) or a notable trick or finger technique.
In my professional practice, I write down the fingering for those reasons, and eventually at some point, I may change the fingering because the new one is really allowing something that was not comfortable before. I do/did that several times with Stravinsky's Petrouchka, Liszt's études transcendantes, or even Godowsky's études.
Hence I advise you to accept his fingering, and to focus on notes, and if you need to practice your sight reading, to find other scores or to focus on the notes and not the written fingering.
For memorizing, you should not ever rely on your fingers only, but notes and the result of your analysis (eg., tonality, degree, modulations, patterns, etc). In the end you'll have to memorize all of those aspects, despite the hierarchy of importance.
My pieces of advice are the same wether you are beginner or professional.