As the comments have suggested, there are a number of ways to define "music", some more subjective than others. But a good starting point would be that music is "organised sound", with the one crucial caveat that it is sound organised temporally (i.e. "in time"). Therefore you could remove arbitrarily many parameters from a performance score and the result could be considered to be some kind of music, as long as this still results in the production of sounds, spaced temporally.
To take your Beethoven 5 example, I would argue that only the pitches and rhythms are crucial to easily recognising this piece of music when performed (assuming one has chosen a sensible set of tempi!) This would result in essentially the same piece of music being performed, but as an unbelievably flat, inexpressive performance. The extra information (accents, dynamics, phrasing etc.) are what create a "musical" performance of a piece of music. If the notes and rhythms are the "cake", these extra elements are the "icing-on-the-cake".
Then, of course, there is the additional interpretation provided by performers; conductor and orchestra, in this case (the "cherry-on-the-icing-on-the-cake"?)
If, on the other hand, you take this score and remove or change pitch or rhythm information, you would in all probability start to produce a recognisably different piece of music.
I would argue that the central question here, is not whether removal of performance indications results in a piece still being music or not, but instead what the relative artistic merits would be of these changes. In other words, how important artistically are the different elements (layers?) within a musical score?
I would also argue that some music relies more heavily upon "the dots" (pitches and rhythms) for its artistic integrity, than other music, which may rely more upon other parameters (articulation, dynamic, phrasing, orchestration etc.) To give an example: I'm pretty happy to hear a Bach Two-part Invention played on any pair of instruments (within reason!), or even while playing Jet Set Willy, but I can't think of any Ravel or Debussy orchestral music that would "work" changed in the same way. (This is surely a matter of personal preference to a certain extent though, as are all judgements of artistic merit.)