Music has been around for a long time, but how about music theory? What was the first instance of a publication on music theory?
Short Answer: The earliest known publication on western music theory is Aristoxenus's Rhythmic Elements of the 4th century BCE; it is fragmentary, and it is the only surviving text from before the 11th century CE (!). His Harmonic Elements of roughly the same time has been reconstructed from later writers.
Long Answer: The music theory community generally agrees that Pythagoras was the first western music theorist identified in the literature. With that said, nothing that Pythagoras ever personally wrote regarding music theory has survived; we only know of his writings from later writers. The earliest of these writers is Aristoxenus, living in the 4th century BC (Pythagoras died around 500 BC). With that said, although Aristoxenus transmits a lot of Pythagoras's concepts, the two are considered to be leaders of their own respective Greek traditions: the Pythagoreans, who emphasized the role of numbers in music and the cosmos, and thus their teachings were not very practical for the performing musician; and the Aristoxenians, who emphasized tools for the practical (that is, performing/composing) musician. (Warning: this is a very broad generalization between the two!)
Pythagoras's emphasis on numbers would continue for centuries; the famous (if apocryphal) story of him passing by the blacksmith and learning the ratios for intervals was recorded by Nicomachus in his Manual of Harmonics in the 2nd century AD. These Pythagorean ratios were largely unchallenged until about the 15th century with Ramis de Pareia, but even then the influence of Pythagoras is sensed in Zarlino (1558) all the way up to the present day in discussions of tuning systems.
(Source: The Cambridge History of Western Music Theory, ed. Thomas Christensen. See especially the chapter "Greek music theory" by Thomas J. Mathiesen.)
Not sure if you are thinking about Western music theory, but people all over the world have had 'music theory' before Pythagoras.
For example this link goes into history of Indian music theory
the first reference to musical theory is found in Rikpratisakhya (400 BC).
There are two questions here. Taking the second first:
What was the first instance of a publication on music theory?
Wikipedia's "Music Theory" page devotes four segments to theory dating to "antiquity":
The earliest surviving music theoretical writings are Mesopotamian.
four single cuneiform texts and a fifth text group [reveal] ... musical theory at least as early as the first half of the second millennium [BCE], but that also provide us with the proper names given to the notes on the scale and to certain intervals of the scale.1
Next oldest, from China,
are inscriptions on musical instruments found in the tomb of Marquis Yi of Zeng, the ruler of a small state in the middle Yangzi region, who died in 433 BC.... The inscriptions about music theory appear on [several] chime stones and the bells. They concern pitches and scales, and the pitches they refer to can still be sounded by the bells.2
This would put the Chinese inscriptions in roughly the same time-period as the Indian texts mentioned in @meowlicious's answer.
When was music theory first studied
The oldest documented music theory is Mesopotamian (see above). By comparison, Pythagoras, who is the legendary, if not actual, founder of Greek music theory, lived a millennium later.
Chinese music itself can be dated back about 8000 years, so one can at least speculate whether music theoretical thought there pre-dated the "more recent" inscriptions described above. And there is evidence of (primitive) music in India as far back as 30,000 years, which further invites wonder at how long ago music theoretical thought might have emerged.
1 Kilmer, Anne Draffkorn. "The Discovery of an Ancient Mesopotamian Theory of Music." Proceedings of the American Philosophical Society 115, no. 2 (1971): 131-49. Accessed December 29, 2020. http://www.jstor.org/stable/985853. (Quotation from pages 131-32; Emphasis mine.)
2 Bagley, Robert. "The Prehistory of Chinese Music Theory." Proceedings of the British Academy 131 (2005): 41–90. Accessed December 29, 2020. https://www.thebritishacademy.ac.uk/documents/2010/pba131p041.pdf (Quotation from pages 41-43.)