A bit of background:
According to a recent study conducted by University of California professor Diana Deutsch, due to genetics, people with perfect pitch had a much larger memory span of sounds, so that they could simply remember different pitches and reproduce them accurately.
Auditory digit span has previously been identified as a genetic component, drawing the conclusion that memory abilities passed on through genes could explain why only some of the children exposed to musical training actually develop a gift for identifying tones.
“Our finding therefore shows that perfect pitch is associated with an unusually large memory span for speech sounds,” said Deutsch in a statement released by ASA, “which in turn could facilitate the development of associations between pitches and their spoken languages early in life.”
Source: http://www.huffingtonpost.com/2012/10/23/perfect-pitch-genes_n_2003151.html
It's interesting that
Absolute pitch per se might not be genetically encoded; rather, the predisposition towards a large auditory memory is.
Source: http://www.bangscience.org/2014/11/some-science-behind-singing/
So, it is simply the genetic trait of better auditory memory that leads way to developing perfect pitch. Now, non-musicians may not have the genes for a truckload of this memory, but there is enough to remember a few pitches for a short span of time.