Trying to notate recorded liturgy a cappella music. I have a problem with getting the notes right. Since the music changes keys from 3/4 to 4/4 etc. in a movement, the ebb and flow changes constantly. The normal way of feeling the rhythm doesn't help me in this case. I tried to do it mathematically, but I don't know if it's correct. Can anybody direct me where I can find material that I can glean from them on this subject. To make clear I'm not looking for time signature, I'm looking to learn the correct way to notate this kind of music.
If you can provide us specific audio examples of music you need to transcribe, that would be helpful.
If you are speaking of the Christian church tradition of chant (Gregorian chant or otherwise) this music is traditionally notated without meter, without bar lines, and is in fact notated without any specific note durations. The notes are to be sung with the rhythm of normal speech, as in reciting poetry. Mostly just pitches are notated; the rest is left up to the singer or choir. The singer or choir are actually expected to work out, agree upon, rehearse and perform the durations of each note/syllable without precise notation.
Get a Catholic or Anglican hymnal and observe how the chants are notated in there, and copy that method.
Wikipedia article on Anglican chant (a starting place for finding other references)
In this example, from an Anglican hymnal, the bar lines don't actually indicate a rhythm with beats you could count with a metronome. The values of the notes (half note, quarter note, whole note) are not absolute like they are in traditional sheet music; they merely approximate the relative length of each syllable. The singers understand how to chant the multiple words on each pitch with the rhythm of normal speech. The text at the bottom includes marks that indicate when to move from one pitch to the next in the sequence of the melody. This is how it's notated.
Here is another method, where the words are aligned below the pitches:
There are specific conventions of notation which vary a bit in usage. This may seem arbitrary, but in most chant books or hymnals the specific notation and singing rules to be used are stated in the preface.
For the example I have shown above, from http://www.ccel.org/cceh/chant/chantehi.htm, the instructions for note durations say this:
One, two, or three syllables may be sung to its final note.
The final note of the Ending provides for one syllable only, except in the words thanksgiving, handmaiden, forefathers, and night season.
The signs used for pointing in this book are as follows:
(a) The upright stroke | precedes the first syllable of either Mediation or Ending, and is occasionally repeated after the first rhythmical foot of the Ending.
(b) The syllable in bold-faced type is always sung to two notes of the chant.
(c) The dot · is placed after, and sometimes before, two syllables to indicate that the second is to be sung to the same note, repeated with the same time value, as the first.
(d) The dash -- indicates the omission of the Reciting Note in the three verses where it occurs on p. 741.
A comma in a line is observed by slightly prolonging the previous syllable.
Be careful not to stress weak opening syllables in a line; and to sing a weak final syllable lightly and quickly.
Performing these pieces successfully depends on the solo singer or the choir director understanding the rules and being able to explain them to the choir and to rehearse them until they can sing the piece together coherently. It is not considered desirable to notate things more specifically than that.
Yes, you are right,
the normal way of feeling the rhythm doesn't help in this case.
With liturgical music it may help you if you forget the modern definition of "rhythm" as (http://beta.merriam-webster.com/dictionary/rhythm)
a regular, repeated pattern of sounds or movements
and start thinking about rhythm as in Plato's much more generic definition (http://www.britannica.com/art/rhythm-music):
an order of movement
@WheatWilliams gave an excellent explanation and description.
Just as an example of a different possible notation is the quadratic notation of Gregorian Chant below. Notice the small vertical marks (ictus) which divide melodies into two- and three-note phrases, each beginning with an ictus. (The ictus is not marked at the beginning of graphically connected groups of notes, where it is always assumed by default.) The ictus does not correspond with the stressed beat in the modern rhythm. However, there is a discussion about it and some perform Gregorian chant with no beat or regular metric accent (https://en.wikipedia.org/wiki/Gregorian_chant#Rhythm).