Working on Analyzing this Motet by Dufay, most of this fits a certain type of notation (I don't know the English name) but some doesn't. What's your opinion?
The older type of notation seems to be a form of mensural notation common in the Renaissance era (when Dufay was active).
Mensural notation wasn't really codified quite like today's notation was, so there are different formats to it; in a way, it's like there are dialects of mensural notation. Here's another sample of a different piece:
Sometimes scores will mix mensural notation with modern notation just to give the performer an idea of the original notation. Usually I see the mensural portion before the piece; I've never seen it at the start of the piece like this. Perhaps there's another name for this hybrid notation that I'm not familiar with.
The double whole notes in the fourth system are still seen today, but the dotted double whole note of the top system, and the stemmed version in the second system, are much more rare. (I've never personally seem then in a context outside of Renaissance music.)
As for the barlines, this is an example of what we call Mensurstrich notation; since barlines were not really a thing in the Renaissance, these barlines are used in modern editions, only written between the staves, to help modern performers who are used to barlines.
And yes, the treble clef with the 8 on the bottom means it sounds an octave below it is written.