What are you are referring to is called harmonic stress. It is the metrical stress accorded to a pulse based on its position within a grouping.
It should be noted that, while sometimes musical performers add accentuation to notes or chords occurring on strong harmonic stresses, this is not always the rule. The strength of the beat is actually not a function of amplitude or musical performance, but of the perceived importance of that pulse within a larger grouping.
As you note, the downbeat (the first beat in a measure) is almost always the strongest harmonic stress. Actually, it is not as simple as having just stressed or not-stressed beats. There are degrees.
In common time, the stress is STRONG WEAK strong weak. (1 and 3 are strongest, 2 is weak and 4 is weakest.)
In waltz time, the stress is STRONG WEAK weak.
In cut time the stress is STRONG WEAK.
In odd time meters, the stress is usually just a direct derivative of one of the above. However, I think your idea about the 5:4 stress is not quite complete. It may be possible to have a measure of 5 that is in the pattern of 4+1 or 1+4 as you mention, but I believe 5:4 is usually either 3+2 or 2+3. This affects both how it is notated and how the harmonic stress is heard. So it's either STRONG WEAK weak STRONG WEAK (as would occur from combining a measure of three with a measure of two) or STRONG WEAK STRONG WEAK weak (as would occur from combining a measure of three and a measure of two.)
7/4 time would be seen as a combination of 4+3 or 3+4, and would inherit the stresses that follow.
11/4 would have to be either 4+4+3, 4+3+4 or 3+4+4, but this is a situation we rarely find ourselves in, isn't it?
As for compound triplet meters such as 6/8, 9/8 and 12/8, they are really just measures of two primary beats, three primary beats, or four primary beats, and would have pretty much the same harmonic stress as a measure of 2, 3 or 4 would in the above.
As for notes not occurring within the pulse, such as the "and of 1", they would have the weakest of any harmonic stress, and would also tend to be perceived as inheriting any of their harmonic stress from the pulse which they are subdivided from. So the "and of 1", if it had any real harmonic stress, would be thought of as having the harmonic stress of 1. Indeed, if the measure began with an eighth note rest, the "and of 1" may possibly be heard as an event on beat 1 that was late to enter. However, worrying about the potential metrical stress of subdivided notes is probably not necessary to have a strong grasp of this idea, so we should probably leave it at the main pulses.