Saying that a piece of music is "in a key" often (even usually?) isn't a statement of fact, but a statement of perspective.
When you say that a piece is in C major, you're saying that
- The tonal centre of the piece is C - that you mostly feel that C is the home note
- The set of notes used by the piece is a reasonable fit with the C major scale - that the notes C, D, E, F, G, A, B are the framework that the piece is built around.
The thing about both those statements is that they leave some room for opinion - they are not absolute, factual statements.
I compose a chord progression F-G-C-Am … can I assume C as the Key?
Not necessarily. It depends on the phrasing of those chords. If Am is the most important chord in the phrasing, then it might sounds more to you like A minor. You have to think whether you feel that C, or A, or something else is the 'home note'.
how do I know I'm not in Am
You can't "know" - but you can feel that the piece doesn't come home to A.
I've seen songs analysis that says a song is in determined key even when the root is not played once in the whole song, how does this works?
Because key is a question of perspective, not a question of fact, you can analyse any song from the point of view of any key. It might be useful and interesting to analyse a given piece from the point of view of a key for which the song didn't land on the tonic chord. In such cases, it's unlikely that that would be the only sensible key to think about.
is there any rule?
No, no absolute rule. People often come up with "rules of thumb" that work for them, but none of them are definitive.
The one way in which being "in a key" could be definitive is if the composer labels it as such - e.g. if I write "Scherzo and Trio in A minor", then in that sense it is defined to be "the key" of the piece. But I don't think that's the sense in which you ask the question.