How involved was the congregation in the performance of music in a Protestant service of Martin Luther and a Catholic service in the Counter-Reformation?
Martin Luther, long before he broke with the Catholic Church, believed that music was a form of prayer, through which the people spoke to God and God spoke back. History states that he didn't actively encourage his congregation to sing along until after his publishing of the 95 Theses and his excommunication, but some sources say he encouraged it at Wittenburg while he and the Wittenburg church were still Catholic. According to most sources it caught on fairly quickly, though I'd imagine the first few weeks of Mass were a bit timid on the part of the congregation.
In the Catholic Church at the time, a decree dating all the way back to the 4th Century and the Nicene councils that only the clergy should sing was still in effect. Therefore, with Protestant congregations singing hymns and the Catholic Church continuing to condemn practically everything Luther said or did (Catholic figures at the time lamented, "the German people are singing themselves into Luther’s doctrines, and his hymns destroy more souls than all of his writings and sermons"), the Counter-Reformation would only have strengthened enforcement of a clergy-only liturgy. It would not be until 1964 that the edicts from Vatican II would take effect and the "New Mass", celebrated in the vernacular of each community, would formally lift this ban on Catholic congregational singing.
However, in certain locations (especially in newly Protestant countries like England and Germany), Catholics found themselves having to do their own singing. They didn't have much choice, as performing the Mass at all was an act of treason, so anyone still considering themselves in the Holy Orders wouldn't be quick to answer calls for Catholic liturgists. William Byrd, a reclusent Catholic in the employ of the Tudor royal family, wrote masses in three, four and five parts specifically for use in private chapels, where there may only have been half a dozen people in attendance, and it would be naive to assume they'd all be in Holy Orders of any kind. Byrd himself wrote in a preface to his "Psalmes, Sonnets & Songs", "Because it is so good a thing, I wish all men would learn to sing... the better the voice, the more fitting it is to honor and serve God therewith; and the voice of man is chiefly to be employed to that end". So, Holy Orders or not, there were Catholics who agreed with the Protestants on this one point even during the Counter-Reformation.
For a little bit more history as well as a primer in the sacred music of the time, I'd point you to the excellent BBC docuseries, "Sacred Music". Episodes are available on YouTube, and I have a playlist here which will take you through them in roughly chronological order. Of particular interest to you will be episodes 3 and 4, which deal most directly with the events of the Reformation in England and in Germany. Episode 2, and the special episode covering T.L. de Victoria, also have some information about how the Reformation and Counter-Reformation impacted still-Catholic nations like Italy and Spain.
I believe in those centuries (16th and 17th), going to church was a very important day each week in most European communities. With that in mind, I believe in both cases, the congregation was as involved as they could be. However, Lutheran hymns are written in the local language, which had been done in order to more greatly involve the congregation. Catholic hymns, on the other hand, were in Latin (until much later), which made it much harder for the layman to participate in singing the hymns.