Things certainly get tricky once we step outside the common time signatures and the solution to choose becomes more and more subjective the further removed you become. Generally speaking, groupings are used to convey phrasing and allow for easier reading. Those are the two things to really consider when trying to notate your music, which it seems you understand at this point. Tradition note groupings usually follow a standard for readability a little more than phrasing. For example, in 4/4 you are usually expected to group the first two beats and last two beats, with some exceptions (like dotted quarter-dotted quarter-quarter), but when you have heavy syncopation, the expectation is that you notate based on readability, not phrasing, with the understanding that the rhythms encountered are against the beat.
Conveying Phrasing
We typically find that odd meters are divided by beats in sets of 2 or 3 subdivisions (the denominator of your time signature). Sometimes you will find groupings of 4 or 6 but this can be interpreted as combining 2 sets of 2 or 3. So we don't typically find a grouping of 1 beat, not that it's necessarily wrong, just that it doesn't follow the convention. The problem ends up being that there are a couple ways to interpret the term phrasing: 1) The way the part that is being notated is phrased; 2) The way the time signature is structured. As I mentioned above, sometimes the approach is to spell things out to delineate your beats, so writing syncopation against the beats doesn't necessarily feel like you are notating the phrasing as outlined in number 1 but it does convey number 2 and could be argued that the syncopation against the beat actually better conveys the phrasing than to base your groupings on the part/line you're playing.
So making the decision based on phrasing can become difficult as to whether you choose to group your notes for a given part or within a more approachable meter for the purpose of syncopation.
Grouping for Easy Reading
I find this to be more challenging than grouping for phrasing. Typically we find that odd meters are just harder to read in general, so it's difficult to choose what the easiest way to write something that will still be difficult to read would be. I would ultimately suggest that finding as much consistency as possible is going to be your best bet for readability. If it's going to be difficult either way, you want to make sure that once it's interpreted, it doesn't need to be reinterpreted every new phrase/measure/etc. I find that it's easiest to read smaller groupings when reading odd meters, so I might prefer to read two sets of 3 1/16 notes instead of one set of 6, or potentially three sets of 2 1/16 notes if that more properly spells out your phrasing.
One suggestion I received from my composition teacher back in college was that it may be easier for players to read if you just leave everything in 4/4. I was writing some stuff that had no consistency to the meter, which I just left in 4/4, and he was suggesting that instead of reworking the piece to have the meter change to match the phrasing, which would be changes pretty constantly and not back and forth between two meters like your piece appears to be doing, it would potentially be easier to read in 4/4. The thought was that the reader could just focus on playing what's there, instead of having to think about the next meter they were approaching every measure. This approach could be helpful to some extent but it doesn't really tell you anything about what's happening in the music that you're reading, so you would surely want to supplement that with phrase markings.
In Conclusion
When you're making your decision, you'll want to determine who will actually be reading this and in what context. If you will be giving it to a performer to have them play it, then you would probably want to make sure that your phrasing is well conveyed, leaving as little interpretation to the player as possible. If you're writing it out for someone to sightread, then you will want to make sure that it's as readable as possible, preventing them from getting confused by odd groupings. If you're just writing it out for yourself, you should probably just write it out however it makes sense to you. On the whole, you just want to make sure that whoever has to read it won't have to guess what your intent was when writing it.