It's not that common, but it's not unacceptable either. The first beat of any bar, 3/4, 4/4, 5/4 etc., is usually the most emphasised - if that feel doesn't come across, maybe the time signature isn't correct.
So, yes, change the time signature for one bar, or two, as that's the way most 'irrational' music gets written. You could stick to say, 4/4 and write accents over the notes that would otherwise become 1st beats in new bars - that is another ploy used by some composers - but then you must be aware that the first beats will still have emphases. Probably best to change time signatures, and write emphases as and when needed as well. Also be aware that if the 'beat' continues in rhythm, the bottom 4 will reflect that.