I guess it depends on your way of looking at it.
What's your definition of down?
What's your definition of small amount...?
I can just give you some examples from Ragtime, because there I transcribed some songs lately. Almost 90% of all rags have a modulation one step down / counterclockwise on the circle of fifths, but often go up in pitch with the note itself:
In this one it goes from Bb to Eb. Which would be 7 semitones down, but also 5 semitones up. This is the regular type of modulation you get in rags and sounds like this (at ~0:20). Bb to Eb
Now let's do your example from A to G which would be 2 steps on the circle of fifths. So this one is down from Bb to Ab
And just to give you a comparison, this is the same moldulation but going from Bb up to Ab. So let's put Ab up an octave:
Bb to Ab +1 octave
Does one sound better to you than the other? I think it's just personal preferences... You can basically modulate up or down to any key you want. Some might sound more pleasant than others. But if that's the sound you're going for, just do it ;)