I have Boss BR 600 recorder and this device have 2 RCA line out.
Is a way to connect this line out to USB for sound quality?
You need a USB soundcard with line inputs (balanced or unbalanced) to connect this device in analog to USB, and suitable cables/adapters. Such soundcards can be pretty tiny and cheap these days. There is no real ceiling on the size and price you can pay if you want to get better quality.
However, for non-live purposes you'd be better served by reading out the data via the BR-600's own USB interface: that way no AD/DA conversion intervenes. For analog purposes, just plug the analog line outs directly into an amp. It doesn't get better than that.
So it's not quite clear what you hope to achieve by tapping the device's analog output and getting that information live in digital.