Below is a shot of the latency numbers from Focusrite's page on the Clarett line specifically:
This chart shows us a few interesting things. First, latency is also a function of sample rate, and more importantly for you, the host software and operating system both affect latency.
So you could try a larger buffer setting with a higher sample rate to see if it give better performance with the same latency, or you could try using software that has lower latency with larger buffer settings (Reaper is very affordable and performs well here).
But most likely what you'll have to do is either upgrade your computer or minimize the amount of work it has to do or both.
Hard drive latency and processing power are the two more likely sources of dropouts and latency issues, but having too little RAM can cause excessive hard drive paging and lead to hard drive problems. Generally you want the most computer you can afford.
But in your particular situation, aside from changing software and increasing the buffer size, you want to reduce the load on your processor and hard drive. It probably goes without saying that you should disable any other background software on the computer. It may also help to disconnect from all networks and disable Bluetooth, etc., essentially go into airplane mode.
After that, there are things you can do within your DAW to free up resources. Plugins are processor intensive, and audio tracks and sample based virtual instruments are disk intensive. The easiest way to free up a lot of computing power is to disable (which is not always the same a mute) as many tracks as you can stand. For any instrument tracks you absolutely need, it usually helps to render them, turn them into audio tracks with all the plugin processing done and rendered, which takes the load off the CPU. Same with audio tracks although you usually get less of a benefit.
As you wrote in a comment, you could bounce the whole thing to two tracks but that's a bit extreme IMHO. Lawrence's suggestion about "freeze" is a good one if your software has it.
Worst case, disable all plugins that are not absolutely necessary. Definitely disable computationally expensive plugins like convolution reverbs and look-ahead peak limiters. Disable anything on the stereo master bus. Disable entire aux return tracks.
Absolute last resort is to double your buffer size and attempt to deal with the latency in your playing. I almost always record audio with low latency monitoring where you don't listen to the track you are recording through the software.