According to the information in this online Audacity manual, you can simply press Ctrl-Z straight after recording to Undo Recording.
According to the manual, Ctrl-Z:
undoes the last editing operation you performed to your project. You can undo as many times as you want, all the way back to when you opened the window. To undo many operations, select History... from the View Menu. The name of this menu item will change to reflect what will be undone; if you just recorded some audio, the menu item name will be Undo Record.
I just checked this method; it seems to only delete the material just recorded.
I guess you might be able to change the function of the backspace key, to make it perform an Undo instead, but then it won't function as you expect it to in other situations (for instance, when deleting a section of selected audio). So, it's probably better just to get into the habit of using Undo: Ctrl-Z.