Here are my settings, there isn't really a "right" answer but is important to be consistent if you want to compare one measurement to another.
First looking at Preferences you want to make sure the sample rate matches the rate you are using in Audio MIDI settings for your output and input devices. If a device has a different sample rate in Audio MIDI settings this will result in Core Audio resampling. If you want to use different sample rates for outputs and inputs (say you want to run the ADC at a higher sample rate) you can do this but you need to run two instances of REW, one running Generator and one running RTA.
Here are my Generator settings. I use no dither and select Lock Frequency to RTA FFT.
View attachment 261832
Here are my RTA settings.
View attachment 261833
I turn Peak off in Trace options.
View attachment 261834
In Appearance I select 64 bit FFT, the other settings don't really matter for this sort of testing.
View attachment 261835
In distortion settings I typically use a 20 Hz to 22000 Hz bandwidth but sometimes will use higher bandwidth depending on the test. I like to use dB over % as it is easier to tell differences.
View attachment 261836
Selecting dBc in the upper right will set the main tone peak at 0 dB and give noise in dB relative to the main tone, not relative to 0 dBFS on the ADC.
View attachment 261838
When setting scales on the RTA I usually like 0 dBc as the max and -160 dBc as min, these should cover you pretty much all DACs other than ones that are really low noise.
End result should look something like this.
View attachment 261841
If you are testing DSP stuff you will need to attenuate the output either in REW or with the DSP volume control such that you are not clipping the output. It is also helpful to run a frequency response sweep so that you can make sure the response looks like what you expect.
Michael