• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

SINAD Measurements

RichB

Major Contributor
Forum Donor
Joined
May 24, 2019
Messages
1,946
Likes
2,611
Location
Massachusetts
I am happy to put forward $1000 reward for him to show that a 60 dB SINAD amp sounds different than a 100 dB one in controlled, double blind test that is recorded for us to see. We have others that can help raise this bounty if needed. I am that confident he can't tell the difference let alone have a reliable preference.

As others have pointed out, If I get to pick the amp and 105 dB efficient speakers, you will lose your money :p

This statement seems to be opposed to the use of SINAD for qualitative analysis.

- Rich
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
Nelson Pass 350

Of course when one monitors using a Pass amp (which could have relatively high 2nd harm. distortion it could be really difficult to test for low distortion levels (say below 0.05%) as the power amp might be higher.
Distortion is only specified at: 1% at full power. Which basically says nothing but of course Pass does not care about this. ;)

I live on the other side of the globe so can't drop by that easily.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827

pma

Major Contributor
Joined
Feb 23, 2019
Messages
4,591
Likes
10,727
Location
Prague
@solderdude is referring to their monitoring system for AB comparisons.

Earthworks has a comparable product, by the way: https://earthworksaudio.com/products/preamps/zdt/zdt-1022/
Yeah I know he spoke about Pass. Isn't that interesting? Audio is not a simple thing at all, that could be wrapped in few numbers. I have similar experience, two almost no distortion components distinguishable through much worse parameters power amp and speakers, in A/B and even abx, sometimes. Let's not oversimplify. And I have personal experience with Millennia preamps, hats off.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827
Let me describe our current AB technique (we generally don't do ABX -- I don't think it's necessary for valid perceptual testing at the individual, non-peer-review level). If you think our test is valid, that's great. We'll go it. If you think it's invalid, then perhaps you can help us with any scientifically fatal errors you perceive. You've set yourself up as an AB authority here. I do not make that claim.

These days, I no longer use analog tape. We use a DAW called Sequoia (we license our bit-reduction software "POW-R" to Sequoia, but that's another conversation). We keep our reference program on DAW (if there's interest, I could put our reference reel on a Dropbox). We use both headphone and room speakers for monitoring. We have a world-class reference room designed by George Newburn and built from the ground up with 9" concrete walls, wall / ceiling decoupling, around 15dB SPL, flat to 24Hz, ideal RT60 (Dunlavy SC5 + Nelson Pass 350). We monitor both mono and stereo sources. We use three different AB paths, depending on the type of circuit we're testing -- DAC, ADC, or pure analog path. We do both reference-to-DUT testing and DUT-against-DUT testing, the latter being qualitatively more difficult.

Our reference material is recorded in 10s snippets, each of which repeats over and over about 10X. We've chosen reference material that we have come to know intimately. The material was chosen for an ultra-wide range of sonic gymnastics: wide spatial, narrow center, fast transients throughout the spectrum, ultra-low freqs, ultra-high freqs, high DR, low DR, voice, pop, acoustic, big band, large hall ambience, dry room, dense, sparse, and so forth. We have around 25 different program sources, but only tiny, repeating snippets of the most critical aspect of each program are employed.

Each snippet is repeated over and over so that the brain can begin to focus on just one very particular sonic aspect. AB snippets are adjusted for better than 0.02dB level matching (which is challenging). Once the listener has suitably focused, the AB selection begins. We use the DAW's "solo" and "mute" functions to facilitate a self-blinded AB selection, using mouse clicks. The mouse is clicked repeatedly until the user does not have a clue if the source is "A" or "B". My personal method is to click the mouse rapidly while counting out numbers randomly. After 3-4 seconds of that, there is zero self-correlation. I guess you could argue otherwise. Until you try it, I suggest suspending judgement. It works. It's random.

But first we do it sighted. The user clicks back and forth, between reference and DUT (or DUT-recorded program, or DUT-DUT), to simply detect any timbre shift, transient variation, etc.. If a difference is detected sighted, then we go blinded. If detectable differences remain, we try to consistently identify a specific tell and stop there. We look at the screen and note the track. Wash, rinse, repeat.

If our "tell" is identified 8/10, we consider it a positive, and do it again. Another 8/10 is qualified. It means that the DUT is not giving us a correct image. Anything less than 2X 8/10 is a null.

Then we move to the next program material and do it all over again. We do this throughout the entire program cycle. What's interesting is that different types of DUTs may perform perceptually identical for most of the program sources, but when you get to that one "particular gymnastic routine" -- a tell is detected. This is why it's important to use the broadest range of program material possible, covering every aspect of sonic hurdle a circuit will be asked to deliver.

Wurcer mentioned doing AP tests of any DUTs in advance, which is of course de rigueur (without it, the test is effectively invalid and a waste of time). And I'm happy to have the "neutral party" clicking the mouse, in fact I would love that. It's a bother. Beyond that, I think our test methodology is scientifically sound.

Thoughts?
An improvement would be to design a system or application which works like foobar's ABX comparator, which calculates the result in the end:
It's important that the switch from one to the other doesn't do anything some portion of the time, and that the number of trials isn't limited to 10. The results also shouldn't be made available until after a few listeners have participated and an impartial person calculates the results (if not done automatically).

The way you describe it, you use a DAW that you know very well, a computer that's routinely used for other duties, and I'm not clear on what you mean by "we" during the listening process—are there multiple people listening in the control room at the same time? Small non-verbal hints can easily give cues.

Could you please share photos and a layout of the control room? And the FR/waterfall results of sweeps? It would be really interesting (even outside of the listening test question).
 
Last edited:

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827
Yeah I know he spoke about Pass. Isn't that interesting? Audio is not a simple thing at all, that could be wrapped in few numbers. I have similar experience, two almost no distortion components distinguishable through much worse parameters power amp and speakers, in A/B and even abx, sometimes. Let's not oversimplify. And I have personal experience with Millennia preamps, hats off.
Well, I really don't think we should be quoting personal experience as evidence (unless we have evidence for that experience). It's not gospel either that the power amp and speakers will swamp other components since the spectrum of each has to be taken into account. But then if as you say your equipment had vanishly low non-linear spurs, then either level, noise or an issue with FR could be responsible. What else is there?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
Until you try it, I suggest suspending judgement. It works. It's random.
I have tried it. Many times.

But first we do it sighted. The user clicks back and forth, between reference and DUT (or DUT-recorded program, or DUT-DUT), to simply detect any timbre shift, transient variation, etc.. If a difference is detected sighted, then we go blinded. If detectable differences remain, we try to consistently identify a specific tell and stop there. We look at the screen and note the track. Wash, rinse, repeat.

If our "tell" is identified 8/10, we consider it a positive, and do it again. Another 8/10 is qualified. It means that the DUT is not giving us a correct image. Anything less than 2X 8/10 is a null.
Oh? All of a sudden you got results of 8 out of 10 right as I suggested? As I said above, I have done this kind of testing. But never do I keep track of the outcome and get to 8 out of 10 right this way. That requires note keeping that is absent in your description.

My sense is that all you have done is play A then B and go back a couple of times and call it done. Not worth much but not zero either.

I suggest doing the test formally as I have suggested and you are now saying. This time keep written notes and show us those notes when done.

As to the hardware fixture, I am still not clear what you are testing and how. We are talking about two different pre-amps, right? If so, you need to feed them common signal and then have an AB switcher on the output. That switcher must not be electronic, nor does it need to be. If it is electronic then we have to worry about transparency of that. The monitoring path after that needs to be 100% pure. So suggest using an ultra low distortion and noise path for that. Otherwise, you are compounding distortion and noise on top of each other. A mess.

On second thought, just post some ABX tests of 320 kbps MP3 versus original for us. It will be a highly sobering experience if you have not been through it. Here is an example for you:

==============
foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/19 19:45:33

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44_01.mp3

19:45:33 : Test started.
19:46:21 : 01/01 50.0%
19:46:35 : 02/02 25.0%
19:46:49 : 02/03 50.0%
19:47:03 : 03/04 31.3%
19:47:13 : 04/05 18.8%
19:47:27 : 05/06 10.9%
19:47:38 : 06/07 6.3%
19:47:46 : 07/08 3.5%
19:48:01 : 08/09 2.0%
19:48:19 : 09/10 1.1%
19:48:31 : 10/11 0.6%
19:48:45 : 11/12 0.3%
19:48:58 : 12/13 0.2%
19:49:11 : 13/14 0.1%
19:49:28 : 14/15 0.0%
19:49:52 : 15/16 0.0%
19:49:56 : Test finished.

----------
Total: 15/16 (0.0%)
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
I've made no claims to being an expert listener, or even that my AB testing is accurate.
And hence the reason a) you will fail the test I am suggesting and b) puts doubt that you can reliably hear any of the stuff you have been reporting.

If you are hearing such effects, it is likely due to frequency response errors or other simple things that we can readily measure. Can you post measurements of two sample units that sounded different to you so we know what we are talking about?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
Around this time, my friend who recorded the San Francisco Symphony (Jack Vad) heard about my micamp design. He asked to borrow some channels, so I lent them. He called back a couple weeks later, asking me to build him 8 channels.
You know how many stories there are in audio? Millions. It never means anything. People don't know how to perform proper, controlled testing of audio. They get something new to test. Their brain works differently now, focusing on what the new device is doing. And Bam! They hear more detail. More air. More nuance. More musicality. Not because the near gear did any of that. But because they are now listening differently.

Because of this faulty method -- which is used by consumers and pros alike -- everything seems to be an improvement. Folks literally sell boxes with dirt in them ("grounding boxes) and a single wire and countless testimonials exist that they do wonders for audio. If we are to believe your story as any kind of valid evidence, we have consider all of these other stories as real too. You want to go there?

Where would I read any paper in Audio Science/AES where such evidence is provided for anything useful? You advocate such, yet tell us stories like this?

I tested the PS Audio DS DAC which clearly distorts the sound due to use of cheap and improper transformers on the output. That DAC came about exactly as your story. A mixing engineer advocated for it, and PS audio decided to productize it. Result is that it both measures bad and sounds bad.

Ultimately you have not shown any evidence of your testing that has any value in this forum. And you keep saying when I show the science and psychoacoustics, it must not matter. Not sure how to advance the conversation anymore.
 

signalpath

Active Member
Forum Donor
Joined
Aug 1, 2019
Messages
112
Likes
102
Oh? All of a sudden you got results of 8 out of 10 right as I suggested?

8/10 x2 is our standard pass/fail. Has been for probably 2 decades. This is also a standard threshold in ABX testing literature, as you know (often 3x @ 8/10 is preferred). What are you inferring?

If you dig back a few days, I mentioned this "8 of 10" metric before you brought it up. Here's the conversation,

"If we can detect a difference in 8 of 10 trials, it is a positive. The harder part is determining if the difference is better, worse, different, or a partial mix. It's hard work, as you know. "

My sense is that all you have done is play A then B and go back a couple of times and call it done.

Huh? Your "sense?" That's not science ;). That's belief. What I've described is our AB method. If you "sense" that I'm not being honest and forthright, then what's the point here? Why bother?

People don't know how to perform proper, controlled testing of audio ... And hence the reason a) you will fail the test I am suggesting and b) puts doubt that you can reliably hear any of the stuff you have been reporting ... If you are hearing such effects, it is likely due to frequency response errors or other simple things that we can readily measure ... Not sure how to advance the conversation anymore.

I have no way to respond to this. If you already know all these things, then why bother? A perplexing conversation. You seem to know so much about my future. (emphasis mine)

On second thought, just post some ABX tests of 320 kbps MP3 versus original for us. It will be a highly sobering experience if you have not been through it. Here is an example for you:

I could post 2 sound files from a recent micamp electrolytic coupling capacitor listening test (-125EIN / -115THD). One file would be the worst of the bunch (10/10) and one would be the best (the series capacitor we selected for the new product). The third file would be the original reference. These are 20 minute files, 48 / 24 IIRC. Could you personally detect the differences? Have no idea. And if you -did- detect a difference, based on the apparent lack of trust in these recent responses, would you trust it?

This is why a fresh, on-site, real-time, neutral observer test is a better solution. I'm willing to do it in our lab, with our methods (which you've not yet shown to be invalid in any way).
 
Last edited:

signalpath

Active Member
Forum Donor
Joined
Aug 1, 2019
Messages
112
Likes
102
If the distortion is fully concentrated in a discontinuity such as crossover distortion IRCC Earl Geddes found this level easily audible.

I was there for Earl's AES paper, and had a lengthy chat with him afterwards, and later via emails. That is a super duper paper, but it left everyone wanting more, including myself. There's so much more to the "distortion story" that needs to be told, and scientifically vetted. It's hard work, and not a lot of people with the inclination/time/resources to make it happen.
 
Last edited:

signalpath

Active Member
Forum Donor
Joined
Aug 1, 2019
Messages
112
Likes
102
Of course when one monitors using a Pass amp (which could have relatively high 2nd harm. distortion it could be really difficult to test for low distortion levels (say below 0.05%)

I love Nelson (he's been my Sierra Foothills neighbor for 30 years), but I failed to mention that most of our AB testing these days is done with cans. I, personally, find them to be more revealing of subtle timbre, even with access to a great room like ours. Reference cans include Senn 600 and 650, Beyer 770 and 990, and the AT 50X. If we get an 8/10 positive (or better), we'll often swap cans around and retest.

For spatial information, speakers still rule the roost, and the Dunlavy first-order SC-V's are among the most spatially revealing I've ever encountered. But they are also surgical in their sweet spot. You pretty much need to keep your head in a vice for their full soundstage potential.
 

pkane

Master Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
5,632
Likes
10,205
Location
North-East
8/10 x2 is our standard pass/fail. Has been for probably 2 decades. This is also a standard threshold in ABX testing literature, as you know (often 3x @ 8/10 is preferred). What are you inferring?

If you dig back a few days, I mentioned this "8 of 10" metric before you brought it up. Here's the conversation,

"If we can detect a difference in 8 of 10 trials, it is a positive. The harder part is determining if the difference is better, worse, different, or a partial mix. It's hard work, as you know. "



Huh? Your "sense?" That's not science ;). That's belief. What I've described is our AB method. If you "sense" that I'm not being honest and forthright, then what's the point here? Why bother?



I have no way to respond to this. If you already know all these things, then why bother? A perplexing conversation. You seem to know so much about my future. (emphasis mine)



I could post 2 sound files from a recent micamp electrolytic coupling capacitor listening test (-125EIN / -115THD). One file would the worst of the bunch (10/10) and one would be the best (the series capacitor we selected for the new product). The third file would be the original reference. These are 20 minute files, 48 / 24 IIRC. Could you personally detect the differences? Have no idea. And if you -did- detect a difference, based on the apparent lack of trust in these recent responses, would you trust it?

This is why a fresh, on-site, real-time, neutral observer test is a better solution. I'm willing to do it in our lab, with our methods (which you've not yet shown to be invalid in any way).

Upload the files, please. The more examples with very low distortion but still audible differences, the better. I assume they are all using the same test signal?
 

signalpath

Active Member
Forum Donor
Joined
Aug 1, 2019
Messages
112
Likes
102
Upload the files, please. The more examples with very low distortion but still audible differences, the better. I assume they are all using the same test signal?

Yes, identical reference program on all files. One is raw ref. The other two are thru identical HV-3 micamp (40dB gain, IIRC), except just different electrolytic capacitors in the front-end series path. (we source a high quality film cap that - in this application - sounds even more accurate than the best electrolytic, but in this new product it simply doesn't fit. The thing is HUGE). Keep in mind, the sonic difference between the best and worst is very subtle, and not something the casual listener would detect. It takes time and practice and patience.

And this brings up an interesting question. If the worst sounding cap and best sounding cap (assuming we detect a true difference) do not impact THD or noise (SINAD) or FR in any way, then what is causing the detected timbre shift? The answer, I think, begins with a reflection on transient (one-off) distortions vs. periodic (static) harmonic distortions. But that's a conversation for another time.
 
Last edited:

pkane

Master Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
5,632
Likes
10,205
Location
North-East
Yes, identical reference program on all files. One is raw ref. The other two are thru identical HV-3 micamp (40dB gain, IIRC), except just different electrolytic capacitors in the front-end series path. (we source a high quality film cap that - in this application - sounds even more accurate than the best electrolytic, but in this new product it simply doesn't fit. The thing is HUGE). Keep in mind, the sonic difference between the best and worst is very subtle, and not something the casual listener would detect. It takes time and practice and patience.

And this brings up an interesting question. If the worst sounding cap and best sounding cap (assuming we detect a true difference) do not impact THD or noise (SINAD) in any way, then what is causing the detected timbre shift? The answer, I think, begins with a reflection on transient (one-off) distortions vs. periodic (static) harmonic distortions. But that's a conversation for another time.

I think we might be able to find some answers if you share the files. There’s a lot of experience and expertise on ASR. We can all put our heads together :)
 
Top Bottom