AES Paper Digest: Do Audio Op-amps Sound Different?

[Jakob1863]

@amirm ,

<snip>

So it is a double blind, AB preference test.

There is a second phase but that did not generate significant results so I won't go into that.

The description in the paper is a bit strange, but isn´t it actually the other way round?
In the other part (the second phase) the authors used a 3AFC protocol (sort the odd one out) as a discrimination and used the preference test results only when the discrimination part (i.e. the 3AFC ) was considered as significant.

The results table (according to the description in the Appendix) lists the 3AFC results (_not_ the results of the A/B preference test) beside the measured numbers.

Anywhere you see a "P" value less than 0.05, the results are "statistically valid."

The term "statistically valid" means something different; the p-values below 0.05 mean the results are "significant" as the authors set their decision criterion at this threshold.

 

[Blumlein 88]

This is a very interesting paper, thanks for bringing it up. A couple small flaws I might have pointed out if I were a referee. Several have noted that the frequency responses were not maintained equal- and the test subjects were relatively young. Several have also noted that the distortion was relatively high, which should perhaps also have been dealt with. The conclusion that the detected differences were due to the distortion is not supported by the data- it might be true, but there's no evidence for it. Their decision to not disclose the important detail of which opamps were used is, to me, a major flaw- full disclosure is needed when making claims of this sort.

But the other missing factor takes me back to a simple experiment I mentioned in an article I did in Linear Audio a few years back where I daisy-chained some opamps configured as buffers to see how many it took before I could reliably distinguish whether they were in or out of circuit. If memory serves, it was about five, and I invited the readers to guess what the difference was that I was able to perceive. As far as I'm aware, not one guessed correctly, and at least one more mendacious sort of fellow was highly bothered by the results. I will reveal it here, especially because it relates back to disclosing opamp identities so that a good correlation could be drawn: noise floor.

I seem to remember reading about that at some point (although I thought it required six from memory).

I've done something similar with multiple loopbacks of AD/DA conversion. When it becomes audible it was due to the noise floor. Which takes many loops with modern digital gear. If my devices had 1% distortion however it wouldn't have been noise that was first audible I don't think.

 

[amirm]

I've done something similar with multiple loopbacks of AD/DA conversion. When it becomes audible it was due to the noise floor.

When I took Ethan's test, I think it was the roll off the high frequencies that i detected in blind tests. My memory is a bit hazy on that too. Either way, I am pretty sure I focused on high frequencies.

Here is the result of that test:

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:40:07
File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass1.wav
06:40:07 : Test started.
06:41:03 : 01/01 50.0%
06:41:16 : 02/02 25.0%
06:41:24 : 03/03 12.5%
06:41:33 : 04/04 6.3%
06:41:53 : 05/05 3.1%
06:42:02 : 06/06 1.6%
06:42:22 : 07/07 0.8%
06:42:34 : 08/08 0.4%
06:42:43 : 09/09 0.2%
06:42:56 : 10/10 0.1%
06:43:08 : 11/11 0.0%
06:43:16 : Test finished.
----------
Total: 11/11 (0.0%)

So 100% detection for even one pass.

Seeing how my high frequency hearing is shot , what I heard must be occurring at well below 20 kHz.

 

[Blumlein 88]

snippage..............

I'll interpret the curves from the McGill paper as showing a 1/3 wide variation at 20 KHz on the order of 2 to 5 dB. According to the graph above it would take more than 4 dB variation (gain or loss) to fail the criteria for matching which is conservative - it is much less than actual expected audibility. It's near or on the edge, but I would expect it to not be a strong effect. OTOH the threshold for audibility of isolated artifacts is around 70 dB (again conservative) and a number of the tests seem to fail that criteria. As others have mentioned numbers for distortion from IM were given, I don't recall the nature of the IM test being mentioned. THD is pretty standard but there are at least two mainstream tests for IM being SMPTE and CCIR. and they are IMO very different.

The IMD test was a 16 khz and 17 khz sine wave.

 

[restorer-john]

Not sure what their thinking is behind doing the circuits this way. Looks like they should have studied Jung's op amp book more thoroughly. It doesn't appear to mimic how the devices normally get used.

The circuit has different characteristics for the inverting and non-inverting opamps in terms of input and output impedances as seen by the devices themselves.

 

[Arnold Krueger]

The circuit has different characteristics for the inverting and non-inverting opamps in terms of input and output impedances as seen by the devices themselves.

Right. Just not typical of real world gear.

 

[oivavoi]

When I took Ethan's test, I think it was the roll off the high frequencies that i detected in blind tests. My memory is a bit hazy on that too. Either way, I am pretty sure I focused on high frequencies.

Here is the result of that test:

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:40:07
File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass1.wav
06:40:07 : Test started.
06:41:03 : 01/01 50.0%
06:41:16 : 02/02 25.0%
06:41:24 : 03/03 12.5%
06:41:33 : 04/04 6.3%
06:41:53 : 05/05 3.1%
06:42:02 : 06/06 1.6%
06:42:22 : 07/07 0.8%
06:42:34 : 08/08 0.4%
06:42:43 : 09/09 0.2%
06:42:56 : 10/10 0.1%
06:43:08 : 11/11 0.0%
06:43:16 : Test finished.
----------
Total: 11/11 (0.0%)

So 100% detection for even one pass.

Seeing how my high frequency hearing is shot , what I heard must be occurring at well below 20 kHz.

That's impressive Amir. Honest question: How did you train your ears to learn to do those things? How can I do the same?

 

[Krunok]

Right. Just not typical of real world gear.

Yep. Unlike multiple loopbacks of AD/DA conversion..

 

[Arnold Krueger]

That's impressive Amir. Honest question: How did you train your ears to learn to do those things? How can I do the same?

The files were arguably prepared in error. +/- 0.1 dB level matching was not preserved.

OTOH, I don't see that much difference > 10 khz, given what that it takes much more to be audible due to the high frequency range.

If this is retried the latest greatest FOOBAR2000 ABX plug in should be used - it takes checksums of the files to try to identify if someone is cheating by editing or renaming files.

Not saying that anybody would do such a thing but we are skeptics, aren't we?

 

[Arnold Krueger]

Yep. Unlike multiple loopbacks of AD/DA conversion..

The multiple loopbacks are often required to get positive results.

Listener training is highly critical for getting reliable results.

One highly effective form of listener training is to accentuate the audible difference being studied until it is as obvious as reasonably possible, and then reduce it to its natural state. So, if it takes 20 reps to get people on board, so be it!

However, in the end the number of repetitions should be backed off in reasonable steps until people go from hearing the artifact(s) to not hearing naturally and as comfortably as possible. Note that Ethan provided what it takes to do things that way.


And that is the crux of the problem with this paper. While they may have done more things right, they didn't say enough about it in the paper. And at the point they published their results, they didn't really have a realistic test system.AFAIK there was no sequel to this test that did things as well as they easily could be done. And that leads to a published conclusion that I will charitably call overblown.

 

[Ethan Winer]

I'll tread lightly here because I'm too busy to get into another audio forum sinkhole. But this caught my attention:

Do the results show any distinction between the discrete and integrated devices?

I just bought half a dozen of these OPA1611 op-amps:

http://www.ti.com/product/OPA1611

They're expensive at $5 each in small quantities, but the specs are astonishing at 1.1 nV / Hz with 0.000015% distortion.

For half the price you can get the OPA1602 which has two op-amps in one package, with only slightly less amazing performance:

http://www.ti.com/product/OPA1602

 

[Jakob1863]

<snip>

I'll interpret the curves from the McGill paper as showing a 1/3 wide variation at 20 KHz on the order of 2 to 5 dB. According to the graph above it would take more than 4 dB variation (gain or loss) to fail the criteria for matching which is conservative - it is much less than actual expected audibility. It's near or on the edge, but I would expect it to not be a strong effect. OTOH the threshold for audibility of isolated artifacts is around 70 dB (again conservative) and a number of the tests seem to fail that criteria. As others have mentioned numbers for distortion from IM were given, I don't recall the nature of the IM test being mentioned. THD is pretty standard but there are at least two mainstream tests for IM being SMPTE and CCIR. and they are IMO very different.

Which way do you calculate the "2-5 dB" ?
Usually the the bandwidth of a 1/3 octave filter with mid frequency of 20 kHz would be ~4.8 kHz and so the difference below 1 dB; even at 30 kHz the max difference would be roughly around 1 dB?!

 

[Arnold Krueger]

Which way do you calculate the "2-5 dB" ?
Usually the the bandwidth of a 1/3 octave filter with mid frequency of 20 kHz would be ~4.8 kHz and so the difference below 1 dB; even at 30 kHz the max difference would be roughly around 1 dB?!

I didn't calculate it, I read it out of the cited AES paper. Amir posted it near the front of the thread. This is it:

And you are right - the difference < 20 Khz is more like a fraction of a dB

Thanks for the help!

It just strengthen's the other evidence - the most audible difference is probably the over-all level difference.

 

[amirm]

That's impressive Amir. Honest question: How did you train your ears to learn to do those things? How can I do the same?

Thanks. It took me a few months of training back when and hours and hours of practice. It also helps a lot to be able to guess where the difference could be. High frequencies tend to be the place. And most of the time you want to focus on notes where that high frequency detail is isolated (i.e. there is less masking).

The trick on training is to start with cases where the distortion is obvious. Then gradually you scale to smaller and smaller levels of distortion. Over time your ears become very good at hunting for distortions and are able to tease them out from the rest of the music.

I always say the training turns you into a human instrument. It sharply increases your accuracy and ability to hear small distortions even in those you are not trained in.

 

[Jakob1863]

The doctoral thesis from Peter-Eric Gaskell offers a lot more information:

digitool.library.mcgill.ca/thesisfile141279.pdf


@Arnold Krueger ;

<snip>
Thanks for the help!

It just strengthen's the other evidence - the most audible difference is probably the over-all level difference.

Always trying...

But what do you mean by the over-all level differences? Beside the different roll off at the upper frequencies there seems to be no level difference?!

 

[Cosmik]

How did you train your ears to learn to do those things? How can I do the same?

Why would you want to?! Don't do it!
The ideal, in my opinion, would be to *not* have any great measurements-emulating 'listening abilities' - it would be a curse, and a pale imitation of real test gear. What would make sense would be to pay attention to real 'objective' measurements in your choice of equipment, but always with a firm grasp of proportion what matters and what things really should cost.

Listening tests are a mug's game. Once you cross the rubicon you can't listen for enjoyment any more!

 

[Jakob1863]

<snip>

Listening tests are a mug's game. Once you cross the rubicon you can't listen for enjoyment any more!

Any objective data available to back that claim up?

 

[oivavoi]

Why would you want to?! Don't do it!
The ideal, in my opinion, would be to *not* have any great measurements-emulating 'listening abilities' - it would be a curse, and a pale imitation of real test gear. What would make sense would be to pay attention to real 'objective' measurements in your choice of equipment, but always with a firm grasp of proportion what matters and what things really should cost.

Listening tests are a mug's game. Once you cross the rubicon you can't listen for enjoyment any more!

Hehe - you might be right, dear Cosmik!

A recent example/illustration of what you say. I recently re-read parts of our new member dr. @Floyd Toole 's wonderful book, the latest edition. I was especially interested in the part on multichannel, phantom sources, etc. Toole explains that one of the problems with phantom images is that comb filtering creates a dip around 2 khz, due to crosstalk (I'm not sure I completely understand the technicalities of why this happens - part 7.1.1. of the book for those interested). While comb filtering in general is not that audible, this dip is, apparently. So I started investigating. Listening in the near-field, I put on music with quite a bit of content just around those frequencies, like female soprano vocals, violins, etc. And I started listening. I moved my head first towards the speakers, then towards the centre/sweet-spot. There was no doubt: The phantom center was markedly darker than the direct sound!

The strange thing is that this has sensitized my ears to a phenomenon I didn't notice before at all. I now perceive near-field listening as less ideal than I did before, because this dip becomes noticable. Whereas before, I thought that the phantom center in the near-field was great.

But just like Toole states, comb filtering in general becomes less pronounced when room reflections fill in the direct sound. So when I move the listening position further back from the speakers, so that I get a different ratio of direct vs indirect sound, I'm not able to perceive this notch any more. I can only perceive it when sitting in the near-field where the direct sound dominates. Unfortunately I do quite a bit of listening to classical music sitting in the near-field when doing creative work on my computer...

Would I be better off being blissfully unaware of this tonal dip in the phantom center? I guess you can make a philosophical case for that.

 

[Wombat]

Any objective data available to back that claim up?

Subjective but noticed: There are threads here for sharing music and listening enjoyment. Those who talk a lot about listening minutiae tend to be somewhat absent in those postings.

 

[oivavoi]

Subjective but noted: There are threads here for sharing music and listening enjoyment. Those who talk a lot about listening tend to be somewhat absent in those postings.

I think that observation is correct. Focusing too much on the technical aspects of listening may detract from actual listening. But FWIW, I listen to music between 1 and 2 hours every day in a focused way, and play background music lots of the time when I work, but rarely talk about it in forums like this. Why? I don't know. Perhaps because I don't have the same need for feedback on music etc.