DAC audible differences are finally proven with ABX?

[pedrojoaquimborges]

Anyone reading this post of yours from nearly 10 years ago on AS, will conclude you haven't changed one bit:

"I've read Amir's forum, and his explanations of how he does his work. I've also read criticisms of his work and methods.

I've come to the conclusion that his methods aren't trustworthy and that he seems to have an agenda. You regard him more faorably. We can agree to disagree. Doesn't mean I have an agenda about Amir. Means I have an opinion based on what I've read. It is as legitimite as yours.

I've read Stereophile for years and haven't yet seen any reason I shouldn't trust JA and what he does. So I trust him and not Amir. The fact that they produced similar measurements of the Regen is one data point among many. Not enough to outweigh what I consider to be many negative data points I've come across in getting to my conclusion about Amir."

Useless, emotional complaining then, and now.

Get the files. Get... the... files.

This is in fact kind of the end-all argument. Providing files would be next to no work compared to running the ABX test and writing a whole report, especially considering Josh has as many units available as described in the article. And it opens the door to peer review/testing.... an essential element for scientific method.

As a personal note, despite the ridiculous level of discourse reached in the AS post, I also do not particularly care for the circular arguments we are now engaging in right here. They are, most of all, disconnected from the content of the matter - audio science, of which there is much to discuss in this case - and entirely focused on social context.

 

[donjoe0]

How do YOU know that is true?

I've actually read enough audio studies to know the tragic numbers of listener participants in most of them (vs. medical standards), and the scarcity of studies focusing specifically on this issue of coloration/FR-non-flatness detection ability. Reminder also, for the hard of hearing at the back of the class: when I see ASR bullies repeatedly skipping the step of showing what studies their bullying is based on, I am perfectly justified in saying "there are insufficient studies to support such bullying". Basic logic should've told you that's an unassailable inference, but it seems nature has not endowed all of us with the superpower of basic logic.

“Colouration detection ability” is your made up term and it’s utterly undefined

Just like every sentence you posted was "made up", yet curiously you expect people to understand it based on a preexisting familiarity with each separate word in it, and with how words in human language work together to encode meaning. Not only that, but if you suspected your readers of any intelligence at all, you might even go so far as to expect them to recognize the same concept in different writings available in other places even if it's named with different words! Crazy stuff, human language and reasoning - it can be shocking to the uninitiated.

Quote your sources.

First place I saw any information on it was here: https://www.audiosciencereview.com/...-of-amp-and-dac-measurements.5734/post-127757
It was called "frequency response deviation", but also quoted from Floyd Toole as "spectral tilt". Zero studies cited.

Then from the current thread I got this: https://www.audiosciencereview.com/...re-finally-proven-with-abx.72444/post-2650711
Here it's called "audibility of resonances" or of "modifications of timbre". Reported tests set up by Toole & Olive seemed to have used - hold on to something - two, four, and a whopping six participants for separate test conditions and endpoints.
Multiple preexisting studies are cited in there, most even older and near-impossible to find, some just paywalled. What I could dig up was a '75 Fryer write-up where he reports using "several" listeners, and since he splits them into skilled/unskilled and female/male, I guess he must've used at least 4 people. :')) A Moulana '75 paper thanks a total of nine subjects by name right in the preamble.

Moulana, BTW, calls their study's focus "tonal colouration" - obviously "jibberish" and "trolling", right? Just like Toole's Loudspeaker Measurements and Their Relationship to Listener Preferences, 1986, going on and on with these "made-up terms" and obvious "trolling" about "audible colorations":

there are nevertheless some recognizable patterns in this collection of findings and they can be described as follows: audible colorations are often caused by resonances in a loudspeaker system. These resonances will be revealed, in varying degrees, in both steady state amplitude responses and time-domain responses to interrupted sounds.

In order for amplitude responses to reveal unambiguous clues to the presence of resonances the measurement must have high resolution in the frequency domain and incorporate some form of spatial averaging to remove the visual clutter caused by acoustical interference. This done, evidence of potentially audible coloration will take the form of upward thrusting peaks deviating from the underlying general shape of the curve. Wide, low-amplitude bumps can be as annoying as much higher, narrow peaks. Dips in the amplitude response appear to be much less important than peaks. The technique is limited, however, in that it will not necessarily reveal the low-Q resonances 10 to 20 dB below the level of the primary signal that have been shown to be sources of audible coloration.

Well-known "troll" Floyd Toole, ladies and gentlemen.

And to finish up, later still in this thread I got this link: https://www.audiosciencereview.com/forum/index.php?threads/“human-ears-are-better-than-measurements-”-“humans-can-hear-things-we-can’t-even-measure-”.72288/post-2651009
These are not strictly on coloration audibility but rather focus on JNDs, but I'll accept them for now considering the scarcity of direct evidence, and for the sake of my point on typical numbers of participants in audio studies:
Yost & Nielsen 1985 - can't find it, unkown number of participants
Jestead 1977 - three participants
Johnson 1993 - paywalled
Norcross et al. 2003 - paywalled, but seems to have had fifteen participants for at least one of the tests, based on the reposted figure (mis-cited in discussions so far as "Scott et al."! CC: @RandomEar )

Where did you pull 0.08 from?

Same thread you're replying to without even reading it already showed you where I got it:
https://www.audiosciencereview.com/...re-finally-proven-with-abx.72444/post-2651081
(Toole & Olive 1988 study, if assuming correlated resonances when doing the decibel arithmetic, but also seems to be corroborated by Norcross et al. 2003 if we assume JNDs are directly translatable to coloration detection ability at normal/high listening volumes.)

I wholeheartedly approve of your change of avatar, you may proceed with my full blessing.

I'm gonna guess that cites for parameters of human hearing may be found in this classic text of psychoacoustics?
In, say, oh, I dunno, maybe chapter 7?
donjoe0, get our your wallet!

That's not how forum discussions work. You either know for sure if that text answers my explicitly described concerns - but in that case you could directly quote the two numbers needed for that: lowest amplitude difference detectable by the best listener under the best conditions, and number of participating listeners in the study - OR you have no idea what the text says that is relevant to this specific discussion, in which case you're just using that link as a bludgeon and not as a proper argument, like so many others love to do around here.

 

[solderdude]

How many studies do you need ?
What do you look for in those studies ?
How many participants, age groups, level of training would you feel is required ?
Have you ever determined your own hearing limits in these particular aspects?
If so what thresholds did you find ?
Have you done this with music instead of white noise or tones ?
Why would level matching within 0.1dB not be sufficient ?

 

[pma]

As a personal note, despite the ridiculous level of discourse reached in the AS post, I also do not particularly care for the circular arguments we are now engaging in right here. They are, most of all, disconnected from the content of the matter - audio science, of which there is much to discuss in this case - and entirely focused on social context

Unfortunately.

 

[donjoe0]

How many studies do you need ?
What do you look for in those studies ?
How many participants, age groups, level of training would you feel is required ?

Thousands of participants, medical-grade reliability. The more demographically diverse the merrier.
I'm looking for the absolute limits of human audibility, and in it a proper foundation for all this talk of "audibly transparent" DACs... ORR... alternatively a toning down of the aggression level in DAC discussions from a few people who haven't yet recognized how poorly they've understood the limits of audibility so far. Some scientist-like moderation in the tone of replies - that would also work for me until we get clearer on what the evidence says and how far it can be extrapolated.

Have you ever determined your own hearing limits in these particular aspects?
If so what thresholds did you find ?
Have you done this with music instead of white noise or tones ?

Irrelevant, see above.

Why would level matching within 0.1dB not be sufficient ?

*FR flatness compliance within 0.1 dB
Because we already have evidence that some people in some conditions can detect FR colorations or resonances down to 0.08 dB, and even this might prove to be too conservative if we get better population sampling in future studies.

 

[Mort]

Thousands of participants, medical-grade reliability.

Sure, you foot the bill for that. You realize we are in a capitalist economy where money to do things like that have to be sourced from somewhere?

 

[Mort]

ASR has a resident sea lion, one that I have blocked because the writing is so prolific and pointless that it resembles AI slop.

 

[Geert]

I'm looking for the absolute limits of human audibility, and a proper foundation for all this talk of "audibly transparent" DACs... ORR... alternatively a toning down of the aggression level in DAC discussions from a few people

If the audibility threshold turns out to be lower than 0,1dB, then what would that do in practice for the discussion about difference in the sound quality of audio components? That discussion is not about the existence of small differences, like Josh observed (for whatever reason). It's about things like veils being lifted and rhytm and pace when listening to ultra expensive audio components. Move your head 2 inches and you already have your 0,1dB difference.

 

[pma]

I'm looking for the absolute limits of human audibility

Then the best you can do is seeking for scientific papers, I am talking about real science and not about a chat at a public forum. There is more than enough good literature. And you have to do the literature review job yourself.

 

[pkane]

I haven't seen the source code, but my guess is that it can estimate a clock drift once it has aligned the start of both samples. To compensate, it needs to do a Dynamic Time Warping, which in turn means aligning any data to fit the source over the given a given window (say 4ppm). This means that I can move freely in that window, but unfortunately would eliminate actual differences in the time domain.

That together with gain alignment, one could ask if we are actually comparing fairly with DeltaWave, since we adjust both axes at the same time to fit the source.

Not sure which two axis you're talking about. Clock drift is a given between any two unsync'ed devices. Measuring device performance using a null test in DeltaWave gives you the option of correcting for clock drift, so it's your choice to use this option or not. If you don't, you'll be measuring primarily the difference due to clock drift, especially if it's sizeable.

 

[pma]

Paul, I think the only valid question is if it is audible or not. If there is a clock drift in the DAC then it is there and one will do a listening test including the drift.

 

[solderdude]

Thousands of participants, medical-grade reliability. The more demographically diverse the merrier.

For what specific aspects exactly ? There are SO much different aspects of audibility thresholds.
Besides why would you need to have absolute thresholds with test signals while we are talking of music which has different thresholds ?

OR... a toning down of the aggression level in DAC discussions from a few people who haven't yet recognized how poorly they've understood the limits of audibility so far.

Ah, so you draw the card of a few people that, according to you, have a poor understanding of audibility limits an thus must tone down 'agression' because this rubs YOU the wrong way... Really ?

Some scientist-like moderation in the tone of replies - that would also work for me until we get clearer on what the evidence says and how far it can be extrapolated.

So you feel offended by the tone of some people and for that reason you need more research (papers) so you can 'give them some scientific arguments'?

Are you for real ?

*FR flatness compliance within 0.1 dB
Because we already have evidence that some people in some conditions can detect FR colorations or resonances down to 0.08 dB, and even this might prove to be too conservative if we get better population sampling in future studies.

So we can't say matching within 0.1dB is 'sciency' enough because in one specific kind of test with tones someone noticed an increase in level of 0.08dB which was a harmonic added to a fundamental and could hear that ? And you claim that was the overall amplitude increase instead of 2 tones combined ?
And for that we need to have millions of $ in additional research to 'prove' that 0.1dB is not scientificall sufficient to match levels ?
Come on ... get a life.


We are talking about an ABX test and level matching in steps of 0.1dB.
If you want less; the firmware in the comparator could be changed specifically for you and then we could have 0.03dB steps... but this probably isn't sciency enough either because the 0.08 is too conservative.

 

[pkane]

Paul, I think the only valid question is if it is audible or not. If there is a clock drift in the DAC then it is there and one will do a listening test including the drift.

I'm certain that by itself, it's not audible when the clock drift is small. But when you rapidly switch between two devices that were synched at the beginning of the test, after a few hours there could be an audible delay difference that could be a tell.

 

[kemmler3D]

Thousands of participants, medical-grade reliability. The more demographically diverse the merrier.
I'm looking for the absolute limits of human audibility, and in it a proper foundation for all this talk of "audibly transparent" DACs... ORR... alternatively a toning down of the aggression level in DAC discussions from a few people who haven't yet recognized how poorly they've understood the limits of audibility so far. Some scientist-like moderation in the tone of replies - that would also work for me until we get clearer on what the evidence says and how far it can be extrapolated.


Irrelevant, see above.


*FR flatness compliance within 0.1 dB
Because we already have evidence that some people in some conditions can detect FR colorations or resonances down to 0.08 dB, and even this might prove to be too conservative if we get better population sampling in future studies.

I don't think this is an unreasonable idea as far as it goes, but I also think the results would almost definitely not justify more DAC performance than we already have.

Suppose we find out 0.1% of the population can hear 23Khz pretty easily. No problem, almost every DAC can run at 96Khz.

Suppose we find out that most people struggle to hear distortion at -70dB (generous) but 0.1% can hear it at -90dB. No problem, many cheap DACs are already way down below -110dB.

Same line of reasoning for noise.

This leaves us looking for novel modes of audible distortion, but so far none have been demonstrated, so...

In every case the theoretical limit of audibility is also beyond the practical one when listening to music.

I also think the existence of lossy codecs has strong implications for our understanding of human hearing that aren't fully appreciated in these discussions. The masking effect is not subtle, and the thresholds of hearing at every audible frequency have been exploited to develop MP3 etc. In some sense this work goes far beyond lab work on thresholds, and every ABX performed between lossy and lossless is a data point.

 

[Geert]

It would instantly confirm that forum bullies who have relied mostly on published DAC measurements, not on reading actual studies(!), never actually had a good scientific foundation for their almost-religious conviction that "all DACs these days are audibly transparent"

There is no believe over here that all DAC’s these days are audible transparent. You’re missing the part that this goes for DAC’s that measure transparent. Even then you might hear a difference in filters, but that probably doesn't matter for the typical audiophile who's more than 50 years old.

This is considering that only a small proportion of DAC measurements to date have even zoomed in to 0.1 dB resolution,

The graphs that Amir provides have enough resolution to show if there's anything that requires further investigation:

"look at this moron thinking they can hear artifacts in a DAC's response

No one is called a moron on this forum. A questionable claim is typically answered with questions about the testing methodology and tips for improvement. However, ASR members are literally called morons on different other forums. Do you also intervene there?

 

[rjp]

Suppose we find out 0.1% of the population can hear 23Khz pretty easily. No problem, almost every DAC can run at 96Khz.

Suppose we find out that most people struggle to hear distortion at -70dB (generous) but 0.1% can hear it at -90dB. No problem, many cheap DACs are already way down below -110dB.

Same line of reasoning for noise.

This leaves us looking for novel modes of audible distortion, but so far none have been demonstrated, so...

In every case the theoretical limit of audibility is also beyond the practical one when listening to music.

This is good, but can we take it even further?

It seems that so many of the arguments here could be addressed with a single sticky post that states the reasons ASR believes certain tests are sufficient to make certain declarations regarding audio transparency. Readers can agree or disagree, but at least the reasoning is clearly stated, vetted, and doesn't suffer in translation to so many individual arguments.

For example, say a device under test receives input signal sig_in, and produces output signal sig_out.

(1) If we perform measurements A, B, and C (list all required), and all results exceed their respective thresholds TA, TB, and TC, we declare sig_out to be audibly indistinguishable from sig_in by an average human listener.

(2) Furthermore, we believe that Tests A, B, and C comprise a sufficient set of tests that cover all modes of audible distortion based on (insert reasoning/links here).

 

[kemmler3D]

This is good, but can we take it even further?

It seems that so many of the arguments here could be addressed with a single sticky post that states the reasons ASR believes certain tests are sufficient to make certain declarations regarding audio transparency. Readers can agree or disagree, but at least the reasoning is clearly stated, vetted, and doesn't suffer in translation to so many individual arguments.

For example, say a device under test receives input signal sig_in, and produces output signal sig_out.

(1) If we perform measurements A, B, and C (list all required), and all results exceed their respective thresholds TA, TB, and TC, we declare sig_out to be audibly indistinguishable from sig_in by an average human listener.

(2) Furthermore, we believe that Tests A, B, and C comprise a sufficient set of tests that cover all modes of audible distortion based on (insert reasoning/links here).

I think this would be good and I think amir started a FAQ. however, @donjoe0 is going a level deeper and contending that small studies don't really establish a "limit of human hearing" that feels reliable. As far as that goes I think it's fair, although I don't know the literature well enough to say whether the objection is actually fair.

My point is just that even if we established that the real limits of audibility are much more demanding than we thought, DACs are already most likely performing beyond that anyway.

 

[pma]

My point is just that even if we established that the real limits of audibility are much more demanding than we thought, DACs are already most likely performing beyond that anyway.

Sure.

 

[voodooless]

Instead of testing thousands of random people, why don’t the people that claim they can hear the differences back up their claims by doing the tests. Seems like a much faster and more efficient way to come at the desired results.

 

[jabbr]

I personally wrote that Josh's attitude, in some situations, really resembled that of a troll, and it wasn't a lazy insult at all, but an observation.
The troll has the ability to insert himself into a community and touch raw nerves by saying enough and convincingly enough, returning to some points and deliberately glossing over others, sparking discussion and then returning to the shadows.
Even if that wasn't his intention, for long stretches Josh behaved EXACTLY like a high-profile troll would.
First throw a bomb that is a very strong trigger on ASR "I can hear the differences between competent DACs in the blind in Abx"
then he begins to respond to the (inevitable) explanations and requests for data and clarifications, he responds with incredible precision to some but without ever really providing useful information. When he is forced to provide recordings that he evidently possesses, he barricades himself behind a
"I'm sorry, I don't have time for this, I have other things to do"
however he continued to stay on the thread and reply with dozens of lines to other things, a very strange and in some ways inconsistent behavior.
Even the idea that a user so immersed in this sector as to buy an abx box, test some DACs, write on a specific site like AS obtaining exceptional results and then, when faced with requests for peer review, say things like "no, I'm not interested in reviewing the results, I only did it for myself and I'm fine with that" is very strange and not very coherent.

Let's get real about this. Josh wrote an article for another site, not for here. The article caused quite a stir and a discussion started here. Josh didn't start the discussion here nor appeared as a troll to start a kerfuffle. He responded to posts about him and his article. He never claimed to have the same modus operendi as people here and posts here.

Given all that he was treated harshly. Demands for recordings from him were made, yet he claims not to have a good way to record. Given the harsh criticisms of his methods and materials I can see why he declined to make recordings.

Remember the article was not posted here. The bomb was thrown somewhere else. He was not welcomed warmly nor politely, that's ok I've seen worse at technical/scientific committees (I have served as a invited external expert on highly technical scientific committees and seen many communication styles)

I don't think this is an unreasonable idea as far as it goes, but I also think the results would almost definitely not justify more DAC performance than we already have.

Suppose we find out 0.1% of the population can hear 23Khz pretty easily. No problem, almost every DAC can run at 96Khz.

Suppose we find out that most people struggle to hear distortion at -70dB (generous) but 0.1% can hear it at -90dB. No problem, many cheap DACs are already way down below -110dB.

Same line of reasoning for noise.

This leaves us looking for novel modes of audible distortion, but so far none have been demonstrated, so...

In every case the theoretical limit of audibility is also beyond the practical one when listening to music.

I also think the existence of lossy codecs has strong implications for our understanding of human hearing that aren't fully appreciated in these discussions. The masking effect is not subtle, and the thresholds of hearing at every audible frequency have been exploited to develop MP3 etc. In some sense this work goes far beyond lab work on thresholds, and every ABX performed between lossy and lossless is a data point.

Given the ABX data presented there would need to be:
1) a substantial anomaly in the ABX setup
2) a substantial difference in the performance of A vs B under the testing conditions -- that is to say: are these conditions possible in the real world?