DAC audible differences are finally proven with ABX?

[voodooless]

Remember the article was not posted here

Why does this matter? It’s for free on the public internet. Anyone can discuss and comment on the damn article all they like.

 

[jabbr]

Why does this matter? It’s for free on the public internet. Anyone can discuss and comment on the damn article all they like.

Simply regarding whether he was trolling this site. He was responding to the discussion here. He didn't start it.

 

[kemmler3D]

Let's get real about this. Josh wrote an article for another site, not for here. The article caused quite a stir and a discussion started here. Josh didn't start the discussion here nor appeared as a troll to start a kerfuffle. He responded to posts about him and his article. He never claimed to have the same modus operendi as people here and posts here.

Given all that he was treated harshly. Demands for recordings from him were made, yet he claims not to have a good way to record. Given the harsh criticisms of his methods and materials I can see why he declined to make recordings.

Remember the article was not posted here. The bomb was thrown somewhere else. He was not welcomed warmly nor politely, that's ok I've seen worse at technical/scientific committees (I have served as a invited external expert on highly technical scientific committees and seen many communication styles)

Given the ABX data presented there would need to be:
1) a substantial anomaly in the ABX setup
2) a substantial difference in the performance of A vs B under the testing conditions -- that is to say: are these conditions possible in the real world?

My point on ABX was more that there has been (less so now, I guess?) sustained and robust ABX testing between different lossless codecs over time. Hydrogenaudio was doing a lot of those for a while, not sure if they still are. These codecs work on the principle of thresholds of hearing, so the process of developing and improving MP3, OGG, AAC etc. implicitly generated a lot of data on thresholds of hearing. And we know these codecs work...

 

[voodooless]

He didn't start it.

What does that matter? One can start trolling at any time one likes.. there aren’t really rules for this

 

[RickS]

Suggest it is time to move on to the next phase of this discussion and focus on a productive evaluation of the effort.

In the near future, will find out more about the test gear. Until then, please cut the speculative posts and banter over how others reacted to the OP.

Thanks!

 

[solderdude]

Demands for recordings from him were made, yet he claims not to have a good way to record.

He has an Cosmos ADC which is perfectly fine for that goal.
It is not complicated either.
He made nulls so he actually had to make a recording so the claim is suspect.

What was asked was a recording of 2 DACs he owns made via the box. His used files via his box into his Cosmos would have given so much information and is very, very easy to do.
Those files could then be AB'xed in foobar by interested people. The files could be analyzed and was basically all that was needed.

His reluctance and avoiding the possible issues is what caused the harsh wording here and there as well as not loaning out and not wanting to part with it but gladly sold it and have someone else pay for donations to a goal he finds important (and not only he).

Why would that not generate some harsh words or mistrust (valid or not) ?

 

[RandomEar]

[...]
And to finish up, later still in this thread I got this link: https://www.audiosciencereview.com/forum/index.php?threads/“human-ears-are-better-than-measurements-”-“humans-can-hear-things-we-can’t-even-measure-”.72288/post-2651009
These are not strictly on coloration audibility but rather focus on JNDs, but I'll accept them for now considering the scarcity of direct evidence, and for the sake of my point on typical numbers of participants in audio studies:
Yost & Nielsen 1985 - can't find it, unkown number of participants
Jestead 1977 - three participants
Johnson 1993 - paywalled
Norcross et al. 2003 - paywalled, but seems to have had fifteen participants for at least one of the tests, based on the reposted figure (mis-cited in discussions so far as "Scott et al."! CC: @RandomEar )
[...]

Ha! You're correct, I mixed up the name. However, you did, too! The correct first author to cite would be G. A. Soulodre. Luckily, the study was linked so everybody still knew which DOI we're talking about and could read it themselves. I also don't see how any of the articles would be paywalled: On the AES site there is a regular, working download button for me.

 

[jabbr]

My point on ABX was more that there has been (less so now, I guess?) sustained and robust ABX testing between different lossless codecs over time. Hydrogenaudio was doing a lot of those for a while, not sure if they still are. These codecs work on the principle of thresholds of hearing, so the process of developing and improving MP3, OGG, AAC etc. implicitly generated a lot of data on thresholds of hearing. And we know these codecs work...

Yes for sure. Interaction between analog output and input stages should be well understood, and would be if we had the same format specifications as we do with codecs ... we could simply simulate them including speakers etc... that software is well developed. The problem is that we are dealing with proprietary black boxes. The reason digital works so deterministically is that it is well behaved. Analog not so much.

 

[pma]

With normal competent design seen in nowadays DACs and preamps such interactions are no issue and non-existent. They are rather seen in exotic designs (tube output stage). You guys sometimes care too much about unimportant things.

 

[pablolie]

Simply regarding whether he was trolling this site. He was responding to the discussion here. He didn't start it.

Clearly someone re-posted here to stir up things, look at the title (and note Josh never claimed the test constituted any "proof" of anything, paradoxically). Your views on Josh's authenticity are to be respected... but so are others'. Not worth keeping the "who was worse?" theme going. We all know we can and will eventually get attacked when posting in forums, and non-moderators trying to moderate are always going to fail.

 

[donjoe0]

In every case the theoretical limit of audibility is also beyond the practical one when listening to music.

You’re missing the part that this goes for DAC’s that measure transparent.
[...]
The graphs that Amir provides have enough resolution

Nope, we don't have anywhere near "every case" covered. Even your graph is an example of what I was talking about: the response not only crosses the 0.1 dB limit within the audio band, but the graph is not even set up to clearly show 0.1 dB resolution, you still have to eyeball it.
As far as I've read measurement-centric reviews, some double-digit percentage of them, up to maybe half of the historical total, either do not zoom in to 0.1 dB on the FR graph for DACs, or the DAC clearly doesn't satisfy the 0.1 limits (usually there's a roll-off), orr some reviews don't include an FR graph at all. If this has changed for the better, it has to be a recent development, I admit I've been following reviews a lot less after getting enough satisfying gear to be mostly out of the market.
Here's a few just from recently active threads:
- no FR - https://www.audiosciencereview.com/forum/index.php?threads/smsl-do100-review-stereo-dac.34198/
- roll-off - https://www.audiosciencereview.com/forum/index.php?threads/topping-d900-dac-review.70067/
- no FR - https://www.audiosciencereview.com/...ds/topping-d90-balanced-usb-dac-review.10519/
- roll-off - https://www.audiosciencereview.com/forum/index.php?threads/smsl-su-1-stereo-dac-review.44029/
and I could keep going. There's a ton of them that do, not, prove, transparency.

No one is called a moron on this forum.

I'm sure it's happened. But you're misquoting me anyway by chopping up a convenient piece you find easy to attack. In reality I said "replies to the effect of", not the exact words "moron" etc. It's usually a tone and attitude thing, not made explicit, so they can get it past moderation.

Instead of testing thousands of random people, why don’t the people that claim they can hear the differences back up their claims

Sure, I would've liked to see that as well. But that's a way to get new information beyond existing studies. My sideshow in this thread has been mostly about what studies already exist and what claims do they allow us to make / what new claims do they allow us to classify as "extraordinary" (thus requiring extraordinary evidence). People seemed to get annoyed at that and repeatedly misinterpreted me as saying I'm demanding someone spend a lot of money and effort to develop new studies especially for me , whereas I was only talking about what should be acceptable discourse and terminology on human audibility, given the existing studies.

There are many ways to approach an ABX. The repetition of a small segment for me is the last step when I'm already quite sure what the exact differences are and what I have to listen for, trying to find a small snippet that highlights it best. But initially, I find it easier to have a sort of constant flow like I would have in normal listening. The reason is simple, once testing starts to feel annoying it is basically game over. It all depends on the artifacts responsible for the difference.

This. But I would take it even further, and say the reason is a bit complex: longer listening exposes you to more combinations of frequencies put through the system and builds up a better impression of its FR than any few-second sample could on its own. That is precisely what is needed for evaluating the subtlest differences, like those between imperfect DACs. The few-second ABX should not be considered the gold standard in this case (unless broadband noise is used, which has support in the research as being more revealing than music, though I can't say I've seen it compared specifically to several sessions of tens of minutes each, of a wide variety of music). And I thought I was going out on a limb again with believing this, but then I found this in Toole & Olive 1988 today while digging through for citations:

It is rewarding to note that the residual frequency-response fluctuations in the most preferred loudspeakers in that reference [Loudspeaker Measurements and Their Relationship to Listener Preferences, 1986, fig. 8] were more closely comparable in scale with those seen here in Fig. 27 [excellent speaker+very low artificially added detrimental resonances], rather than those in Fig.26 [excellent speaker+high artificially added detrimental resonances]. Apparently, integrated over the duration and musical variety of a series of those listening tests (typically four to six 30-min sessions), listeners were able to detect and to establish clear preferences based on very small changes in timbre caused by resonances.

I wouldn't be shocked to find out from a study explicitly comparing resonance detection through long-duration music listening vs. short-duration broadband noise that similar performances can be reached with music. Hell, I could even see the former being ultimately better than noise, because with meaningful material we can also use pattern recognition brain circuits that evaluate e.g. staging and imaging as ways for subtle FR differences to manifest, which you can't do with random noise.

 

[kemmler3D]

Nope, we don't have anywhere near "every case" covered.

An experiment to determine the limit of audibility would necessarily isolate the audible phenomenon in order to find the true minimum / maximum. So, test tones and musical passages selected to maximize audibility of a given type of distortion, etc.

Practical use means listening to music in a normal way. The threshold of audibility means just barely audible in the optimal case. Whatever that is, it will be (more or less by definition) less audible while music is playing.

I guess a counterexample is not impossible but I can't think of one off the top of my head.

 

[amirm]

Given all that he was treated harshly.

The only people objectively treated harshly are the few of us who had fork over a lot of money to buy his bloody ABX box instead of spending the same on other needs for our community.

Josh on the other hand, got paid for article/testing by his boss at AS and we made said boss oh so happy by sending a ton of traffic to them. He unloaded the ABX box on us so no longer has any risk of owning a comparator that potentially can't be used to make comparisons. And forced us to donate to his charity. Whatever "emotional damage" you or think he has suffered, will pass.

Demands for recordings from him were made, yet he claims not to have a good way to record. Given the harsh criticisms of his methods and materials I can see why he declined to make recordings.

You shouldn't if you care at least a little bit about figuring out what is going on with his testing.

Listen, we are investigating a case here. We can never be in his head and determine what he heard. Instead, we have to look to other evidence. He has said he has recorded a bunch of stuff that he then evaluated using different tools. Get him to share those files with us. There are people here who are great in examining such data and determine what they mean. Maybe they confirm what he has said about those captures. Maybe not. Either outcome is useful to get the truth about audibility of DACs.

So get the darn files. Don't be his PR person. We know the situation. We don't need a volunteer translator. You are there posting as well as here. Ask him to put the files up on a share and give us a link. Should take all of 10 minutes.

 

[PristineSound]

I have looked at buying the Van Alstine comparator and even contacted the company a few years ago. Alas, they never responded to me.

I've contacted AVA long time ago about the ABX the ABX comparator.

This is what was explained to me:

The design of the ABX box is not owned by Audio by Van Alstine. We had a marketing agreement with the designer/owner where he provided us the finished product and we sold it under the Audio by Van Alstine umbrella of products. When essential parts became unavailable he decided to stop producing it and has informed us that he has no plans on bringing the unit back into production.

 

[jabbr]

You shouldn't if you care at least a little bit about figuring out what is going on with his testing.

All I care about is sorting it out between possibilities:
1) the fact that his 'numbers' are so strong indicates to me that there is an unquestionable measurable correlate that allowed him to distinguish (I don't believe he made this up)
2) there is either a systemic issue with the testing setup e.g. an issue with the box
3) the circumstances of the testing brought out distortions/changes in the audio signal that are readily audible

I have given several hypotheses why this might occur, but these are just examples. I don't claim to know. Nothing in his testing tells me what the cause is. I have an open mind.

Listen, we are investigating a case here. We can never be in his head and determine what he heard. Instead, we have to look to other evidence. He has said he has recorded a bunch of stuff that he then evaluated using different tools. Get him to share those files with us. There are people here who are great in examining such data and determine what they mean. Maybe they confirm what he has said about those captures. Maybe not. Either outcome is useful to get the truth about audibility of DACs.

So get the darn files. Don't be his PR person. We know the situation. We don't need a volunteer translator. You are there posting as well as here. Ask him to put the files up on a share and give us a link. Should take all of 10 minutes.

I'm not yet convinced that recordings don't mask differences e.g. if they are all made with 100k input impedance devices. But I could be convinced if recordings were properly compared to the DAC itself, so its chicken and egg issue at the moment. I've said this before so its not a view specific to this argument. I'm not anyones PR person nor am I a volunteer translator, nor do I have any commercial relationships, nor is anyone paying me to give my opinions. I'm usually browsing and reading while software compiles etc. If you wish to have a discussion about the sociology of different sites and groups that is a different discussion but really outside of anything audio specific. I am always happy to give you my own impression of these aspects but at this point, you have decided to obtain the ABX comparator and as I've said, you will find that it either has a fault, and if so the efforts here to create a better ABX comparator will be necessary.

What I do recommend, is that instead of narrowing the specifications for adjunct equipment, best to try to stress the system i.e. with low input impedance devices etc so that differences are brought out, rather than masked.

 

[RickS]

Thread locked for maintenance per my previous post…

Maintenance complete and thread unlocked. Please remember to make productive posts and avoid personal comments!

 

[levimax]

I believe in applying the principle of charity as much as possible in terms of people’s intentions, for many good reasons.

^ This.
Asserting that you can see into someone's heart and mind to determine their intentions and motivations is unscientific at best.

 

[amirm]

If you are speaking about me, no looking into hearts and mind was necessary. Objective actions led the plot all the way to the end. We are sitting here still with no files. We have spent thousands just to get access to a piece of this puzzle (comparator). We are going to invest a ton more time and energy because we have a reluctant witness. We could have sorted this in the first few days of disclosure.

 

[amirm]

All I care about is sorting it out between possibilities:
1) the fact that his 'numbers' are so strong indicates to me that there is an unquestionable measurable correlate that allowed him to distinguish (I don't believe he made this up)
2) there is either a systemic issue with the testing setup e.g. an issue with the box
3) the circumstances of the testing brought out distortions/changes in the audio signal that are readily audible

Every one of these hypothesis could have been tested if Josh was willing to participate. He has everything at his disposal: the gear, the setup, his ears, etc. And we were/are standing here in full force to help.

The files that members asked for, will also help answer some of these questions. So I implore you again to ask him to send those bits over.

We heard way early in the thread that Josh didn't want to loan the box because he was planning to do a lot more testing with it. Well, that effort could have been focused to see if we could get this chapter right before writing the rest of the book.

I'm not yet convinced that recordings don't mask differences e.g. if they are all made with 100k input impedance devices.

Hard to imagine an analog comparator doesn't do that but an ADC does. Again, let's get the files and find the answer. This is immaterial in his case anyway since his Cosmos ADC has much lower input impedance.

 

[pma]

I've contacted AVA long time ago about the ABX the ABX comparator.

This is what was explained to me:

I know such situation very well. In my case it was a prototype designed and built and then it was produced and sold under “umbrella” of another company. They also made some modifications to reduce production cost. This is probably quite usual procedure, as I had similar offers later.