"The DAC Scam" and follow-up video by YouTube channel "AP Mastering"

[RandomEar]

The Channel "AP Mastering" has done a slightly unusual blind test of a DAC last year. In essence, he claims that all well designed DACs are audibly transparent and transparent devices can't be differentiated by listening. He then proceeds to record some music via his studio DAC using his ADC and carefully cuts together this looped recording with the original digital file of the music. The resulting audio with multiple switches from looped to orignal and back is played in the YouTube video and he challenges his viewers to tell him how often he switched back and forth via a Google form.

In a follow-up video, he evaluated the Google forms result and the comments below his last video. He even published a paper about this evaluation

Unsurprisingly, statistics show that absolutely no one could reliably tell the number of switches. In stark contrast, slightly over half of the comments of his first video claim to be able to hear such differences.

I think this is a fun test and definitely good to get some clicks/engagement, which is essential to perform well on YouTube. I also see some methodical problems, which in my opinion mean that his results do not actually prove his claims:

• The DAC output and the original have gone through YouTube compression
• The DAC output and the original are rendered by the listeners own DAC
• The switching isn't A/B/X, but "forced" A/B, making it more difficult for the listener
• The music keeps playing through the switches, making comparisons more difficult as you can't A/B the same section of the track repeatedly

I'm sure there are a couple of other problems. Overall, it's a fun exercise and it definitely suggests that even the DAC/ADC-loop is good enough that you will not detect clear differences. But the test setup also makes it more difficult for the listener than a self-controlled A/B/X setup where you can repeatedly play the exact same time stamp in the file with instant switching. I do think it's understandable that AP Mastering did not initially provide the WAV file of the test track, as that would allow listeners to analyze it using external software and thereby cheat in the challenge.

 

[amirm]

He then proceeds to record some music via his studio DAC using his ADC and carefully cuts together this looped recording with the original digital file of the music.

Such practice will mask audible differences. In a proper AB test, you are in control and know you are playing this sample or the other. You can then select sections that may have audible differences vs others. With the scheme he is using, it removes all of these tools, violating standard practice for such testing. It is usually designed to get more negative outcome, demonstrating the person's lack of knowledge, or intent to cover differences.

Our job in performing such tests is to do everything in our power to find audible differences. After all, if there is lack of transparency, we want to find and fix it. From bible of testing for small impairments, ITU-R BS1116 Recommendation:

"In the preferred and most sensitive form of this method, one subject at a time is involved and the selection of one of three stimuli (“A”, “B”, “C”) is at the discretion of this subject. The known reference is always available as stimulus “A”. The hidden reference and the object are simultaneously available but are “randomly” assigned to “B” and “C”, depending on the trial. "

The testing he is doing is only good for gross differences such frequency response variations. Not for small non-linear impairments.

Further, content needs to be selected to be sensitive to such differences. On this ground, you would put such a control in there as to see whether the test itself is working, or listeners are skilled enough to find these differences. Again, from BS1116:

"3.2.2 Post-screening of subjects
Post-screening methods can be roughly separated into at least two classes; one is based on inconsistencies compared with the mean result and another relies on the ability of the subject to make correct identifications. The first class is never justifiable. Whenever a subjective listening test is performed with the test method recommended here, the required information for the second class of post-screening is automatically available. A suggested statistical method for doing this is described in Attachment 1.

The methods are primarily used to eliminate subjects who cannot make the appropriate discriminations. The application of a post-screening method may clarify the tendencies in a test result. However, bearing in mind the variability of subjects’ sensitivities to different artefacts, caution should be exercised. "

It is trivial to create tests where clearly audible differences in systems are not found due to arbitrary subject selection.

Net, net. Do as I did. Don't watch the video. It is clickbait stuff with no value.

 

[amirm]

On this mention in his paper:

"Although the original digital recording reference cannot be heard directly without a DAC, a well established, straightforward, yet effective test often used to assess transparency of converters is the loopback test. This test involves connecting the output of a DAC to the input of an analog-to digital converter (ADC) and recording the resulting signal. The process of passing audio through this loop once constitutes a single generation of the test. This procedure can be repeated for an arbitrary number of additional generations, allowing for a cumulative assessment of signal degradation. With modern, high-quality DACs and ADCs, little to no perceptible difference should be present between the original signal and the first-generation loopback recording. Likewise, the first-generation loopback should exhibit no meaningful deviation when compared to the second-generation recording. However, as the number of generations increases, signal degradation accumulates. By the time the original is compared to the 100thgeneration, audible differences tend to become sufficiently pronounced that even an untrained listener can detect them"

As it happens, I recently repost me passing such a test even with one generation,. The key was first to try to pass the high number of generations first. Then, you realize what the difference is. With that knowledge, I was able to go to fewer and fewer loops until I achieved one generation:

----
-------
Here is an example test you can take to show us you do have good hearing acuity. https://ethanwiner.com/loop-back.htm

It is a piece of music that has gone through a DAC, then ADC, then back to DAC and so on. And on really bad DAC/ADC as audiophile standard go: a $25 Soundblaster X-Fi.

This is me finding the difference double blind with just one pass through DAC/ADC:
foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:40:07

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass1.wav

06:40:07 : Test started.
06:41:03 : 01/01 50.0%
06:41:16 : 02/02 25.0%
06:41:24 : 03/03 12.5%
06:41:33 : 04/04 6.3%
06:41:53 : 05/05 3.1%
06:42:02 : 06/06 1.6%
06:42:22 : 07/07 0.8%
06:42:34 : 08/08 0.4%
06:42:43 : 09/09 0.2%
06:42:56 : 10/10 0.1%
06:43:08 : 11/11 0.0%
06:43:16 : Test finished.

----------
Total: 11/11 (0.0%)

And of course with 20 loops:

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 05:38:16

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass20.wav

05:38:16 : Test started.
05:39:05 : 00/01 100.0%
05:39:27 : 00/02 100.0%
05:39:44 : 01/03 87.5%
05:40:01 : 02/04 68.8%
05:40:18 : 02/05 81.3%
05:40:30 : 03/06 65.6%
05:40:58 : 04/07 50.0%
05:41:09 : 05/08 36.3%
05:41:19 : 06/09 25.4%
05:41:28 : 07/10 17.2%
05:41:38 : 08/11 11.3%
05:41:53 : 09/12 7.3%
05:42:02 : 10/13 4.6%
05:42:18 : 11/14 2.9%
05:42:29 : 12/15 1.8%
05:42:42 : 13/16 1.1%
05:42:53 : 14/17 0.6%
05:43:03 : 15/18 0.4%
05:43:16 : 16/19 0.2%
05:43:27 : 17/20 0.1%
05:43:40 : 18/21 0.1%
05:43:53 : 19/22 0.0%
05:43:58 : Test finished.

----------
Total: 19/22 (0.0%)

As you see, 0% of guessing.

---------

The way I passed it was to focus on high frequency roll off that both ADC and DAC have. Unless this is perfect, which it rarely is, every time to loop through, you wind up changing the frequency response. This was easy to detect, for me anyways , with pass 20. Passing 1 loopback was harder but as you see above, was certainly doable.

This also brings in what I always say: you have to have deep knowledge of the system to know how to pass such a test. Or how to construct it. If you don't know what DAC and ADC filters are, then you can't use the method I used to find differences. You would be focusing on parts of the music where such a difference may not exist.

 

[RexrothPigeon]

definitely good to get some clicks/engagement, which is essential to perform well on YouTube

this is a concise articulation of why "successful youtuber" is about as appealing as a successful STI

 

[Svperstar]

this is a concise articulation of why "successful youtuber" is about as appealing as a successful STI

I've been watching his channel a while and I like how he does his best to dispel bad info "everyone knows" about audio and inform his audience about the pro audio seen. Not everyone is like us and looks up data, many of his viewers probably think their Air Pods are high end audio reproduction devices and think spending more then $100 tops on an amp is insane.

 

[amirm]

I've been watching his channel a while and I like how he does his best to dispel bad info "everyone knows" about audio and inform his audience about the pro audio seen. Not everyone is like us and looks up data, many of his viewers probably think their Air Pods are high end audio reproduction devices and think spending more then $100 tops on an amp is insane.

$100 for an amp is insane??? The only people who might think that are non audiophiles. Not anyone who would watch such videos.

 

[kemmler3D]

$100 for an amp is insane??? The only people who might think that are non audiophiles. Not anyone who would watch such videos.

There are a lot of people who claim to care about sound quality (the most basic definition of audiophile) who also think $100 is a lot to spend on speakers, headphones, or any associated equipment. I should know, I used to try to convince them to spend that money.

I agree with your opinion here but the general public doesn't always catch on, some people will seek audiophile gear / sound for hamburger money and be bewildered.