DAC listening test files - update

[RandomEar]

Hi,
you may digitally compare both files provided by pma and hear and see the diff (in audacity e.g.):
sox -m -v 1 file_1.wav -v -1 file_2.wav file_diff.wav

sox -m -v 1 DAC1.wav -v -1 DAC1.wav DAC1_1-diff.wav

If the compared files would be equal, the diff would be deep silence (you can test it easily by comparing same files).

The DAC-files differ extremely compared to the file_x - files above.

What does the sox command do here? It mixes two files together - but the second file is inverted. If the second file is identical, it completely cancels out each sample of the first file and deep silence is the result. If there are any differences, the samples can't cancel each other out any more and you get an audible diff.

You now may argue if this diff is audible or not if you compare each of the test files serially with your ears.

While this is true and easily tested, the method has one major flaw: Time differences and clock drift can lead to a very bad nulling result, even if the actual files were 100% identical otherwise. I assume @pma did time-align the files to within one sample. I don't know if he corrected clock drift for all samples provided here. But as far as I understood his posts, there was at least no sub-sample alignment. In consequence, the nulling result won't be perfect (a 1/2 sample error is expected on average, increasing over the length of the recording) and can be misleading about the actual differences between the DACs.

DeltaWave can and does correct this and allows you to listen to the corrected diff. That's a much better way to judge the nulling result.

 

[oleg87]

Hi,
you may digitally compare both files provided by pma and hear and see the diff (in audacity e.g.):
sox -m -v 1 file_1.wav -v -1 file_2.wav file_diff.wav

sox -m -v 1 DAC1.wav -v -1 DAC1.wav DAC1_1-diff.wav

If the compared files would be equal, the diff would be deep silence (you can test it easily by comparing same files).

The DAC-files differ extremely compared to the file_x - files above.

What does the sox command do here? It mixes two files together - but the second file is inverted. If the second file is identical, it completely cancels out each sample of the first file and deep silence is the result. If there are any differences, the samples can't cancel each other out any more and you get an audible diff.

You now may argue if this diff is audible or not if you compare each of the test files serially with your ears.

Such comparisons are generally of limited use, as they are extremely sensitive to small differences in phase we can't really hear.

 

[danadam]

I would treat danadam's action as a proof to inform others about the possibility to cheat, because you specifically mentioned about it:

the test interface avoids cheating.

True. I should've been more upfront about it, sorry.

 

[NTTY]

I have closed that online test as there was a confessed cheating here. I am glad of the confession (the courage to confess), however I cannot let the test continue regarding collecting further results. The files are still working and the test files can be listened to, just without a possibility to get results.

Before closing, the test was played 70 times and the results were stored 12 times. From those 12 results, I have discarded the cheated 10/10 result and also the worst 3/10 result. 10 results remaining give this:

View attachment 551349

The 8/10 result that would start considerations on a successful audible discernment has not been reached.

View attachment 551351

Thanks Pavel,

This is very interesting. Too bad you closed the site, I wanted a friend to give it a try.
But I see you shared the files, perfect, thank you!

Cheers
Flo

 

[pma]

Hi Flo,

Yes the files are available to anyone, in this post. Online files work as well, but without evaluation option. I cannot admit false, cheated results to pollute results. I am sorry to anyone, however we need a kind of professional attitude.

 

[pma]

True. I should've been more upfront about it, sorry.

No problem. At least it shows that we all need to be careful when dealing with results of web based tests. Not to mention single-person, no-data available tests, though (maybe) performed in good faith. That would separate professional and hobbyist approach, which it is all about.

 

[NTTY]

No problem. At least it shows that we all need to be careful when dealing with results of web based tests. Not to mention single-person, no-data available tests, though (maybe) performed in good faith. That would separate professional and hobbyist approach, which it is all about.

Indeed. And it is a good learning, meaning that Foobar ABX is subject to people cheating it too, hence voiding any attempts at using it to request a "can you differentiate them" type of test.
Does it mean we are left with the only option to ask "which one do you prefer"?

 

[pma]

Does it mean we are left with the only option to ask "which one do you prefer"?

I would say no. The number and frequency of cheating of foo abx tests or online tests is quite, or very low. We still get some valuable feedback. We only need to be careful. Some say "trust your ears", I am saying "trust your real results and experience". The 10/10 result in the online test was so suspicious to me that I asked if it came from our members. @danadam was hopefully so honest to confess it was his cheated result. I appreciate it, I appreciate people telling truth.

 

[amirm]

Indeed. And it is a good learning, meaning that Foobar ABX is subject to people cheating it too, hence voiding any attempts at using it to request a "can you differentiate them" type of test.

Someone has figured out how to deal with the cryptographic hash signature?

 

[pkane]

Someone has figured out how to deal with the cryptographic hash signature?

Much easier than that. You just need to monitor the output of Foobar with some measurement software (REW, Multitone, etc.) Even a small difference in the filter frequency response can be viewed in real time so that when X is playing, you can tell which filter it is, A or B. Zoom in RTA near the stop band of the filter, and you'll be able to tell the difference visually in no time to make your selection in Foobar. You can even turn off the sound completely and still get a perfect score

 

[amirm]

Ah ok. I was accused of using AP that way!!!

 

[pma]

Someone has figured out how to deal with the cryptographic hash signature?

He used pulseaudio for captures during replay. Described here.

 

[NTTY]

Someone has figured out how to deal with the cryptographic hash signature?

Theoretically, you use the foobar2000 ABX Log Signature Tool that reads the hash to confirm if a pasted test log is authentic, in case of sharing our results as we did. You need to paste all the log of Foobar ABX. It will check you did not alter the results manually.

But like @pkane said, it’s too easy to monitor the output via a real time analyzer to win as we want.

 

[pma]

By now we were talking about tinniest differences between DACs "sound". But, will you hear a difference if we test such poor component like this one is?

Test - Topping D10s/Cosmos ADC loop vs. Tube preamp (6J1) in the loop

link to the test files:

6J1_test.zip​

drive.google.com

Please feel free to try the test and tell me if you hear the difference, and do not forget to post foobar abx test report.
I have also included a delta file (Deltawave difference) for you to hear how horrible is the "sound" of tube distortion in isolation. But, can you hear it when playing music?

I have also created an on-line version of the test:

6J1 tube preamp vs. Topping DAC/Cosmos ADC loop · FunkyABX​

abx.funkybits.fr

 

[DavidEdwinAston]

By now we were talking about tinniest differences between DACs "sound". But, will you hear a difference if we test such poor component like this one is?

Test - Topping D10s/Cosmos ADC loop vs. Tube preamp (6J1) in the loop

link to the test files:

6J1_test.zip​

drive.google.com

Please feel free to try the test and tell me if you hear the difference, and do not forget to post foobar abx test report.
I have also included a delta file (Deltawave difference) for you to hear how horrible is the "sound" of tube distortion in isolation. But, can you hear it when playing music?

I have also created an on-line version of the test:

6J1 tube preamp vs. Topping DAC/Cosmos ADC loop · FunkyABX​

abx.funkybits.fr

Have just listened to the lovely piece of music! Couldn't discern any difference.

 

[pma]

Have just listened to the lovely piece of music!

Thank you! Usually I get complaints .

 

[Holy Spirit]

After comparing 6J1 files for a few minutes in Foobar ABX, I do not think I can tell any difference... Too bad that a few weeks ago I ordered transducers for a speaker project. Clearly a waste of money in my case. Now I wonder if I can hear 128kbps mp3 vs wav.

 

[cgidesign]

Many thanks for the test files.
I tried the 6J1_test.zip
If I understand the description of the test, it is a comparison of a:

Topping D10s (109 €) -> E1DA Cosmos ADC -> wav file
vs.
FE-6J1-2 Tube preamplifier DIY kit (18 $) -> E1DA Cosmos ADC -> wav file

Do I hear a difference? I don't think so, but just did a few A/B switching tests by using both files in a DAW (final output was REM Babyface to headphones).

Then I put both files in a spectral analyzer -> made screenshots of both waterfalls -> created a difference file:

Waterfall of file_x.wav

Waterfall of file_y.wav

Spectral difference file_x and file_y - the main difference between the two files is in the area below 30 Hz. EDIT: The signal strength in the region with a noticeable difference is around max. -52 dBFS.

I missed to capture the scale in the waterfalls, so here it is as a separate screenshot (slightly different height, so not pixel perfect):

 

[somebodyelse]

Now I wonder if I can hear 128kbps mp3 vs wav.

You'll have to be specific about which encoder version and encoding options were used. Unlike decoding, the encoding process isn't standard, and different encoder choices can make a significant difference to the decoded output for the same bit rate.

 

[pma]

After comparing 6J1 files for a few minutes in Foobar ABX, I do not think I can tell any difference... Too bad that a few weeks ago I ordered transducers for a speaker project. Clearly a waste of money in my case. Now I wonder if I can hear 128kbps mp3 vs wav.

Depending on music sample, you can usually hear a difference. I may prepare a test, with both samples stored as wav files (mp3 re-converted to wav). However, such test is prone to cheating, easily, because the mp3 128kbps file immediately shows cut of high frequencies if you monitor the soundcard output. But just for curiosity, why not to make it. I may have time for it in a week from now.