ABX complex non-linear distortion test, about 5% on a music sample

[bobbooo]

If the tracks are played online through a browser we have no idea if they were subject to sample rate conversion (common on Android), and on PC are likely played in Widows shared audio mode by most users. There's also no verifiable log with signature. The Foobar ABX Comparator is simple enough for anyone to do, so less controlled online tests just muddy the waters really.

 

[pfzar]

So glad that I offered an objective view to hosting the test.

 

[bigjacko]

It may be googled from probability theory. Below is the chart for X successful trials from 10 trials.

View attachment 98204

But if I am purely guessing, getting 0 out of 10 and 10 out of 10 would be the same very low possibility. Getting 5 out of 10 should be most likely I am guessing. This chart does not look like it can go both ways, only one way which is getting it right, wonder if it is correct.

 

[MarsianC#]

Law of large numbers. Flip a coin 10^6 times and it will be more are less (very very close) 50/50, as one would expect. But it won't if you do it only 10 times. Hence the need for tests to make sure your result is likely to be correct or not. Bu I'm getting things mixed up. Look into it if you're really interested, it's no crazy math. Actually it's super boring math.
Never liked statistics and probability...

 

[maty]

Come on Pavel, it is a very easy test. With only 5", JRiver MC. The difference is very evident!!! Who does not immediately notice the difference is that it has a large bottleneck in its audio system.

I want something like this: H2 at -70dB vs H2 at -80dB vs H2 at -90dB.

You know, I want H2 at -73dB as Nelson Pass (jazz, small group, instrumental and natural voice). Or something H2 at -83 dB to listen to ALL kind of very good recordings.

 

[pma]

If you would like to take the test here ABX test. It's available until 12-31-20, It's completely anonymous. I will post the data after the test is concluded.

No problem, thank you!

But, frankly, I prefer Foobar ABX, for the reason that one can make preliminary training and make a choice of the time interval where the difference is the biggest. In case of very similar samples, like here, it would be difficult to tell the difference by continuous streaming. Depends what we expect, to hear small differences or to hear big, substantial differences.

 

[pfzar]

Just realized it's only doing 2 trials. Editing a new test that will have more,updated link

 

[pma]

Come on Pavel, it is a very easy test. With only 5", JRiver MC. The difference is very evident!!! Who does not immediately notice the difference is that it has a large bottleneck in its audio system.

I want something like this: H2 at -70dB vs H2 at -80dB vs H2 at -90dB.

You know, I want H2 at -73dB as Nelson Pass (jazz, small group, instrumental and natural voice). Or something H2 at -83 dB to listen to ALL kind of very good recordings.

You may say that, but you are not posting any ABX log signature, so unfortunately I cannot take your opinion into account. I also do not take the complaints that Foobar "does not sound good". I made numerous tests that confirmed that Foobar sends bit perfect 16 and 24 bit files. Data were sent through optical SPDIF, received, and the two files compared by subtraction and the result was 00000.................00000. Nothing. Nada. Of course you need properly installed drivers and no OS resampling and "enhancements" to get the proper result.

Regarding the music for the test, I beg to differ. I do not use (jazz, small group, instrumental and natural voice) for testing purposes. But please feel free to make your own test with the music that you like and at distortion levels that you have mentioned. There is a SW by @pkane that will allow you to make so (but it was not used in my test). However, unless there is not the ABX result, I cannot take it serious, I apologize.

 

[escape2]

Since I can't hear any difference between the two files in the OP, I guess there is no point in me even attempting the ABX test.

 

[pma]

Since I can't hear any difference between the two files in the OP, I guess there is no point in me even attempting the ABX test.

Try the time interval selection between 16 s and 24.5 s, or 19.5 s and 24.5 s. Audible distortion is highest there.

 

[Blumlein 88]

Since I can't hear any difference between the two files in the OP, I guess there is no point in me even attempting the ABX test.

Maybe, but not necessarily. I've found on a few signals I will think I'm hearing nothing different yet somehow pick correctly almost all the time. Not just for one set of trials, but repeatedly. So do a quick run of 10 or 15.

BTW, my preference would have been 20 trials minimum. I stopped at 10 because the OP did, and I thought he'd want consistency of results. My usual method of doing 20 is to do 10, rest for a few minutes and do 10 more.

 

[Blumlein 88]

Try the time interval selection between 16 s and 24.5 s, or 19.5 s and 24.5 s. Audible distortion is highest there.

I did listen for a few minutes to pick where to test. I keyed in around those places you just listed, but ended up going with 11.9 to 13.9 seconds if I remember rightly.

 

[pma]

Maybe, but not necessarily. I've found on a few signals I will think I'm hearing nothing different yet somehow pick correctly almost all the time. Not just for one set of trials, but repeatedly. So do a quick run of 10 or 15.

BTW, my preference would have been 20 trials minimum. I stopped at 10 because the OP did, and I thought he'd want consistency of results. My usual method of doing 20 is to do 10, rest for a few minutes and do 10 more.

In fact I did 20 trials as well, both 8/10 and posted, however there was a 1 day rest . Seriously, 20 trials in a row, in case of small sound differences like here, is too much, the concentration is fading.

 

[maty]

Well, I will not participate more. If it had been more difficult I would have done several tests, with loudspeakers, headphones, foobar2000 (with and without ABX plugin), JRMC ... but the difference is so noticeable that it is not worth trying. With DISTORT I was complicating the tests but the initial one was not as easily detectable as yours.

 

[LTig]

If you would like to take the test below. It's available until 12-31-20, It's completely anonymous. I will post the data after the test is concluded.

updated LINK

Sorry, can't hear any difference between them all... (too old, hearing too bad ). Well, good to know that I can spare a lot of money, and the main reason to get the RME ADI-2 PRO fs was measuring.

 

[CaféNoir]

Hi Pavel, nice to see you come back with some tests!
Ok here is my first attempt. I had to guess during the test that A stays A, and B stays B throughout, only X and Y changes, which did not help. I will try to install wasapi or ASIO next.

foo_abx 2.0.6d report
foobar2000 v1.6.2
2020-12-09 21:55:09

File A: Quintet1.flac
SHA1: 78941da5875a6dc51169e29bbbb909399b75c377
File B: Quintet2.flac
SHA1: 2cea48c9ec2b493807e65e1d2b49deb19b543939

Output:
Default : Primary Sound Driver
Crossfading: NO

21:55:09 : Test started.
22:01:50 : 01/01
22:02:41 : 02/02
22:08:40 : 03/03
22:09:06 : 04/04
22:11:10 : 04/05
22:12:12 : 05/06
22:13:07 : 06/07
22:13:44 : 06/08
22:14:10 : 06/09
22:14:33 : 07/10
22:14:33 : Test finished.

----------
Total: 7/10
p-value: 0.1719 (17.19%)

-- signature --
6a05d8090e3d3a2108473701283f1b6b9b189839

Music feels quite "distorted" or inharmonic in itself, which makes it a rather tricky and fatiguing test.
It is very strange that the difference is quite immediate at first, but ABX seems to make it abnormally difficult maybe due to repetition, listening fatigue, or other psychologic reasons. Like eating the same meal over and over, you lose the taste.

Can we talk about what we think is the distorted sample, and perceived subjective differences, or not yet?

 

[dougi]

Here is mine just using ATH-ES7 headphones off the laptop into my 52 YO, 10kHz limited, tinnitus riddles ears.

Worse than guessing!

foo_abx 2.0.6d report
foobar2000 v1.5.5
2020-12-10 17:35:38

File A: Quintet1.flac
SHA1: 78941da5875a6dc51169e29bbbb909399b75c377
File B: Quintet2.flac
SHA1: 2cea48c9ec2b493807e65e1d2b49deb19b543939

Output:
DS : Primary Sound Driver
Crossfading: NO

17:35:38 : Test started.
17:37:37 : 00/01
17:38:00 : 01/02
17:38:14 : 01/03
17:39:02 : 02/04
17:39:59 : 02/05
17:40:43 : 02/06
17:42:04 : 03/07
17:43:00 : 04/08
17:43:37 : 04/09
17:44:27 : 04/10
17:45:33 : 04/11
17:45:55 : 04/12
17:46:42 : 05/13
17:47:10 : 06/14
17:48:19 : 06/15
17:48:59 : 07/16
17:48:59 : Test finished.

----------
Total: 7/16
p-value: 0.7728 (77.28%)

-- signature --
f1c6d1042cd1fd69e2f2c7336dc95666bbcf7ede

 

[pma]

Hi Pavel, nice to see you come back with some tests!
Ok here is my first attempt. I had to guess during the test that A stays A, and B stays B throughout, only X and Y changes, which did not help. I will try to install wasapi or ASIO next.

Music feels quite "distorted" or inharmonic in itself, which makes it a rather tricky and fatiguing test.
It is very strange that the difference is quite immediate at first, but ABX seems to make it abnormally difficult maybe due to repetition, listening fatigue, or other psychologic reasons. Like eating the same meal over and over, you lose the taste.

Can we talk about what we think is the distorted sample, and perceived subjective differences, or not yet?

Thank you for your kind words. I understand the situation that you feel you can hear the clear difference by listening the files and then the ABX has a negative result. This is based on the fact that we use ear/brains processor for the listening and not some technical measuring instrument. Both ear (biological/neurological sensor) and brains (mind driven processor) have great influence to what we hear. It is difficult to get a non-biased result without a double blind test, and it is also difficult to get adapted to double blind test conditions.


So please feel free to describe your feelings but please take into account that I consider the ABX test result.


Regarding the “distortion” of the original music sample, this may be again related to high concentration to the test listening and ear/brains processing and mind evaluation. The question is what is perceived as a “distortion” - would it be real non-linear distortion, or reflections, room acoustics, or anything else? Again difficult to say. To me, the sample sounds very natural and that is why I have chosen it for the test.

Thank you and also @dougi and @LTig for participating in the test.

 

[pma]

One day pause, one more try, so I think it is proven. 8/10, 8/10 and 9/10. Tested interval was 7s - 12s. This time I took my headphone amp with close to zero output impedance (Sennheiser HD598 headphones).

foo_abx 2.0.2 report
foobar2000 v1.4.8
2020-12-10 20:26:03

File A: Quintet1.flac
SHA1: 78941da5875a6dc51169e29bbbb909399b75c377
File B: Quintet2.flac
SHA1: 2cea48c9ec2b493807e65e1d2b49deb19b543939

Output:
ASIO : Creative Sound Blaster ASIO
Crossfading: NO

20:26:03 : Test started.
20:28:32 : 01/01
20:29:04 : 02/02
20:29:28 : 02/03
20:29:44 : 03/04
20:30:07 : 04/05
20:30:24 : 05/06
20:30:48 : 06/07
20:31:19 : 07/08
20:31:35 : 08/09
20:31:59 : 09/10
20:31:59 : Test finished.

----------
Total: 9/10
Probability that you were guessing: 1.1%

-- signature --
05713a351055ed3def9a5c9b72de31182d3308f6

 

[Thomas_A]

No problem, thank you!

But, frankly, I prefer Foobar ABX, for the reason that one can make preliminary training and make a choice of the time interval where the difference is the biggest. In case of very similar samples, like here, it would be difficult to tell the difference by continuous streaming. Depends what we expect, to hear small differences or to hear big, substantial differences.

You can select the time interval where the difference is the biggest also with the linked trial.