What kind of evidence is sufficient?

[garbulky]

No your in the majority, have a look at the frailty of eye witnesses.. yet the law says..

People are not programmed to care about things that counter their own significance..

Well yeah, I probably am if you consider audio enthusiasts as a whole. But over here, I would say in the minority.

 

[Thomas savage]

Well yeah, I probably am if you consider audio enthusiasts as a whole. But over here, I would say in the minority.

In order to communicate anything even hinting at being worthwhile we need recognisable, established values..

Else it’s all just one guy looking for affirmation and that’s no basis of discussion..one can affirm or taunt themselves in a mirror if that’s the limitation of ambition to communicate.

 

[Blumlein 88]

I'm probably in the minority. For me, the evidence I need is obtained from extended listening to the device in the same way that I would normally enjoy such a device. And as for sounding good, I do have some pretty stringent nit picky ideals, which not everybody shares. So I think suffice it to say it has to sound good to me.

I would use people whose listening descriptions tend to match mine to see if some gear would be interesting to check out as well. However, I usually also would like it to meet some somewhat low standards for measurements. For instance, say an SNR of above 75, THD of less than 1%, <5% for tubes. A wide bandwith 20-20khz with no huge amounts of roll off. Though this doesn't preclude me from tube amps, I would prefer an amp to handle a 4 ohm impedance as well. If it's a DAC, at this stage I would prefer it to process 96khz files as I have several of them from my recordings. In the future, it would be nice if they could do 192 khz files. Luckily most DACs nowadays do 192 khz without issue.

At no point though, do I think I'm doing science.

This all sounds very logical and reasonable. I used to think similarly. But you know what? Its all rationalized malarky. I say that will all due respect. I do mean that. I'm trying to cut thru the gordian knot to communicate so this isn't a personal insult. Simply a blunt description from someone who has been there and done that.

There is some truth is every part of what you wrote above. And it all is nonetheless not really helpful to the level you seem to think it is.

 

[DuxServit]

Well, blind tests are certainly necessary because of all the impossible nonsense people claim to hear, which never ends.

"The published results must be repeatable by others using the identical process."

I am not so sure about this since I passed blind tests between high bit rate lossy vs. lossless years ago even though many people claimed that lossy should be transparent, even at low bit rates. I have not used lossy since.

Interesting that you interpret this in the context “blind tests” — I was thinking more of the kind of repeatable measurements that Amir performs. Repeatability is a fundamental requirement in published research especially in the physical/natural sciences.

 

[garbulky]

This all sounds very logical and reasonable. I used to think similarly. But you know what? Its all rationalized malarky. I say that will all due respect. I do mean that. I'm trying to cut thru the gordian knot to communicate so this isn't a personal insult. Simply a blunt description from someone who has been there and done that.

There is some truth is every part of what you wrote above. And it all is nonetheless not really helpful to the level you seem to think it is.

I hear ya. The important thing imo is that you are enjoying yourself. I think most people on this forum have figured out what works for them and rings true as well.

 

[Theo]

Definitely. Just not easy to find on all tracks and for all people. Here is my sample results:

Did you do this ABX test using headphones? Can you tell at what SPL level you were listening?

 

[amirm]

Did you do this ABX test using headphones? Can you tell at what SPL level you were listening?

Headphones. No SPL numbers. But I don't need high loudness to tell compression artifacts.

 

[vert]

Definitely. Just not easy to find on all tracks and for all people. Here is my sample results:

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/19 19:45:33

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44_01.mp3

19:45:33 : Test started.
19:46:21 : 01/01 50.0%
19:46:35 : 02/02 25.0%
19:46:49 : 02/03 50.0%
19:47:03 : 03/04 31.3%
19:47:13 : 04/05 18.8%
19:47:27 : 05/06 10.9%
19:47:38 : 06/07 6.3%
19:47:46 : 07/08 3.5%
19:48:01 : 08/09 2.0%
19:48:19 : 09/10 1.1%
19:48:31 : 10/11 0.6%
19:48:45 : 11/12 0.3%
19:48:58 : 12/13 0.2%
19:49:11 : 13/14 0.1%
19:49:28 : 14/15 0.0%
19:49:52 : 15/16 0.0%
19:49:56 : Test finished.

----------
Total: 15/16 (0.0%)

====

foo_abx 2.0 beta 4 report
foobar2000 v1.3.5
2015-01-05 20:26:27

File A: On_The_Street_Where_You_Live_A2.mp3
SHA1: 21f894d14e89d7176732d1bd4170e4aa39d289a3
File B: On_The_Street_Where_You_Live_A2.wav
SHA1: 3f060f9eb94eb20fc673987c631e6c57c8e7892f

Output:
DS : Primary Sound Driver

20:26:27 : Test started.
20:27:01 : 01/01
20:27:09 : 02/02
20:27:16 : 03/03
20:27:22 : 04/04
20:27:28 : 05/05
20:27:34 : 06/06
20:27:40 : 06/07
20:27:51 : 07/08
20:28:01 : 08/09
20:28:09 : 09/10
20:28:09 : Test finished.

----------
Total: 9/10
Probability that you were guessing: 1.1%

-- signature --
7a3d0c1aaaf8321306ff6cfdd1f91ff68f828a54

Excellent! I didn't go to such lengths because first of all I don't have the training. Though what I offered instead, I felt, was valid, and objective in its own way, but the guy declined to take my "test", which was a bit like, here's a painting, and the same painting in the exhibition catalog, can you tell the difference?

 

[Grave]

Interesting that you interpret this in the context “blind tests” — I was thinking more of the kind of repeatable measurements that Amir performs. Repeatability is a fundamental requirement in published research especially in the physical/natural sciences.

I would think that repeatability in tests involving different people would not mean much. But with measurements, sure.

 

[andreasmaaan]

Excellent! I didn't go to such lengths because first of all I don't have the training. Though what I offered instead, I felt, was valid, and objective in its own way, but the guy declined to take my "test", which was a bit like, here's a painting, and the same painting in the exhibition catalog, can you tell the difference?

@vert I'm sorry if you've already answered me, above, I just couldn't seem to find an answer anywhere.

I'm still curious as to your test conditions. Was it done double-blind ABX or some other method? Thx

 

[vert]

@vert I'm sorry if you've already answered me, above, I just couldn't seem to find an answer anywhere.

I'm still curious as to your test conditions. Was it done double-blind ABX or some other method? Thx

I did reply quickly from my smartphone "Why would that be necessary?"
I explained my "procedure" above. It may not have been scientific, but it tested something objective, details in a high register percussion part, not impressions, using a simple question, how do those details compare between FLAC and 320 kbps? And bam, they were near gone in the 320 kbps. Now that is a crying shame, given the insane subtlety Brazilian music sometimes has. It's part of the reason why some recordings can be listened to dozens of times, because you keep noticing things you hadn't before.

It still took me close to one hour to do, which I thought was a lot of time to spend on this on a weekend instead of time with my family.

Yes, I knew the track very well, it wasn't random. I knew what to look for, a detail buried deep in the recording. As I understand, this isn't how such tests are normally performed. But I don't see any problem with doing it the way I did. I think double-bind ABX was what the guy wanted me to do to continue the conversation. Which. other than the fact that I don't have the necessary training, struck me as ridiculously out of proportion with the case at hand. (From my perspective, also, there was someone defending the point of view of a company, Apple, marketing some rather pricy phones with very small storage space, who would have a vested interested in promoting 320 kbps as identical in quality to lossless. The questions I had in that respect were simply ignored)

No doubt there can be a kind of dogmatism in some science or science-minded circles that is very damaging. When I offered some contrary, yet interesting empirical results instead, there was no reply because they didn't fit the narrow frame of what the guy considered was the only possible valid procedure. Now some would say that is a very unscientific attitude in itself. Regardless, Amirm got to the same conclusions using a "bona fide", much more extensive process so to me it's, well, case closed.

 

[andreasmaaan]

O

I did reply quickly from my smartphone "Why would that be necessary?"
I explained my "procedure" above. It may not have been scientific, but it tested something objective, details in a high register percussion part, not impressions, using a simple question, how do those details compare between FLAC and 320 kbps? And bam, they were near gone in the 320 kbps. Now that is a crying shame, given the insane subtlety Brazilian music sometimes has. It's part of the reason why some recordings can be listened to dozens of times, because you keep noticing things you hadn't before.

It still took me close to one hour to do, which I thought was a lot of time to spend on this on a weekend instead of time with my family.

Yes, I knew the track very well, it wasn't random. I knew what to look for, a detail buried deep in the recording. As I understand, this isn't how such tests are normally performed. But I don't see any problem with doing it the way I did. I think double-bind ABX was what the guy wanted me to do to continue the conversation. Which. other than the fact that I don't have the training to do such a thing, struck me as ridiculously out of proportion with the case at hand,. No doubt there can be a kind of dogmatism in some science or science-minded circles that is very damaging. When I offered some interesting empirical results instead, there was no reply because they didn't fit the narrow frame of what the guy considered was the only possible valid procedure. Now some would say that is a very unscientific attitude in itself. Regardless, Amirm got to the same conclusions using a "bona fide" process so to me it's case closed.

Ok thanks for clarifying You explained the procedure in the original post about it, but didn't mention whether it was blind or sighted or ABX or whatever else. What you did was a sighted test, trying to work out if what you heard correlated to what you knew you were listening to, correct?

 

[andreasmaaan]

I will do this as soon as I get a chance, sure

Ok, as promised yesterday, here is a rundown of the test I did that hopefully allows you to try yourselves if interested.

The software: phase audibility

The settings I tested myself with were:
- Polarity: in phase
- Low filter: off
- Mid filter: on
- F: 1210Hz
- LR 24dB/Oct

Or as a screenshot:

The track I used was this one in 16/44.1 WAV format. I hadn't actually heard the track before doing the test actually Just happened to want to listen to the album for the first time the same night I came across the test...

EDIT: fwiw, the headphones I used were Audio-Technika ATH-M50x.

Cheers,
Andreas

 

[vert]

O


Ok thanks for clarifying You explained the procedure in the original post about it, but didn't mention whether it was blind or sighted or ABX or whatever else. What you did was a sighted test, trying to work out if what you heard correlated to what you knew you were listening to, correct?

Yes