Master Thread: Are measurements Everything or Nothing?

[solderdude]

Requires a MOT tag ?

 

[Reynaldo]

AKA trolling?

Awesome...

But Chinese companies also launch new devices every few months.
Always with some measurement a little higher.
At least the packaging is new.
Isn't that also trolling too?

 

[Galliardist]

High end audio. I forgot the name of his company (something like AWX?) but they sell very expensive DACs and other frippery.

edit: AMR. Abbingdon Music Research. Also magic fuses, wires, power cords, the whole shtick.

Gold Reference Replacement Fuses by AMR Audio

The golden touch. The AMR Gold Fuse is a reference replacement fuse for all electronic products. Just add gold and both audio and visual products from amplifiers to LCD televisions will no longer be "degraded" by running the electricity through a compromised, mass produced fuse.

amr-audio.co.uk

This AMR? Oh, good. Maybe we can see some proper evidence from a fuse seller at last. Quite the series of claims there.

 

[Purité Audio]

I remember trialling the AMR cd player when PA was first starting, it looked splendid, it had various output options, Ss/valve I believe and it was the only player I tried that sounded different, unfortunately not in a good way.
Keith

 

[RayDunzl]

propose a mechanism why that may sharpen the leading wavefronts and produce music with a little more punch as a result. Add this to the rhythm chords, and you may find the music has more drive. (Or not - we could test!)

Good idea!

Sometimes harmonics are not necessarily in lockstep phase with the fundamental, though.

Here would be the most extreme case, where the fundamental and the third harmonic are (sort of) 180 degrees out of phase (third is inverted)

 

[ahofer]

Why point out one problem clearly when you can wave your hands vaguely at a thousand?

And we should reject the compelling evidence we have in favor of theoretically better evidence we do not.

This is intentional confusion.

 

[Galliardist]

I remember trialling the AMR cd player when PA was first starting, it looked splendid, it had various output options, Ss/valve I believe and it was the only player I tried that sounded different, unfortunately not in a good way.
Keith

They're entitled to make a player that sounds different and if it does sound different they can then make claims that we may find such a sound more enjoyable, I guess.

I'm more concerned with a particular claim made about the fuses. The site states that

AMR Gold Fuses not only improve audio, but they also enhance the performance of your TV screen too, which means you’ll enjoy your favourite shows and movies with:

• Sharper picture definition
• Richer colours
• Blacker blacks
• Massively reduced picture “noise”

So as I understand it, those claims for picture quality can be objectively tested outside of the subjectivist mess that surrounds audio, and doesn't involve giving away any special audio secrets or invoking mysteries of the art. I'd like to see evidence of this claim, please.

 

[Purité Audio]

Sounds iffy doesn’t it.
Keith

 

[Thorsten Loesch]

I agree with the fact that it is limited. The downsides you mention also works in a clarifying way. Some extra input could be a confidence number assigned to the test by the one taking the test that is entered before the test results are given.

Confidence numbers are naturally all back down to statistics.

What this table shows is that for a small change (p=.6) 9/10 correct out of 16 give around equal type A/B (or as it says there 1/2) Error risk, in the region of 30ish %.

That is out of 10 results (null or non-null) 3 can be down to chance.

With 13 correct out of 16 the risk of type A error is only 1% but the risk of type B error is ~ 94%, that is only one in 100 non-null result is likely due to error but 94% of null results are due to error.

Also notable, with 100 trials and 58 correct we have a lower than 1% risk of all statistical errors and can be mollified that g*ds of statistics have been suitably mollified.

But if (say) 10 individuals too the same ABX test and scored 6/10 the common interpretation would be to accept the null hypothsis (all subjects heard nothing), though meta analysis would suggest that a difference existed with very high confidence.

BTW, any ABX where the detailed data is given (sadly fewer than one would like) allow the average (amateur) statistician to re-evaluate the raw data with different criteria and then draw her or his or it's conclusion differently to the sated ABX conclusion.

The test can show that differences are too small to be picked.

BY THAT SPECIFIC TEST, within the stated statistical limits.

That said... I have done some long term blind AB tests where I was not aware if the one I asked to make a change in the system made a change but was logged and after more than 10 'evaluations' I thought were enough I had a result. Turned out the one I asked (the wife) never changed anything as she forgot and we never discussed it afterwards. Yet totally convinced I heard differences.

Same issue as mentioned. Expectation bias. You set yourself up for this.

This is pretty basic psychology.

So long term, quick (level matched) ABX, AB and other tests all fall flat on their face as soon as the brain comes into play.

No, they fail as soon as BIAS comes into play.

The whole point of blind testing is to remove bias of course, but let's just say neither your long term blind AB test nor Audio ABX as practiced are sufficiently blind to remove bias. It is the bias that causes erroneous outcomes in case statistics themselves are sound (pun intended).

I don't trust the brain except for it letting me know what I like and do not like. For that it works well.

And hence my tests ask you "How do you like this?" several times for several identical and different items.

Do you see where I am coming from?

So measurements it is... the difficulty here is what are the real hearing limits and to whom and what circumstances... pesky hearing/brain.

Measurements are synthetical bench tests. As such they are just data. Data in itself is not actionable. Information is.

Information is Data in context. So the context is to say this dataset compares to the requirements as follows.

Without that crucial step data is just noise. No Signal.

Thor

 

[Thorsten Loesch]

High end audio. I forgot the name of his company (something like AWX?) but they sell very expensive DACs and other frippery.

edit: AMR. Abbingdon Music Research. Also magic fuses, wires, power cords, the whole shtick.

I had nothing to do with these companies since 2019 and have in the time since not worked on audio or publically written about it (contractual obligations).

So no. And not every AMR or iFi product is directly mine.

The fuses have a REALLY interesting story behind them. Ask me one of these days.

Thor

 

[Thorsten Loesch]

Requires a MOT tag ?

I messaged Amir about this.

 

[Thorsten Loesch]

I'm more concerned with a particular claim made about the fuses. The site states that

So as I understand it, those claims for picture quality can be objectively tested outside of the subjectivist mess that surrounds audio, and doesn't involve giving away any special audio secrets or invoking mysteries of the art. I'd like to see evidence of this claim, please.

Talk to the marketing futzies.

I would not write that. In fact, had I had my way I would have never had these fuses offered for sale.

Thor

 

[killdozzer]

@Thorsten Loesch So... How has it been working for you so far? Satisfied? What's your estimate?

 

[manisandher]

The obvious Ad Hominem attacks against @Thorsten Loesch in this thread suggest a total lack of refutation against his central point. Speaks volumes...

 

[Thorsten Loesch]

@Thorsten Loesch So... How has it been working for you so far? Satisfied? What's your estimate?

What is "IT"?

Thor

 

[Rottmannash]

Pawel (I think in Czechia it would use a w not a v?) , yes, I have at times used nome du plume/guerre instead of my real name, but Thorsten Loesch is real. Check the videos linked in my profile.

Yes, there are no simple answers. There are only simple comforting lies and complex and inconvenient truth's.

I think too few people these question their beliefs and what they are told, something that (should) come as absolutely essential and naturally to anyone growing up and living under "Real Existierendem Sozialismus" (der -ismuss wo jeder mitmuss or die Stasi hilft Dir bei mitmuessen).

[real existing socialism, a slogan in the 80's to disguise the systemic failings of the eastern block system, the rest is a funny ditty in german, it means that socialism is an ism where everyone must join in, or the secret police will help you develop the correct attitude]

Thor

Totally OT but am currently reading the new Clancy book and it reflects much of what you say about East Germany in those days. Interesting to delve back to that time in history.

 

[killdozzer]

The obvious Ad Hominem attacks against @Thorsten Loesch in this thread suggest a total lack of refutation against his central point. Speaks volumes...

Perhaps you have a completely subjective understanding of ad hominem (no, the Romans didn't capitalize each first letter), but I haven't seen one. Not even one. It's the same ol' lack of evidence we've seen numerous times. In fact Thorsten himself said he deliberately avoids giving evidence. Which makes your comment resemble sucking up.

 

[killdozzer]

What is "IT"?

Thor

You keep working on that one. I'm sure you'll arrive at the right conclusion. I don't see you as unintelligent.

 

[IPunchCholla]

Perhaps you have a completely subjective understanding of ad hominem (no, the Romans didn't capitalize each first letter), but I haven't seen one. Not even one. It's the same ol' lack of evidence we've seen numerous times. In fact Thorsten himself said he deliberately avoids giving evidence. Which makes your comment resemble sucking up.

I was stuck on the idea of refuting the central point, when as far as I have been able to make out, there hasn’t been one.

 

[solderdude]

Confidence numbers are naturally all back down to statistics.

That's not what I meant.
When someone is taking the test and before the results are known they should enter a number stating how confident they were about their answers.
When someone enters 'really confident' and they score near 'gambling' you have feedback concerning the one taking the test.

Same issue as mentioned. Expectation bias. You set yourself up for this.

This is pretty basic psychology.

Yes, unavoidable. That's the whole issue with all but controlled tests with statistical validity. Hardly anyone takes these.

However, when one wants to know for themselves if they like something or not OR if they can hear a (worthwhile) difference at all AND the test is done in a good enough way that test is valid for that individual.

No, they fail as soon as BIAS comes into play.

There will always be some bias. Knowing some of your abilities is being tested does that. There is no way around that.
For perception testing one should remove as much bias as possible. Trying for 0 bias is not possible anyway. Also not when just stating a preference (sighted or not sighted)

Measurements are synthetical bench tests.

Nulling.... ? Also not perfect but when possible (only in the electric domain) but it can show differences and how big they are and what the difference sounds like.
Well... one could argue how to null a signal for 'air' what would that sound like ?

I can go along with your line of thinking quite a long way but to me it seems that while you base a lot of measurements you do not trust them much.
I, on the other hand, know that signal fidelity is measurable (in the electrical domain) and is only kind of inaccurate and basic for acoustic measurements (on amateur level).
It has also become clear to me NOT to trust the hearing when it comes to signal fidelity.
Preference and signal fidelity are different things. Sometimes there is correlation, sometimes not. In that case the brain/hearing cannot be trusted. It can be somewhat trusted for preference.

Measurements are synthetical bench tests. As such they are just data. Data in itself is not actionable. Information is.

Measurements are data, interpretation of measurements gives information.
Correct interpretation requires many different measurements (all giving pieces of information) which combined give the needed information.

The caveats are: Not enough measurements, reliability of the measurements and interpretation as well as presentation.
Knowing how to interpret the measurements to form an educated opinion is key.
IMO one needs to have done extensive testing including listening tests and knowledge how to condense that in a language laymen can relate to.

The biggest issue I have with tests is that when 1 out of a number of testers shows he seems to be acing a test (hearing something others could not) is the way this is handled.
A: You see 1 person is trained enough... or he can definitely hear something and the rest can't so that one person has really proven something
B: That 1 person should be tested again to see if he/her/X can discern this aspect reliably and more research is needed. We may need to find more of these nuggets.

B would be handy for audibility research etc..
A only says something about that for that person and that test there might be a difference detectable. That would not be clear evidence of the existence of audible difference in general or determining development of some gear.