Master Thread: Are measurements Everything or Nothing?

[welwynnick]

As I say, we can guess all we like, but do a proper controlled test, and you know the truth.

I used to do lots of them through AVF, but it's easier said than done. They're hard work and very time consuming, and keyboard warriors always pick holes in what you do.

 

[antcollinet]

When I said the "most expensive kit would win every time" it was an ironical reposte to the usual ASR attitude that sighted listening is subject to congitive bias every time.

That is not what it is said.

What is said is that Cognitive bias MIGHT Be involved every time there is an uncontrolled listening test. It means such tests are unreliable every time because we DON"T KNOW if cognitive bias played a part.

A subtle but important difference.


Then when devices that are compared are so good that the measurements tell us they will be audibly indistinguishable - and someone still claims a difference can be heard in uncontrolled listening - we can reasonably assume the reason is bias. Just probabilities.

AND ALSO IMPORTANT.

Bias is not necessarily about what we consciously expect. They are a subconscious process, and will be impacted by all sorts of subconscious "expectations"** based on a lifetime of experience.

** There are also many other biases than expectation bias.

 

[welwynnick]

What is said is that Cognitive bias MIGHT Be involved every time there is an uncontrolled listening test. It means such tests are unreliable every time because we DON"T KNOW if cognitive bias played a part.

That ought to be the response, but my point was that it usually isn't.

 

[antcollinet]

Agreed.

And if someone hears a difference in sighted listening, then it is "probably" cognitive bias.

Not probably.

I can hear the difference between a dog and a cat. I don't need a blind test to prove that.

Similarly I can hear significant noise from the speakers.

We can start to suspect bias at play when the measurable differences are so small as to be unlikely to be audible. That doesn't mean if we perceive a difference as "night and day". It is quite possible to perceive night and day difference due to bias. They are that significant.

 

[antcollinet]

That ought to be the response, but my point was that it usually isn't.

That's because usually the claims are made of devices when the differences are so small as to be likely inaudible.

 

[Yorkshire Mouth]

I used to do lots of them through AVF, but it's easier said than done. They're hard work and very time consuming, and keyboard warriors always pick holes in what you do.

I totally agree.

But all you can do is your best, and if people don’t like it for logical reasons they can say so, and if they don’t like it due to ignorance they can f*** off.

But at the same time, we must always reevaluate our own position.

 

[john11]

I've had enough of this. That is simply not true. I can prove it.

Some time ago I borrowed some equipment to prove a point that I'd long been theorising about. I expected that compressed soundtracks sounded better when they were decoded by the processor rather than the player. It was a longstanding obsession, and I'd heard other people's listening impressions (about which I was somewhat sceptical) but I had never heard it for myself.

Then the opportunity came and the anticipation was overbearing. However, in a sighted, level-matched comparison I couldn't hear any difference. Everyone else must have been imagining it, or I must be deaf, or my understanding of audio electroncis wasn't as good as I thought it was. I had nailed my colours to the mast, and this was embarassing, but I immediately wrote about what I heard. On this ocassion the expectation bias (nothing to do with shiney or expensive) was overwhelming, but try as I might, player and processing decoding sounded just the same to me.

Price list for new Onkyo PR-SC886

Right, some impressions. There's a great deal to say, and I won't be able to do it all at once. I've had the golden opportunity to answer a lot of questions, many of which relate to the Denon DVD 3800BD. For the purpose of this thread, though, I'll just consider it as a good HD MC audio...

www.avforums.com

my own am
As it happened, I had made a mistake and unwittingly compared the same things. It was a dummy control test that I didn't realise I had set myself. #I was expecting to hear a significant difference, but there was none, and that was what I heard. I then fixed and repeated the test and did hear the difference I was expecting (processor decoding did sound better).

I'm only speaking for myself there, and I'm sure that cognitive bias does have a significant influence on sighted listening, but not in this instance.

This is why my signature says I didn't imagine what I heard.

Good points. I have had many similar experiences where i expect hardware to sound better or worse, but upon listening they sound the opposite.
I build my own amps and speakers, and often go back to the circuit diagram to try and figure out what is happening
Critics on here are demanding proof and suggesting i am not being scentific, but when the critics themselves are using words like " Unlikely " and " Likely " this shows that they are putting forward a point of view or opinion and are not being scientific themselves and are armchair critics
The way i see it if people are willing to pay £250 for the PA5 II, then 50p for a capacitor is easy enough.
But if critics are afraid to step out into the real world and perform and mod, and only want to live in a world of points of view or opinions then that is up to them

 

[Julf]

Good points. I have had many similar experiences where i expect hardware to sound better or worse, but upon listening they sound the opposite.

That is not how unconscious bias works.

Critics on here are demanding proof and suggesting i am not being scentific, but when the critics themselves are using words like " Unlikely " and " Likely " this shows that they are putting forward a point of view or opinion and are not being scientific themselves and are armchair critics

And that sentence makes me think you don't understand science. In science one is always prepared for exceptions and uncertainties.

But if critics are afraid to step out into the real world and perform and mod, and only want to live in a world of points of view or opinions then that is up to them

Not at all. If you claim something makes a difference, that is a subjective opinion until you provide objective proof - and the burden of proof is on you.

 

[antcollinet]

Good points. I have had many similar experiences where i expect hardware to sound better or worse, but upon listening they sound the opposite.

AND ALSO IMPORTANT.

Bias is not necessarily about what we consciously expect. They are a subconscious process, and will be impacted by all sorts of subconscious "expectations"** based on a lifetime of experience.

** There are also many other biases than expectation bias.

 

[Yorkshire Mouth]

…when the critics themselves are using words like " Unlikely " and " Likely " this shows that they are putting forward a point of view or opinion and are not being scientific themselves…

Sorry, but that’s not accurate.

Science is often to do with probability rather than certainty.

Take the Covid vaccines. You don’t get absolute 100% immunity, but you’re less likely to die once vaccinated.

Bias (which is what we’re discussing) is far more about likelihood than certainty.

 

[Killingbeans]

I have had many similar experiences where i expect hardware to sound better or worse, but upon listening they sound the opposite.

Read any subjective review of a snake-oil product, and you'll be almost guaranteed to see a similar experience being descibed.

"I was extremely sceptical about this product doing anything, but it works!"

Cognitive bias is not controlled by our logical reasoning. You'd might have had knowledge about the technical aspects of piece of hardware and predicted a better or worse sound from those, but you got a subconscious expectation from the visual design of the hardware or some other non-technical aspect, and it flipped your impression of the sound without you even knowing it.

It's completely out of our control.

Amps all sound the same when you get to a certain level

But then you add a capacitor to the PSU input, and they all sound better

 

[welwynnick]

Read any subjective review of a snake-oil product, and you'll be almost guaranteed to see a similar experience being descibed.
"I was extremely sceptical about this product doing anything, but it works!"some

There's a difference. John isn't rrying to sell anything. He's just trying to make a positive contribution to a forum, which is more than I can say for some.

Perhaps this should be renamed the Audio Psuedo-Science Forum, where the entry requirement is to have read a scientific paper.

 

[Killingbeans]

There's a difference. John isn't rrying to sell anything.

I don't believe most snake-oil reviewers are trying to sell anything either. Their employers might be, but I suspect that the reviewers themselves genuinely believe that they are witnessing the most cutting-edge innovations in audio reproduction technology.

That's what makes it really depressing. It would be way easier to distance ourselves from it, if they were all just unscrupulous con men.

He's just trying to make a positive contribution to a forum, which is more than I can say for some.

How is a wild claim with no reliable verification a "positive contribution"?

If he had made it clear that his experience should be taken with a sizable grain of salt, then yeah sure. It would just be a bit of fun noise that people could chose to ignore.

But he presented it with conviction, making it an enticement to magical thinking.

 

[Julf]

I don't believe most snake-oil reviewers are trying to sell anything either. Their employers might be, but I suspect that the reviewers themselves genuinely believe that they are witnessing the most cutting-edge innovations in audio reproduction technology.

Then there are the ones that are trying to sell an audiophile belief system, as it supports their blogs/youtube channels/reputation. Some of them act on behalf of vendors who give them discounts or lend them equipment.

 

[Induna]

All human knowledge is contingent, and upon what is very often unclear to us, or even impossible to elaborate. That being said all of us are usually able to self-correct our grossest misinterpretations of the world around us, especially when it affects our survival or our love-life . With some discipline and experience we are able to incorporate self-critical routines into many of our thought processes. They may even become habitual.

Although I can't prove it because it isn't an appropriate topic of scientific inquiry, I suspect that Humility and self-awareness (as opposed to self-consciousness ) have an awful lot to do with the ability to see the world around us more as it is and less as we want or are accustomed to seeing it. I think it is inaccurate to depict us all as hopelessly naive with respect to our own biases and incapable of accounting for our cognitive vulnerabilities. I also think it is rather childish to insist that one can hear a -120db anomaly when the likelihood of that is vanishingly small. (That was a pathetic attempt to bring my ramblings back on topic.)

 

[antcollinet]

hopelessly naive with respect to our own biases and incapable of accounting for our cognitive vulnerabilities

Again - it is not about being naive or stupid or anything along those lines. It is about being human. Even when we know we are bing misled by our biases we are unable to stop it happening. It is simply the way we are wired.

 

[pundel]

I want to give my review on 8 devices: smsl su-x, smsl sh-x, topping a70 pro, smsl su-10, smsl do400, aune s17pro, sound blaster ae-9, sound blaster ae-5. Overall, in the order of purchase, briefly: sound blaster ae-5 works the same as sound blaster ae-9. I didn't understand it back then, but later I learned that I should have bought an amplifier, but by that time I had decided to buy something else. Rating 3/10. Then I bought smsl do400. I bought it at a discount for $386.09. This one is top-notch, I really liked it, the soundstage is large and detailed everywhere, except for the height, meaning it's hard to say at what height exactly the sound is, but the sound is detailed around the sides, and I'm satisfied. I only listen to pop and electronic music depending on my mood, classical music. I'd rate it 6/10, for its price, you can't find a better device, and I'll still sell it in a few days. I want to say that the sound is perfectly balanced only with the native amplifier, as I connected it to aune s17pro. I bought it for $492.64 on Black Friday discount, the sound was better, but my headphones had 16 ohms resistance, and the amplifier was for 32, it was my mistake, there were distortions at high volume, in general, this amplifier lacks volume, and I can't say much about it because I returned it right away. I bought both s17pro and su-10 at the same time, I immediately understood that su-10 is better than my do400 at 8/10, it's warm by 40% and precise by 20%, so the sound as if warms the ears, and it's nice in music, but not always good in games. I wanted to keep it for myself, but then the new smsl su-x came out, and I immediately understood that while I could return it, I bought su-10 for $658.46 on Black Friday discount. Topping a70 pro, I bought it for $412.30 and connected it to smsl do400, the sound was the same, nonsense, I thought, I increased the volume and was very surprised that it muted the sound at 10 dB and above, and I was afraid that the headphones would suffer from abrupt turning on and off at high volume, I thought, the DAC is crap, now the new smsl su-x will come, and it will be fine, no, it's the same, the firmware was the newest from the factory, and I didn't change anything in the settings, as it arrived, I even reset the settings to factory ones, it didn't help, it still muted, in short, I didn't like it both in design and in sound, because my built-in amplifier for $200 sounded the same, I wrote to the seller, asked, he said, there's power protection, why the hell give such power, idiotic device, rating 5/10, maybe you need to use it only with topping, I switched all the settings, but you can't disable the protection, my headphones are Hifiman XS. smsl sh-x, the sound is the most transparent, it will sound as you set it up, namely, there are 3 types of amplification: low, the sound sounded simple, and I lacked bass, MID — this is my favorite sound, very strong bass, strong and fast, high I haven't tried, it arrived only 2 days ago, I bought it for $602.30, I want to say that when connected to do400 it sounded better, but at high volume there were distortions in the mode without filter, if you switch the filter, there's almost no noise, in general, the amplifier interfered with the balanced sound of do400, my rating for smsl sh-x is 10/10, the best I've listened to, for its flexible adjustment, design, it's louder than all the amplifiers I've listened to here, and I like everything about it, and in my new DAC smsl su-x 10/10 I bought smsl su-x $854.05 — it's my perfect sound for both music and games, I'd define the sound as 25% warm, 25% precise — it's the perfect balance for all kinds of music and games, the soundstage is the widest of all, the detail is perfect in terms of balance, precision, and warmth, it's pleasant to listen to. It perfectly matches the sound of SMSL SH-X, definitely much better than Topping. Topping just suffocates the sound of SMS SU-X. The filter settings are audible and not at the level of seeming, but definitely with a sound tint, I like to listen without filter and sound tint, standard, I'll post my review in other topics of these devices' data. If you have any questions, I'll be happy to answer in more detail.

 

[pundel]

I want to give my review on 8 devices: smsl su-x, smsl sh-x, topping a70 pro, smsl su-10, smsl do400, aune s17pro, sound blaster ae-9, sound blaster ae-5. Overall, in the order of purchase, briefly: sound blaster ae-5 works the same as sound blaster ae-9. I didn't understand it back then, but later I learned that I should have bought an amplifier, but by that time I had decided to buy something else. Rating 3/10. Then I bought smsl do400. I bought it at a discount for $386.09. This one is top-notch, I really liked it, the soundstage is large and detailed everywhere, except for the height, meaning it's hard to say at what height exactly the sound is, but the sound is detailed around the sides, and I'm satisfied. I only listen to pop and electronic music depending on my mood, classical music. I'd rate it 6/10, for its price, you can't find a better device, and I'll still sell it in a few days. I want to say that the sound is perfectly balanced only with the native amplifier, as I connected it to aune s17pro. I bought it for $492.64 on Black Friday discount, the sound was better, but my headphones had 16 ohms resistance, and the amplifier was for 32, it was my mistake, there were distortions at high volume, in general, this amplifier lacks volume, and I can't say much about it because I returned it right away. I bought both s17pro and su-10 at the same time, I immediately understood that su-10 is better than my do400 at 8/10, it's warm by 40% and precise by 20%, so the sound as if warms the ears, and it's nice in music, but not always good in games. I wanted to keep it for myself, but then the new smsl su-x came out, and I immediately understood that while I could return it, I bought su-10 for $658.46 on Black Friday discount. Topping a70 pro, I bought it for $412.30 and connected it to smsl do400, the sound was the same, nonsense, I thought, I increased the volume and was very surprised that it muted the sound at 10 dB and above, and I was afraid that the headphones would suffer from abrupt turning on and off at high volume, I thought, the DAC is crap, now the new smsl su-x will come, and it will be fine, no, it's the same, the firmware was the newest from the factory, and I didn't change anything in the settings, as it arrived, I even reset the settings to factory ones, it didn't help, it still muted, in short, I didn't like it both in design and in sound, because my built-in amplifier for $200 sounded the same, I wrote to the seller, asked, he said, there's power protection, why the hell give such power, idiotic device, rating 5/10, maybe you need to use it only with topping, I switched all the settings, but you can't disable the protection, my headphones are Hifiman XS. smsl sh-x, the sound is the most transparent, it will sound as you set it up, namely, there are 3 types of amplification: low, the sound sounded simple, and I lacked bass, MID — this is my favorite sound, very strong bass, strong and fast, high I haven't tried, it arrived only 2 days ago, I bought it for $602.30, I want to say that when connected to do400 it sounded better, but at high volume there were distortions in the mode without filter, if you switch the filter, there's almost no noise, in general, the amplifier interfered with the balanced sound of do400, my rating for smsl sh-x is 10/10, the best I've listened to, for its flexible adjustment, design, it's louder than all the amplifiers I've listened to here, and I like everything about it, and in my new DAC smsl su-x 10/10 I bought smsl su-x $854.05 — it's my perfect sound for both music and games, I'd define the sound as 25% warm, 25% precise — it's the perfect balance for all kinds of music and games, the soundstage is the widest of all, the detail is perfect in terms of balance, precision, and warmth, it's pleasant to listen to. It perfectly matches the sound of SMSL SH-X, definitely much better than Topping. Topping just suffocates the sound of SMS SU-X. The filter settings are audible and not at the level of seeming, but definitely with a sound tint, I like to listen without filter and sound tint, standard, I'll post my review in other topics of these devices' data. If you have any questions, I'll be happy to answer in more detail.

 

[Killingbeans]

That being said all of us are usually able to self-correct our grossest misinterpretations of the world around us, especially when it affects our survival or our love-life .

Sorry if this was a joke that went over my head, but both of those are notorious for throwing logic out the window and letting bizarre shortcuts dominate

 

[marcom22]

I want to give my review on 8 devices: smsl su-x, smsl sh-x, topping a70 pro, smsl su-10, smsl do400, aune s17pro, sound blaster ae-9, sound blaster ae-5. Overall, in the order of purchase, briefly: sound blaster ae-5 works the same as sound blaster ae-9. I didn't understand it back then, but later I learned that I should have bought an amplifier, but by that time I had decided to buy something else. Rating 3/10. Then I bought smsl do400. I bought it at a discount for $386.09. This one is top-notch, I really liked it, the soundstage is large and detailed everywhere, except for the height, meaning it's hard to say at what height exactly the sound is, but the sound is detailed around the sides, and I'm satisfied. I only listen to pop and electronic music depending on my mood, classical music. I'd rate it 6/10, for its price, you can't find a better device, and I'll still sell it in a few days. I want to say that the sound is perfectly balanced only with the native amplifier, as I connected it to aune s17pro. I bought it for $492.64 on Black Friday discount, the sound was better, but my headphones had 16 ohms resistance, and the amplifier was for 32, it was my mistake, there were distortions at high volume, in general, this amplifier lacks volume, and I can't say much about it because I returned it right away. I bought both s17pro and su-10 at the same time, I immediately understood that su-10 is better than my do400 at 8/10, it's warm by 40% and precise by 20%, so the sound as if warms the ears, and it's nice in music, but not always good in games. I wanted to keep it for myself, but then the new smsl su-x came out, and I immediately understood that while I could return it, I bought su-10 for $658.46 on Black Friday discount. Topping a70 pro, I bought it for $412.30 and connected it to smsl do400, the sound was the same, nonsense, I thought, I increased the volume and was very surprised that it muted the sound at 10 dB and above, and I was afraid that the headphones would suffer from abrupt turning on and off at high volume, I thought, the DAC is crap, now the new smsl su-x will come, and it will be fine, no, it's the same, the firmware was the newest from the factory, and I didn't change anything in the settings, as it arrived, I even reset the settings to factory ones, it didn't help, it still muted, in short, I didn't like it both in design and in sound, because my built-in amplifier for $200 sounded the same, I wrote to the seller, asked, he said, there's power protection, why the hell give such power, idiotic device, rating 5/10, maybe you need to use it only with topping, I switched all the settings, but you can't disable the protection, my headphones are Hifiman XS. smsl sh-x, the sound is the most transparent, it will sound as you set it up, namely, there are 3 types of amplification: low, the sound sounded simple, and I lacked bass, MID — this is my favorite sound, very strong bass, strong and fast, high I haven't tried, it arrived only 2 days ago, I bought it for $602.30, I want to say that when connected to do400 it sounded better, but at high volume there were distortions in the mode without filter, if you switch the filter, there's almost no noise, in general, the amplifier interfered with the balanced sound of do400, my rating for smsl sh-x is 10/10, the best I've listened to, for its flexible adjustment, design, it's louder than all the amplifiers I've listened to here, and I like everything about it, and in my new DAC smsl su-x — it's my perfect sound for both music and games, I'd define the sound as 25% warm, 25% precise — it's the perfect balance for all kinds of music and games, the soundstage is the widest of all, the detail is perfect in terms of balance, precision, and warmth, it's pleasant to listen to. It perfectly matches the sound of SMSL SH-X, definitely much better than Topping. Topping just suffocates the sound of SMS SU-X. The filter settings are audible and not at the level of seeming, but definitely with a sound tint, I like to listen without filter and sound tint, standard, I'll post my review in other topics of these devices' data. If you have any questions, I'll be happy to answer in more detail.

Great so you dont have any problem with your HifiMan headphones with the SMSL SH-X, is it right? And seems that you had our same problem with the Topping A70 Pro, correct?