• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Consumer evaluation strategies personal evaluation or relying on external evidence

RayDunzl

Grand Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
13,198
Likes
16,981
Location
Riverview FL
Just confirming that the batts are 7" thick? If so, why so thick?

184mm.

Thicker guesstimated to have more attenuation at lower frequencies.
 

Sal1950

Grand Contributor
The Chicago Crusher
Forum Donor
Joined
Mar 1, 2016
Messages
14,072
Likes
16,604
Location
Central Fl
Today's (and some of yesterdays) electronics are for the most part transparent, that leaves you with the big speaker decision.
For me, I have owned Dr Hsu of Hsu Research products since the early ninety's. I've talked to him personally a number of times on the phone, and in later years via emails and forums. He's always impressed me as a honest person, stood my his products, and is a highly competent designer with a good ear. He also offers the standard 30 satisfaction thing so trying out his latest line was a no-brainer for me when I was looking for a new system.
I saw no reason to send them back.
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
There's a certainty of self deception, fallibility and faulty conclusion Jacob with all human subjective appraisals of audio devices.

Certainty? Not sure. :)
It´s more a probability, means in the long run, everyone is most likely sometimes wrong, but always? Certainly not. There are of course people, who are wrong nearly everytime. Reminds me to a story told in one of the fundamental textbooks about sensory evaluation, describing a longtime participant in tests, who was always sure there was no difference, but his data showed that he detected one everytime in around 20 tests.

The conclusions audiophiles ( and it's only audiophiles as 'normal ' humans just accept the sound coming from a device more or less) make reminds me of those who wish to assign human feelings and meaning onto animal behaviours.

Might be, as said before, "audiophiles" are as heterogenous as other groups as well, but imo the main problem is the generalization of any issue and the jumping to conclusions about the causalities.

But back to the decision strategy; stick to measurements is at first glance a good advice, but if there is the choice between two (or more) alternatives, all departing from the ideal to a similar degree but in different ways, it gets difficult.

Johnny Morris and the like.. it's very appealing but it's total bollocks.

Unfortunately i don´t understand the "Johnny Morris hint".....

There's 2 certainties in life, death and doubt, the rest is just the best we can do.

So, it´s more a mix depending on more than one variable?
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
<snip>

I have no idea what you are talking about here. If it is some sighted listening impressions just say so. I'll ignore the issue. Audio electronics should accurately with low distortion, low noise and flat response reproduce the signals they are fed. DACs are widely available at relatively low cost that accomplish this to nigh on theoretical perfection.

I was referring to the measured problems over the decades- this some threads in this forum and in other sources as well - with DACs (so i don´t know why you had an association to "sighted" listening).
Which of these problems would have been predictable (and are predictable now) from the usual specifications that manufacturers publish?

I rely on the best evidence available. For items that have some goal or standard those getting closest to that are better. Whenever possible I do not wish to rely on recommendations by others. I prefer external evidence. Some areas and some products are simply not that cleanly delineated. So it will be a mix. If it becomes a preference, then you'll need to figure out what your preferences lean toward and use recommendations of those with similar preferences. Now there are other aspects of products like reliability. If I look on Amazon and actual owners liked how a product works, but are saying it went kaput in a few weeks or it broke or it wore out too soon then I'll of course pay attention to that. I will repeat, I'll use best evidence available.

So, as you´ve said, it´s a mix; regarding your first paragraph about listening to loudspeakers first; what about the fallability, that Thomas Savage argued with?
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
Certainty? Not sure. :)
It´s more a probability, means in the long run, everyone is most likely sometimes wrong, but always? Certainly not. There are of course people, who are wrong nearly everytime. Reminds me to a story told in one of the fundamental textbooks about sensory evaluation, describing a longtime participant in tests, who was always sure there was no difference, but his data showed that he detected one everytime in around 20 tests.



Might be, as said before, "audiophiles" are as heterogenous as other groups as well, but imo the main problem is the generalization of any issue and the jumping to conclusions about the causalities.

But back to the decision strategy; stick to measurements is at first glance a good advice, but if there is the choice between two (or more) alternatives, all departing from the ideal to a similar degree but in different ways, it gets difficult.



Unfortunately i don´t understand the "Johnny Morris hint".....



So, it´s more a mix depending on more than one variable?
Well, given we are a faulty unreliable measuring device and can't known when we are right or wrong best avoid the trouble and stick to objective measurements that take the human out of the equation.

Unless there is no better solution..

It's not particularly complicated, I'm mystified as to why you insist on trying to make it so.

We are all human, so you can certainly 'generalise' to that effect.

Determining fidelity and determining preference are to different things. Ones a value statement and the other a opinion.

Without a objective correlation to ones subjective assertions we simply can't know if we are right. You can prefer something but even that, you might change your mind later but maybe not because you have ' convinced' yourself of your preference so have lost objectivity.

It's not what we hear that's a problem , it's how we chose to relate to it and the value and significance we assign to what we hear.. that interpretation is not a objective process and the conclusions are unreliable as they incorporate our imaginative self and all sorts of faulty memories etc.

That's just being human.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
I was referring to the measured problems over the decades- this some threads in this forum and in other sources as well - with DACs
How were the measurements made? Presumably with ADCs - based on the same technology. If we don't believe our DACs are right, why should we believe our ADCs?

Of course there will always be errors with ADCs and DACs. Just keep zooming in, and you will find them - you don't even need to make any measurements because you can calculate what the best case errors are at 16 bits, 24 bits, 32 bits, 64 bits. When you 'normalise' them for display on a page, they always look the same: huge. Only the small print on the axes tells you how big they really are, but no one actually knows what those numbers mean. People stopped being able to hear the errors at about -80dB and they are now just abstract numbers. No matter how accurate digital audio is, people will always be able to claim it has errors - because it does.
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
What is your point here Jacob, you seem to argue with negatives and nebulous questions but never ever ever seem to assert your own position with any kind of positive affirming assertion.

I'm not sure members will carry on showing much intrest in this type of discussion dynamic.
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
No, it is just a question of probabilities...

Maybe my pov is different, as we have in germany a legally garantueed money back choice since the invention of online commerce (in the case of B2C). Reasoning was that no evaluation at the dealers place before buying is available, so customers were given the chance for evaluation at home. Given for 14 days but a lot of dealers expand that to 30 days.


Yes, probabilities and standalone preferences that do not in any way indicate a 'better' system or any applicability outside the experiment. I am left wondering what the point of them is!

The applicability outside the experiment depends strongly on the specific experimental conditions. ;)

As said before, i don´t share the argument that only "unsighted listening" is able to provide useful results; the people producing the music don´t rely on "unsighted listening" and nobody i know does only "unsighted listening" for evaluation purposes. Controlled listening tests are nevertheless important for various reasons and within that is the chance to get corrobation for sighted impressions.
 

Don Hills

Addicted to Fun and Learning
Joined
Mar 1, 2016
Messages
708
Likes
464
Location
Wellington, New Zealand
... Controlled listening tests are nevertheless important for various reasons and within that is the chance to get corrobation for sighted impressions.

Or, just as likely, refutation.
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
@ThomasSavage,

the "fallability argument" is something i´ve quite frequently heard in forum discussions over the years and imo there are some problems/inconsistencies obvious and so i´m trying to find out, which reasoning is given for the various decision strategies.

What else should i do, if asking is not appropriate?

Furthermore, the "first one who said semantics" wins, we had already; is "nebulous questions" an expansion of that? :)
If it´s not just part of the game, please be more specific about which questions are "nebulous" and i´ll try to explain......

@Don Hills,

of course....
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
As said before, i don´t share the argument that only "unsighted listening" is able to provide useful results; the people producing the music don´t rely on "unsighted listening" and nobody i know does only "unsighted listening" for evaluation purposes. Controlled listening tests are nevertheless important for various reasons and within that is the chance to get corrobation for sighted impressions.
You may not have seen earlier discussions where I said exactly the same things - I am basically a DBT sceptic. My stance is that an audio system is a relatively simple, human-defined, designed and manufactured system, not a natural system that we should observe scientifically. I'm not even all that excited by measurements, which I see as trying to bolt the stable door after the horse has bolted. For me, as long as the system is designed to meet the definition of an amplifier, DAC, speaker, etc. and the measurements confirm it is working properly, the job is done. After that, it is perfectly reasonable to use one's ears to tweak any room-dependent settings and shuffle the speakers around, etc. (But I do not hold with room correction, except maybe for the bass). It is also perfectly reasonable to use one's ears to decide whether you need bigger speakers, omnidirectional speakers and so on.

It's the best of all worlds: only a small repertoire of technology meets the definition, helping in the neuroticism-reduction stakes, and one can be perfectly relaxed about subjective impressions...:)
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
You may not have seen earlier discussions where I said exactly the same things - I am basically a DBT sceptic. My stance is that an audio system is a relatively simple, human-defined, designed and manufactured system, not a natural system that we should observe scientifically. I'm not even all that excited by measurements, which I see as trying to bolt the stable door after the horse has bolted. For me, as long as the system is designed to meet the definition of an amplifier, DAC, speaker, etc. and the measurements confirm it is working properly, the job is done. After that, it is perfectly reasonable to use one's ears to tweak any room-dependent settings and shuffle the speakers around, etc. (But I do not hold with room correction, except maybe for the bass). It is also perfectly reasonable to use one's ears to decide whether you need bigger speakers, omnidirectional speakers and so on.

It's the best of all worlds: only a small repertoire of technology meets the definition, helping in the neuroticism-reduction stakes, and one can be perfectly relaxed about subjective impressions...:)
This is a fine strategy for staying sane in audio and not spending your retirement fund.
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
Interesting that mainly to address the "fallability" argument is carefully avoided.... :)

Could it be that is mostly be used as a knock-out argument in the case of "damned audiophile gadgets/tweaks?

Obviously the posters in this thread seem to follow a more pragmatic approach which reflects what people do in all other areas of everyday life where also decisions have to be made. Reading the often narrow or dogmatic statements in audio forums, one could get the impression that the audio field is so special that people aren´t able to get anything done.

Fallability of human senses is something to keep in mind, but that´s also true for the other parts of our life.

@Cosmik,

that digital audio (and analog audio as well) presumably never works perfect is obviously true - Shannon afair had already mentioned the constraints of sampling in his papers - but the "at -80 dB" statement still isn´t reflecting the reality. We discussed it before, the whole dither solution shows that it sometimes matters what happens at <= -80 dBFS (theoretically it is even somewhere around -86 dBFS).
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
Interesting that mainly to address the "fallability" argument is carefully avoided.... :)

Could it be that is mostly be used as a knock-out argument in the case of "damned audiophile gadgets/tweaks?

Obviously the posters in this thread seem to follow a more pragmatic approach which reflects what people do in all other areas of everyday life where also decisions have to be made. Reading the often narrow or dogmatic statements in audio forums, one could get the impression that the audio field is so special that people aren´t able to get anything done.

Fallability of human senses is something to keep in mind, but that´s also true for the other parts of our life.

@Cosmik,

that digital audio (and analog audio as well) presumably never works perfect is obviously true - Shannon afair had already mentioned the constraints of sampling in his papers - but the "at -80 dB" statement still isn´t reflecting the reality. We discussed it before, the whole dither solution shows that it sometimes matters what happens at <= -80 dBFS (theoretically it is even somewhere around -86 dBFS).
It's how you chose to relate consciously to the information your auditory sense have relayed that's the issue, along with the chosen significance you might attach to those impressions.

If you could stick a probe into the brain and get the 'raw' data then that would be fine.. before our egotistical conscious 'selfs' get involved.

We corrupt things, fidelity in audio in terms of maintaining a original signal is not a POV thing. It's not about wonderings and imagining.. sitting down and listening to cables and such.

It's just about reproduction of a signal, keeping as faithful as possible, low IMD etc..

Preference, well that's down to you.. you want to prefer something then best listen and make up your mind just don't start using those assumptions and asserting a universal value out of them.. that would be wrong.

I like Beverly Hills cop .....,

Beverly Hills cop is a great film .....,

These are two different statements.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
I was reading a bit of Karl Popper last night while eating my tea - that's the live-life-at-the-edge kind of guy I am.

He was drawing a distinction between Einstein's theory of relativity and Freud's theories, and defining pseudoscience as anything where the experiments are designed to confirm a theory rather than disconfirm it. It occurs to me that in order to practise pseudoscience, you at least need to have a theory (or hypothesis) to disconfirm!

In these listening tests that are designed merely to "establish whether there is a preference", what is the theory or hypothesis that the experiment is hoping to confirm or disconfirm? If you establish a preference for a certain digital cable, what does that get you? You haven't found the reason why there is a preference, so you can't extrapolate anything from it. It doesn't help you to make a better cable, because you haven't even established what it is that caused the preference. If you can't even imagine how a digital cable could improve the sound, (and the rest of your experiment is constructed from similar cables!), you are just groping about in the dark. If you do have some superstition theory about noise and grounding or whatever, you need to do experiments with noise and grounding, not messing about with random cables. This general non-scientific-ness of listening tests extends to the blather talked about MQA, etc., etc.

Making up a theory (e.g. introducing aliasing for supposed better timing) and then running listening tests does not demonstrate that the theory is correct. It cannot be extrapolated to anyone else's ears/music/room/equipment even if the particular experiment purports to demonstrate a supposed preference.

Basically, as a potential purchaser, the external evidence of someone else's supposed listening test is worthless unless it homes in on specifics very convincingly indeed. If it makes the extraordinary claim that a deviation from 'high fidelity' is preferable, it needs some extraordinary evidence.

At the end of the day, the question is: do you want 'hi-fi' that meets a definition with measurable specifications, or do you want something *undefined* that some bloke tells you people prefer listening to, with an explanation that is no better than astrology?
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
It's how you chose to relate consciously to the information your auditory sense have relayed that's the issue, along with the chosen significance you might attach to those impressions.

If you could stick a probe into the brain and get the 'raw' data then that would be fine.. before our egotistical conscious 'selfs' get involved.

Yeah...., but...., as we know that´s a fundamental issue and by now we should know that and keep it in mind.

We corrupt things, fidelity in audio in terms of maintaining a original signal is not a POV thing. It's not about wonderings and imagining.. sitting down and listening to cables and such.

It's just about reproduction of a signal, keeping as faithful as possible, low IMD etc..

The starting point of our discussion was a bit more specific; your argument was explicitely the "fallability" of human listening and the conclusion that interestend consumers/listeners should not use perceptual evaluation in case that any external reference is missing.

I´d say that the judgement if a certain set of measured data really ensures something like "audio transparency" is even more challenging and might suffer from all sorts of "fallabilities".
As said before, especially if measurements - see for example the threads in this forums about dacs and digital "gadgets" - show that problems might occur that aren´t (at least often) predictable by the published technical data of the units.

Preference, well that's down to you.. you want to prefer something then best listen and make up your mind just don't start using those assumptions and asserting a universal value out of them.. that would be wrong.

I totally agree, but would expand it further, asserting universal/categorical validity is often quite dangerous or another kind of "fallability".
 
OP
J

Jakob1863

Addicted to Fun and Learning
Joined
Jul 21, 2016
Messages
573
Likes
155
Location
Germany
I was reading a bit of Karl Popper last night while eating my tea - that's the live-life-at-the-edge kind of guy I am.

He was drawing a distinction between Einstein's theory of relativity and Freud's theories, and defining pseudoscience as anything where the experiments are designed to confirm a theory rather than disconfirm it. It occurs to me that in order to practise pseudoscience, you at least need to have a theory (or hypothesis) to disconfirm!

Poppers falsification approach and in a broader sense a discussion of the philosophy of science would be worth a new thread.

In these listening tests that are designed merely to "establish whether there is a preference", what is the theory or hypothesis that the experiment is hoping to confirm or disconfirm?

The specific hypothesises are a seperate question, but the underlying theory is often that - see those discussions around "audiophile´s stuff" - establishing a preference can´t happen, as according to the known thresholds of hearing, no difference could be detected.
Brief insert: establishing a preference is meant under the conditions of a controlled listening experiment, that incorporates propper randomization. If under theses conditions a preference is established, then a difference must exist, within the constraints of the construct, means probabilities etc.
As said before, often these tests are directional tests (although calling for a preference answer), so are discrimination tests.
And of course it depends on asking a single participant or a group, as a group may have segmented preferences at the same level, which might lead to wrong conclusions.

Maybe all that is worth a new thread about sensory testing in general too.

Another aim is to find out, what consumers prefer to justify developments overall or specific alterations of products.
See for example the question if it is worth to think about two channel stereophonic reproduction instead of monophonic systems.


If you establish a preference for a certain digital cable, what does that get you? You haven't found the reason why there is a preference, so you can't extrapolate anything from it. It doesn't help you to make a better cable, because you haven't even established what it is that caused the preference. If you can't even imagine how a digital cable could improve the sound, (and the rest of your experiment is constructed from similar cables!), you are just groping about in the dark. If you do have some superstition theory about noise and grounding or whatever, you need to do experiments with noise and grounding, not messing about with random cables. This general non-scientific-ness of listening tests extends to the blather talked about MQA, etc., etc.

Talking about model constructs. Listening tests are in a way, despite the artificiality, a real life test of the model validity, if there is a theory about the reasons.
If there is not a theory, positive results from controlled listening tests are providing evidence for something, that, according to another theory, is impossible which for itself is imo worth the effort.
Development of a theoretical framework might follow.

Making up a theory (e.g. introducing aliasing for supposed better timing) and then running listening tests does not demonstrate that the theory is correct. It cannot be extrapolated to anyone else's ears/music/room/equipment even if the particular experiment purports to demonstrate a supposed preference.

It depends.....

Basically, as a potential purchaser, the external evidence of someone else's supposed listening test is worthless unless it homes in on specifics very convincingly indeed. If it makes the extraordinary claim that a deviation from 'high fidelity' is preferable, it needs some extraordinary evidence.

At the end of the day, the question is: do you want 'hi-fi' that meets a definition with measurable specifications, or do you want something *undefined* that some bloke tells you people prefer listening to, with an explanation that is no better than astrology?

At that point we are running in cirles; measurements can´t tell anything about the audibility because audibility is bound to the perception of a human listener.
If, as you´ve argueed, there is no possibility of extrapolation from listening tests to other "ears/music/room/equipment" there is no hope. :)
Even if you restrict your critic to preference tests, it would still hold for other tests as well, especially the point of "other ears".

As said before, it often depends on majority decisions, and the goal of small sample tests is to estimate parameters of the underlying population, as testing the whole population is rarely possible.
 
Last edited:

Old Listener

Senior Member
Joined
Apr 28, 2016
Messages
499
Likes
556
Location
SF Bay Area, California
I make rare buying decisions and expect to keep and use gear for a long time. When I'm in the market for something, I collect all the information that I can find so that I can make a well informed decision with a high probability of success. My information sources in roughly the order I value them:

- Full specs from the manufacturer. If a company won't describe their product fully, I am much less likely to buy their product. They know the details and can document them much more easily than I discover and document those details.

- Results of properly conducted tests and measurements. It is nice when a manufacturer provides such information. It is even better when an independent source supplies confirming test results. I don't have test and measurement gear so I'm dependent on others for high precision measurements. (Speaker frequency response at low frequencies I can do.)

- Reviews with only sighted listening tests. These can be useful in pointing out flaws and shortcomings. The potential for error in sighted tests has been well documented over several decades.

- Reports by people who have purchased and used the product. There are obvious problems with such reports but sometimes that's all you have. Sometimes user reports can point out real life problems such as the firmware and smartphone control app flaws in the KEF50W speakers.

Sometimes I can get all this information. That's lovely. Other times, I have to make a decision with partial information. That's life.

Trying out a product in a high-end audio dealer's store isn't on my list anymore. Dealers don't carry the products that interest me and the last time I visited a dealer to listen to a DAC, the salesman put on an over-the-top demonstration of his incompetence. For years I have bought audio gear through the web. I am happy to have a return privilege and have used it a few times. However, I much prefer to do my best to make the right choice before I place an order.

I am very grateful to Amir and to Archimago for collecting and publishing useful measurements of DACs. Results for speakers are also welcome. I look forward to seeing test results for the JBL 705P and 708P speakers. Thanks to everyone who is supplying information that helps me make good decisions.
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
3,075
Likes
2,180
Location
UK
The specific hypothesises are a seperate question, but the underlying theory is often that - see those discussions around "audiophile´s stuff" - establishing a preference can´t happen, as according to the known thresholds of hearing, no difference could be detected.
Brief insert: establishing a preference is meant under the conditions of a controlled listening experiment, that incorporates propper randomization. If under theses conditions a preference is established, then a difference must exist, within the constraints of the construct, means probabilities etc.
If the effect of a digital cable is the item in question, but we don't understand how it could possibly introduce a difference, surely that means that we don't understand how the rest of the experimental apparatus works, either..? It is no good pointing to the switching mechanism and saying that it is truly random, etc. because it may also contain these mysterious effects that shouldn't be there. At some point, you have to draw a line and say "We understand how a wire works in the context of where we are using it". Otherwise, we don't understand anything.
 
Top Bottom