Limitations of blind testing procedures

[Jinjuku]

I agree with much of that, but I feel myself being gently nudged away from the real issue (again!). I didn't mean that we may lose all of our discerning ability when under stress, just a proportion of it - maybe just the bit at the outer edges that can discern between 16 and 24 bits (as an audiophile would claim).

And I would say that sighted 'evaluation' swings the pendulum grossly in the other direction for describing what isn't actually there.

 

[Jinjuku]

You are homing in on a very specific application of listening tests: debunking claims made by probably deluded people. I think the thread is more general than that.

You have to start somewhere as in most experimentation.

 

[March Audio]

I agree with much of that, but I feel myself being gently nudged away from the real issue (again!). I didn't mean that we may lose all of our discerning ability when under stress, just a proportion of it - maybe just the bit at the outer edges that can discern between 16 and 24 bits (as an audiophile would claim).

And only a few aspects of audiophile listening ability are functions of straightforward physiology that can be confirmed with clicks and bleeps. Much of the reconstruction of the 'audio scene' takes place in the brain.

Take stereo imaging. What is it? Is it real? Can it be measured? We think we hear stereo imaging - which is mainly why we bother with two speakers. But what conditions are necessary for 'good' imaging? We could embark on a series of empirical experiments to try to work it out - but if we have trouble defining what the word means, and it only takes place inside our heads, we are already on non-scientific, aesthetic judgement territory. It probably won't end well.

Mr. Putzeys achieves brilliant imaging with his Kii Threes, apparently. Did he perform thousands of listening tests to get there? Or did he just build a system that is closer to the formal definition of stereo than anyone else's? Ultimately, do we really think that a listening test-based approach would come up with something different from this? If so, it wouldn't actually be a stereo hifi system at all, but an effects box.

I don't think that stereo imaging is as mysterious as you imply. "tricks" are used all the time by producers to create imaging. The acoustics of your room and the speaker time coherence are the major players on the replay side.

Blind testing can confirm this.

 

[Jinjuku]

I don't think that stereo imaging is as mysterious as you imply. "tricks" are used all the time by producers to create imaging. The acoustics of your room and the speaker time coherence are the major players on the replay side.

Blind testing can confirm this.

Actually it's all 'tricks' and mathematically defined at that.

 

[amirm]

I think there are two goals in us discussing blind tests:

1. Show the other side they are wrong. This is the "social" aspect I was talking about.

2. Show that a system is fully transparent for all people, all music and all conditions.

I think we all agree that #1 is our primary goal usually given how many times we hear that a "veil has been removed [yet again]" and "night and day difference."

My goal is usually #2 in that when I say something is transparent, I like it to be that way. This one does require that the test give all the help it can in finding differences. Without maximizing this chance, we are fooling ourselves as it is very easy to get negative results.

 

[Jinjuku]

I think there are two goals in us discussing blind tests:

1. Show the other side they are wrong. This is the "social" aspect I was talking about.

2. Show that a system is fully transparent for all people, all music and all conditions.

I think we all agree that #1 is our primary goal usually given how many times we hear that a "veil has been removed [yet again]" and "night and day difference."

My goal is usually #2 in that when I say something is transparent, I like it to be that way. This one does require that the test give all the help it can in finding differences. Without maximizing this chance, we are fooling ourselves as it is very easy to get negative results.

My goal is to let the results speak for themselves. I am stipulating under what conditions I would be willing to be wrong. I can't say the same for the subjectivists. I think most of them lack any intellectual humility.

 

[Cosmik]

I don't think that stereo imaging is as mysterious as you imply. "tricks" are used all the time by producers to create imaging. The acoustics of your room and the speaker time coherence are the major players on the replay side.

I am not implying that imaging is mysterious. I am taking the opposite view: that whatever it is (I have my own definition for it which may not be the same as yours), it is simply a function of 'stereo'. My educated guess is that it falls out of coherency between left and right, but also is improved by absolute accuracy in all respects. Controversially, this means that I think that low phase distortion at the driver level is a factor, too. Like I said: stereo.

But, there are people who swear that their vinyl images really well, or that (their version of) imaging is better in mono.

Blind testing can confirm this.

You would first have to define what "imaging" was. For some it will be precision of a single source left to right; for others it will be front-to-back; and others will be listening for 'spaciousness'. You could never be sure they were listening for your version of "imaging". Maybe they think that "good image" is the equivalent of "vivid".

What would the blind testing consist of? Would you be asking the listeners to give a figure between one and ten? Ask them to turn a wheel to point to 'the voice'? Rank various choices in terms of preference?

As I said previously, if your testing confirms that 'stereo' gives the best imaging, you will merely have confirmed something very obvious - so why bother doing the test? If your testing finds something different, you will be advocating something that is not stereo or hifi. I would seriously doubt that it would be universally repeatable and we would have to look very seriously at 'where you had gone wrong'!

 

[Blumlein 88]

@Cosmik

The existence and effectiveness of Qsound shows that someone has figured out imaging. It is not public what is the algorithm, but nevertheless they can manipulate imaging with precision.

You are making the assumption that accuracy will result in best perceived imaging. The processing of something like Qsound among other things would indicate that is perhaps not the case.

In formal testing of simple stereo as well as surround microphone techniques people listened to recordings and indicated where various instruments were located. The accuracy of that was tabulated for various techniques. It has been done a few different ways. Some also asked for general perceptions like apparent hall size or a sense of space. In some of those for stereo a blumlein technique was found most accurate at placing instruments. In some others it was judged second best using slightly different methodology. In surround techniques there was less consensus and accurate placement seemed to be a trade off for other aspects. Plus some tests were defining their goals differently. None of which means you can't test for this. Some of it is of course preference which I think you are mixing up with absolute accuracy. One thing also apparent is when given instructions about what each tester used for imaging definitions, listeners were able to adapt. They weren't left to make up their own definition.

I rather prefer the second approach Amir listed above:
2. Show that a system is fully transparent for all people, all music and all conditions.

So for instance in the testing done for 'stereo' effect I prefer those that asked for placing instruments into their position vs where they were to those that asked which people found more enjoyable. You need to do that first before working on general preferences. And despite general preferences there will always be outliers in preference.

 

[fas42]

Does it????

Well in that case listen to a crap hifi for a few minutes and you wont be able to tell the difference between it and the worlds best hifi. Save yourself a fortune. No need to buy expensive (good kit - not that it follows that it should be expensive to be good). Just fool your brain into not knowing the difference.

...mmmmm....something's wrong with that conclusion

Ummm ... no need to go silly - every time you hop into a car and enjoy a piece of music you know well on the standard kit there this is what happens - the brain "fills the gaps".

Which is great, when you do know what's supposed to be there - a good test is to listen to a piece you don't know, in a style you don't like - my wife can't stand female opera voices on ordinary systems, the screechiness irritates her intensely; it takes a pretty good setup to make her want to listen to such.

 

[Dynamix]

Meh.

 

[fas42]

You would first have to define what "imaging" was. For some it will be precision of a single source left to right; for others it will be front-to-back; and others will be listening for 'spaciousness'. You could never be sure they were listening for your version of "imaging". Maybe they think that "good image" is the equivalent of "vivid".

Imaging is the result of the distortion and anomalies being sufficiently attenuated to the point that the information in the recording starts to dominate what you are aware of, acoustically. When of a high enough standard then the speakers completely "disappear", the typical audiophile "sweet spot" is the starting point of getting this to happen. Anything in the sound field that draws your attention to the speaker drivers damages this, and the "imaging" is much poorer.

Which makes the process of refining, and evaluating systems very easy - is one getting closer to, or further from, this listening experience? One could even 'measure' it, by the distance one can move from a perfectly central position, before the imaging "fails".

 

[Jinjuku]

Imaging is the result of the distortion and anomalies being sufficiently attenuated to the point that the information in the recording starts to dominate what you are aware of, acoustically. When of a high enough standard then the speakers completely "disappear", the typical audiophile "sweet spot" is the starting point of getting this to happen. Anything in the sound field that draws your attention to the speaker drivers damages this, and the "imaging" is much poorer.
.

Jesus wept. Imaging is a result of the mastering process and your room and your speakers.

 

[fas42]

Jesus wept. Imaging is a result of the mastering process and your room and your speakers.

No. I have done this endless times over the years, with various bits of kits, in all sorts of rooms. The quality of the playback dictates what one hears, in the subjective sense - I've given up listening to 'correct' systems of others with with my test recordings, because they are always so limited in what they present - I have no interest in hearing "tiny" reproduction ...

 

[Jinjuku]

No. I have done this endless times over the years, with various bits of kits, in all sorts of rooms. The quality of the playback dictates what one hears, in the subjective sense - I've given up listening to 'correct' systems of others with with my test recordings, because they are always so limited in what they present - I have no interest in hearing "tiny" reproduction ...

Thank for confirming you don't know what the hell you are talking about.

 

[March Audio]

I am not implying that imaging is mysterious. I am taking the opposite view: that whatever it is (I have my own definition for it which may not be the same as yours), it is simply a function of 'stereo'. My educated guess is that it falls out of coherency between left and right, but also is improved by absolute accuracy in all respects. Controversially, this means that I think that low phase distortion at the driver level is a factor, too. Like I said: stereo.

But, there are people who swear that their vinyl images really well, or that (their version of) imaging is better in mono.


You would first have to define what "imaging" was. For some it will be precision of a single source left to right; for others it will be front-to-back; and others will be listening for 'spaciousness'. You could never be sure they were listening for your version of "imaging". Maybe they think that "good image" is the equivalent of "vivid".

What would the blind testing consist of? Would you be asking the listeners to give a figure between one and ten? Ask them to turn a wheel to point to 'the voice'? Rank various choices in terms of preference?

As I said previously, if your testing confirms that 'stereo' gives the best imaging, you will merely have confirmed something very obvious - so why bother doing the test? If your testing finds something different, you will be advocating something that is not stereo or hifi. I would seriously doubt that it would be universally repeatable and we would have to look very seriously at 'where you had gone wrong'!

I think we are in agreement for the most part but I think we do know what imaging is. However unless you have a specifically defined acoustic environment you aRe not going to be able to define instrument x should appear in coordinates x,y,z.

Play with acourate and you will discover how imporyant time coherence is for the imaging.

As has been mentioned, q sound.

 

[March Audio]

Imaging is the result of the distortion and anomalies being sufficiently attenuated to the point that the information in the recording starts to dominate what you are aware of, acoustically. When of a high enough standard then the speakers completely "disappear", the typical audiophile "sweet spot" is the starting point of getting this to happen. Anything in the sound field that draws your attention to the speaker drivers damages this, and the "imaging" is much poorer.

Which makes the process of refining, and evaluating systems very easy - is one getting closer to, or further from, this listening experience? One could even 'measure' it, by the distance one can move from a perfectly central position, before the imaging "fails".

Sorry fas thst is wrong. Do you even know what you mean by "distortion"?

 

[Thomas savage]

Jesus wept. Imaging is a result of the mastering process and your room and your speakers.

Largely i have to agree, iv taken my system to other locations and main thing thats diffrent is the stereo image more persifically the front to back depth.

My room limits this it would seem. However i can still change the image characteristics by moving my speakers about in my room... wider larger and smaller but more definded are the main options.

Fa to do with mystery 'distortions' ..

 

[fas42]

Sorry fas thst is wrong. Do you even know what you mean by "distortion"?

I'm talking of the distortion generated by the playback system - this is easy to hear with most systems by moving your head close to the tweeter on one side, while system playback is at elevated levels - if this has a harsh, spitty, irksome, unpleasant, "distorted" quality, especially with 'challenging' recordings, well, that's distortion. That's something that goes away, when the system is in good shape ...

 

[Cosmik]

@Cosmik

The existence and effectiveness of Qsound shows that someone has figured out imaging. It is not public what is the algorithm, but nevertheless they can manipulate imaging with precision.

You are making the assumption that accuracy will result in best perceived imaging. The processing of something like Qsound among other things would indicate that is perhaps not the case.

QSound is a recording technique, but we are concerned solely with playback. Sure, you can do whatever you like with DSP and turn a mono recording into pseudo-stereo if you want, but if you want to sell it to other people, you have to assume that they are going to play it back sans DSP manipulation. Straightforward accuracy in this will ensure the best imaging of what is in the recording.

We, as listeners, could have a special hobby where we took existing recordings and 're-mixed' them to our own taste - but of course without access to the original multitrack tapes (and possibly not even then, depending how they have been created) we would only have crude control of the sound. Some people even like to press the 'hall effect' button on their home theatre amplifier or car stereo now and again, apparently - I've never quite understood what this is for, myself.

But this is a distraction. We are still at the point of not being able to say that people don't lose (some) of their audio discerning abilities when they know they are taking part in an experiment. As long as this is true, audiophiles will never believe what your tests show (and, possibly, nor should they). The consensus seems to be that it is "highly unlikely" that we lose our listening abilities, but without any attempt to demonstrate whether it is true or not we cannot know.

And after a hundred years of listening tests (or whatever), we are at the stage where audio systems are still slowly progressing towards being 'straight' i.e. linear in all respects. This could have been worked out on day one (and probably was), with only a technology shortfall preventing it. We didn't need listening tests at all, in other words.

 

[March Audio]

I'm talking of the distortion generated by the playback system - this is easy to hear with most systems by moving your head close to the tweeter on one side, while system playback is at elevated levels - if this has a harsh, spitty, irksome, unpleasant, "distorted" quality, especially with 'challenging' recordings, well, that's distortion. That's something that goes away, when the system is in good shape ...

No idea what you are talKing about fas and that has nothing to do with imaging.