Any correlation between measurements and perceived sound quality?

[JustIntonation]

I'll take a stab at it.

These thresholds vary between individuals and are also dependent on training of that individual.

There are those that believe blind tests are flawed and only long term familiarity with systems can give meaningful results.
When one has that opinion start worrying and read this post again.

Well certainly blind tests are flawed to a very large degree when it comes to establishing the audibility threshold of certain types of distortion and jitter etc. If only for the reason that all playback systems are severly flawed!
One would need a truly fully anechoic room with drivers, cabinets, amps, crossovers, DACs etc that are flawless to perform truly accurate tests. None of which exist other than the anechoic rooms. And worst of all are headphones. One shouldn't do any serious testing with any headphone yet headphones are most often used for exactly these types of (pseudo) "scientific" tests.
I'm not saying these tests are meaningless but their results are certainly not applicable directly to real high-end audio.

As for long term familiarity. Well first you say threshholds are dependent on training of an individual and then you say people should start worrying when they think long term familiarity is better than a blind test. These statements are fairly in conflict with eachother.
Long term familiarity is nothing else than a form of ear/brain training. I've often found myself liking a system at first yet after more time I begin to identify and pick upon it's faults and things that bother me. This is very common.

I do think blind tests are very meaningful for personally comparing differences in a certain system. But these blind tests will often improve in quality after more familiarity. And I think results from such blind tests are not at all guaranteed to translate to other systems. Sometimes they will, sometimes they won't.

I think the above is a realistic description of reality in audio. And to me it means I hold measurements in very high regard. I also take into consideration thresholds found by others as valuable but not as scientific fact applicable to my personal system and hearing.
This leads me to value using my own hearing as the final arbitrator as to what differences I can and cannot hear in my system, with all the pitfalls this brings (when a proper quick ABX blind test is not practical). To me in my systems and with my ears this has often lead to hearing what are to me important differences between DACs and USB interfaces that others may say shouldn't matter much. But to me it does. And yes, these differences are often clearer to me with time/familiarity.

 

[SIY]

Huh, funny, when I did a double blind test selecting original versus MP3s encoded at different rates, and put them in correct order of rate, the guy running the test had me use headphones. I didn’t know it wouldn’t work and was pseudoscience. If I had, perhaps I couldn’t have done that.

Peeking definitely increases the rate of positive identifications!

 

[JustIntonation]

Huh, funny, when I did a double blind test selecting original versus MP3s encoded at different rates, and put them in correct order of rate, the guy running the test had me use headphones. I didn’t know it wouldn’t work and was pseudoscience. If I had, perhaps I couldn’t have done that.

Peeking definitely increases the rate of positive identifications!

What I meant is that headphones have way more trouble accurately reproducing audio than speakers do. First of all you'll never get the HRTF (head related transfer function) right with headphones meaning you'll also never get anything resembling a flat frequency response or natural representation of the audio. Other than that a single driver has many issues covering the whole audio range, especially regarding frequency response/ringing etc. Transparent critical treble is non existant in headphones when compared to a great dedicated tweeter in speakers (in a great treated room). Etc etc.
So good for you if you heard differencs between mp3 bitrates on headphones. But I meant it the other way around ofcourse. That when you stop hearing differences on headphones this doesn't mean those differences can't still be audible on a great speaker/room system. To use headphones for testing such audibility thresholds and then to state the tresholds found can be applied in general would be pseudo science in my opinion.

 

[andreasmaaan]

One would need a truly fully anechoic room with drivers, cabinets, amps, crossovers, DACs etc that are flawless to perform truly accurate tests. None of which exist other than the anechoic rooms. And worst of all are headphones. One shouldn't do any serious testing with any headphone yet headphones are most often used for exactly these types of (pseudo) "scientific" tests.

You're being too general here IMO. For some forms of distortion (e.g. nonlinear distortion), audibility thresholds tend to be lower when the test system uses speakers, while for others (e.g. group delay) headphones tend to be more revealing.

As for long term familiarity. Well first you say threshholds are dependent on training of an individual and then you say people should start worrying when they think long term familiarity is better than a blind test. These statements are fairly in conflict with eachother.
Long term familiarity is nothing else than a form of ear/brain training.

Yes and no IMHO. Long term familiarity may provide training, or it may provide the exact opposite: conditioning. It's certainly no substitute for the type of training in which a single parameter is varied, giving the subject the opportunity to identify its effect on the sound.

 

[SIY]

A common tactic of proponents of peeking is to object that there is a single way to design all audibility experiments. And of course that one way is bizarrely impractical.

 

[JJB70]

Anechoic chambers are awful things, I spent a bit of time in one doing certain tests on some equipment to be installed on board warships and it felt very odd, quite disconcerting.

 

[JustIntonation]

Anechoic chambers are awful things, I spent a bit of time in one doing certain tests on some equipment to be installed on board warships and it felt very odd, quite disconcerting.

I spent almost a year in a semi-anechoic room. Only the floor was not anechoic.
I liked it! Extremely quiet and music is a completely different experience in one. I'm not a religious man but the experience of music in a (semi) anechoic room can be described as such It's magic! (though trust me, you'll hear every flaw in a speaker to a fault)
But two of my friends who had a listen in my room had a panic attack! They got very dizzy and racing heart etc. I read this happens with some people where the brain can't make sense of seeing the room but not hearing it.

Btw are you sure you were in a good enough anechoic room?
Often people call a poorly treated room an anechoic room while it's only anechoic starting somehwere in the mids up. These are truly horrible rooms, the bass and often lower mids will still resonate/reverb in the room while all the highs are cut. This gives the sensation of pressure on the ears and a sensation of having your ears clogged etc, very very unpleasant.
A good anechoic room has very thick absorption material of a low gas flow resistance. Some 10cm rigid fibreglass boards with some eggfoam on top of it will give horrible results. Often people have an experience with such a badly done anechoic room. Well done rooms are much rarer.

edit: uhh if the room you were in was an actual very big army anechoic room them it was probably one of the best anechoic rooms around Yes it's not for everybody. But if you're ever in such a room again bring speakers, even portable speakers will sound crazy in such a room

 

[JJB70]

I spent almost a year in a semi-anechoic room. Only the floor was not anechoic.
I liked it! Extremely quiet and music is a completely different experience in one. I'm not a religious man but the experience of music in a (semi) anechoic room can be described as such It's magic! (though trust me, you'll hear every flaw in a speaker to a fault)
But two of my friends who had a listen in my room had a panic attack! They got very dizzy and racing heart etc. I read this happens with some people where the brain can't make sense of seeing the room but not hearing it.

Btw are you sure you were in a good enough anechoic room?
Often people call a poorly treated room an anechoic room while it's only anechoic starting somehwere in the mids up. These are truly horrible rooms, the bass and often lower mids will still resonate/reverb in the room while all the highs are cut. This gives the sensation of pressure on the ears and a sensation of having your ears clogged etc, very very unpleasant.
A good anechoic room has very thick absorption material of a low gas flow resistance. Some 10cm rigid fibreglass boards with some eggfoam on top of it will give horrible results. Often people have an experience with such a badly done anechoic room. Well done rooms are much rarer.

edit: uhh if the room you were in was an actual very big army anechoic room them it was probably one of the best anechoic rooms around Yes it's not for everybody. But if you're ever in such a room again bring speakers, even portable speakers will sound crazy in such a room

This was a full anechoic room, it was used for testing and measuring some extremely expensive equipment for military users and I never asked how much it cost. The feeling was very odd and almost vertigo inducing, not my cup of tea I am afraid.

 

[JustIntonation]

You're being too general here IMO. For some forms of distortion (e.g. nonlinear distortion), audibility thresholds tend to be lower when the test system uses speakers, while for others (e.g. group delay) headphones tend to be more revealing.

Yes I was being general indeed.
But as for headphones vs speakers in an anechoic room. I don't think there's any area at all that speakers won't beat headphones.
Even for group delay is my guess, though that would be the only area a single driver has an advantage, a multi driver speaker can be made to be flat in phase at the expense of some pre-ringing and my guess would be that all the other advantages of a speaker in an anechoic room would by far allow more critical listening also to phase.

Yes and no IMHO. Long term familiarity may provide training, or it may provide the exact opposite: conditioning. It's certainly no substitute for the type of training in which a single parameter is varied, giving the subject the opportunity to identify its effect on the sound.

Yes I follow you here. But still, with familiarity and learning how to listen "into" a piece of music and what to listen for both good and bad has value to me. Not only to learn what to listen for, but also for setting preference of what sounds "good". If presented for the first time with 2 different sounds not heard before it's not only harder to pick out the differences but also to say which one is prefered.
Couple this with how tests are often done, a group of people which have to put on a headphone and listen to differences. Well differences heard in such a test can only correspond to the very minimum that should be audible in a high-end setup. And certainly in my opinion the results of such tests should not be seen as the strict threshold for audibility. Just my opinion.

 

[andreasmaaan]

Yes I was being general indeed.
But as for headphones vs speakers in an anechoic room. I don't think there's any area at all that speakers won't beat headphones.
Even for group delay is my guess, though that would be the only area a single driver has an advantage, a multi driver speaker can be made to be flat in phase at the expense of some pre-ringing and my guess would be that all the other advantages of a speaker in an anechoic room would by far allow more critical listening also to phase.

I tend to agree with you here, notwithstanding the fact that all we can do is speculate

In fact adding to the force of your argument, in an anechoic room the pre-ringing caused by digital phase linearisation would be effectively nullified, since the ringing occurs only off-axis.

Yes I follow you here. But still, with familiarity and learning how to listen "into" a piece of music and what to listen for both good and bad has value to me. Not only to learn what to listen for, but also for setting preference of what sounds "good". If presented for the first time with 2 different sounds not heard before it's not only harder to pick out the differences but also to say which one is prefered.

Ok, but why assume that controlled listening tests involve only sounds/music not heard before? And even if the selected signals are not familiar to subjects prior to testing, this familiarity is gained in any case through the training process. In all cases, I just can't see how becoming familiar with a particular system in a particular room (other than the test system/room) could be more effective than pointed, rigorous training for a formal listening test. And the potential pitfalls of treating this kind of uncontrolled familiarisation with a particular system are many.

Couple this with how tests are often done, a group of people which have to put on a headphone and listen to differences. Well differences heard in such a test can only correspond to the very minimum that should be audible in a high-end setup. And certainly in my opinion the results of such tests should not be seen as the strict threshold for audibility. Just my opinion.

I don't any rational person would argue that a single test is adequate to determine any kind of strict threshold for audibility. Each test needs to be looked at critically and assessed on its merits. Equally importantly, results of tests must always be informed by the body of research already out there, and not only previous research into the same specific phenomenon. To illustrate what I mean here, I always try to keep in mind results of research into auditory masking when considering the results of research on distortion audibility, etc.

 

[JustIntonation]

I tend to agree with you here, notwithstanding the fact that all we can do is speculate

In fact adding to the force of your argument, in an anechoic room the pre-ringing caused by digital phase linearisation would be effectively nullified, since the ringing occurs only off-axis.



Ok, but why assume that controlled listening tests involve only sounds/music not heard before? And even if the selected signals are not familiar to subjects prior to testing, this familiarity is gained in any case through the training process. In all cases, I just can't see how becoming familiar with a particular system in a particular room (other than the test system/room) could be more effective than pointed, rigorous training for a formal listening test. And the potential pitfalls of treating this kind of uncontrolled familiarisation with a particular system are many.



I don't any rational person would argue that a single test is adequate to determine any kind of strict threshold for audibility. Each test needs to be looked at critically and assessed on its merits. Equally importantly, results of tests must always be informed by the body of research already out there, and not only previous research into the same specific phenomenon. To illustrate what I mean here, I always try to keep in mind results of research into auditory masking when considering the results of research on distortion audibility, etc.

Well.. perhaps I'm a bit biased here. I'm in particular familiar with how tests are often done and published for understanding how the brain processes music. These tests are true horrors. Tone probe tests with headphones in groups of people and then presented as scientific evidence, combined with conclusions based on that data that could not be more wrong. I won't elaborate on them but it's awful and anything but "scientific". And not at all capable of even seperating music processing from sound processing etc etc.
My whole view of this type of testing a paper publishing is perhaps negatively tainted because of this. I don't think I've ever read any papers concerning physchoacoustics etc though. So perhaps I speak too much Only hearing the results from them mentioned on this forum a lot and right away think of the many ways such tests can not be done with any perfection, it's the brain we're dealing with which we can't tie to hard science.

Anyhow about familiarity. Yes I do think that with familiarity one can pick up on things that one would not necessarily pick up on a system one is not familiar with. With time we're focussing our attention on many different aspects of sound and make an internal picture of it, and the brain perhaps learns how to process certain things. I think this familiarity can greatly help to focus ones attention to a certain aspect of the sound that would otherwise be easy to miss.
A related but different example. I worked a lot with different synthesizers. Often their "sound"/colouration becomes apparent only after a certain time spent with them. But once you pick up on this sound signature you can recognize it right away in the future. Well the same thing happens with DACs and amps and speakers etc for me. So this familiarity could in some aspects lead me to failing a blind test at first and after more familiarity I'd pass it. I imagine atleast as I've never tested this

 

[jsrtheta]

A related but different example. I worked a lot with different synthesizers. Often their "sound"/colouration becomes apparent only after a certain time spent with them. But once you pick up on this sound signature you can recognize it right away in the future. Well the same thing happens with DACs and amps and speakers etc for me. So this familiarity could in some aspects lead me to failing a blind test at first and after more familiarity I'd pass it. I imagine atleast as I've never tested this

There is a way to test this. We are talking about the scientific method here, and there is no big exception carved out for audio just because it offends sensibilities.

There is an easy way to test your hypotheses. When you do so, I will listen. For now, it's just PIDOOMA.

 

[JustIntonation]

There is a way to test this. We are talking about the scientific method here, and there is no big exception carved out for audio just because it offends sensibilities.

There is an easy way to test your hypotheses. When you do so, I will listen. For now, it's just PIDOOMA.

There is no real exact scientific method here. With the exception of doing a proper ABX blind test.
I'll be sure to be doing a lot of ABX blind tests once my new system and room are finished.
But as far as the scientific method. I think it is probably most meaningful and valid to get a collection of confirmed ABX blind tests that have been passed by anybody on any system, and have those thresholds set as a minimum of audibility. For instance if somebody passes an ABX of 320kbs mp3 vs wav then it's fair to say that 320mp3 is audible to some. Etc. And I think you'll find the thresholds of audibility to be very different than those often portayed on this forum by this method.
One other thing. Often measurements measure an average level of jitter or distortion. This is also not how we hear. If we have a minute long of perfect audio and 1ms of audible distortion then this is audible yet wouldn't show up on the type of measurements done on this site. This is an extreme example, but there's reason to believe certain types of jitter for instance behave in that direction?
Again, love the measurements and there isn't anything more valuable. But I do think some on this site take it a bit too far in saying anything that measures well enough sounds identical and differences people hear are made up.

 

[jsrtheta]

There is no real exact scientific method here. With the exception of doing a proper ABX blind test.
I'll be sure to be doing a lot of ABX blind tests once my new system and room are finished.
But as far as the scientific method. I think it is probably most meaningful and valid to get a collection of confirmed ABX blind tests that have been passed by anybody on any system, and have those thresholds set as a minimum of audibility. For instance if somebody passes an ABX of 320kbs mp3 vs wav then it's fair to say that 320mp3 is audible to some. Etc. And I think you'll find the thresholds of audibility to be very different than those often portayed on this forum by this method.
One other thing. Often measurements measure an average level of jitter or distortion. This is also not how we hear. If we have a minute long of perfect audio and 1ms of audible distortion then this is audible yet wouldn't show up on the type of measurements done on this site. This is an extreme example, but there's reason to believe certain types of jitter for instance behave in that direction?
Again, love the measurements and there isn't anything more valuable. But I do think some on this site take it a bit too far in saying anything that measures well enough sounds identical and differences people hear are made up.

Name one valid scientific test that shows jitter is audible. And not a test where jitter is exaggerated to get a result, which has been done. But take a look at what's been written about jitter audibility on Archimago's site.

Jitter is like UFOs - often reported, never substantiated.

 

[JustIntonation]

Name one valid scientific test that shows jitter is audible. And not a test where jitter is exaggerated to get a result, which has been done. But take a look at what's been written about jitter audibility on Archimago's site.

Jitter is like UFOs - often reported, never substantiated.

I don't know if there's a valid scientific test or not that shows real world jitter is audible. I have no idea where the threshold lies for jitter audiblity.
I do know that when I personally switched from the on board USB input of my Anedio D2 DAC to the Armature Hecate (Singxer F-1) USB to S/PDIF converter into the D2 DAC that the difference/improvement was clearly audible to me.
Here lies perhaps the problem. How to test this properly blind?
One way is to have a friend switch this. Record this process with a video recording perhaps as "proof" (which could still be faked / agreed upon beforehand, so not real proof). The only easy way for people to prove something at home is by using a program like ABX comparator for Foobar2000, which I have and use when I can. But in this case it's useless as I have to switch between 2 USB interfaces. One solution would be to record the output of the D2 DAC with a reference quality ADC and then load the two files in Foobar2000 and do the ABX comparison (which gives hashed proof that can't be tampered with). That is, if the difference survives the ADC process.
Alas, most people don't have a reference quality ADC to do this test with. So yes, there are a lot of reports by serious people (and even many many more by not so experienced people who I wouldn't trust personally either).
But does this disqualify the result I got? Well surely not for me. I know my own experience with audio and I'm not at all into fooling myself and know very well the risk of this. When I know for sure that the difference is readily audible to me and I would easily pick it out in a blind test had I the opportunity to do a proven ABX blind test with it.

 

[JustIntonation]

And again regarding jitter. This is not that easy to simulate. One would have to simulate possible packet loss (as far as I understand this can still happen with async USB) and has to reach a certain level to cause the extremely audible dropouts, but there is loss in audio conversion before that. Again this would not show up that clearly on an averaged measurement like THD+N if I understand it clearly. Then there's random jitter / timing errors, then there are all kinds of correlated jitter. I'm not a jitter expert, but again the proper way to test this would be to connect a DAC to an ADC that has a proven extremely low jitter. Again not something people or even studios have at hand.
Not that easy to test audibility of jitter it seems to me..
Btw, weren't there some studies that showed very high sensitivity to a certain type of jitter that gives a small modulation of pitch that is highly audible?

 

[jsrtheta]

"I know my own experience with audio and I'm not at all into fooling myself and know very well the risk of this. "

That's special pleading. There are test results available on Archimago. Lots of them.

 

[JustIntonation]

"I know my own experience with audio and I'm not at all into fooling myself and know very well the risk of this. "

That's special pleading. There are test results available on Archimago. Lots of them.

Ah didn't know that blog. Will have a read.
Though the first thing I'm reading is a plea against 24-bit audio vs 16-bit (of which I personally haven't even heard a difference in sound quality btw of music in this format..), but the argument is that "why bother" as the majority of people couldn't tell the difference. Well my argument would be the other way around. Apparently some people have (proveably) showed they could hear the difference. This shouldn't be an argument against, it's an argument for.
Hell the masses have horrible sound systems in even more disastrously horrible rooms coupled with inexperienced bad ears. The day I start building my music system based on things that "the majority of people" can't hear is the day I should just quit audio alltogether..

 

[andreasmaaan]

My whole view of this type of testing a paper publishing is perhaps negatively tainted because of this. I don't think I've ever read any papers concerning physchoacoustics etc though. So perhaps I speak too much

I suggest you read this.

Not only is it a quite comprehensive introduction to the field of psychoacoustics, but it also outlines the landmark studies into virtually all aspects of the human auditory system. For me, it provided a solid basis for interpreting evidence from the kinds of studies that I think you're talking about.

 

[andreasmaaan]

Apparently some people have (proveably) showed they could hear the difference. This shouldn't be an argument against, it's an argument for.

Are you referring to the Bob Stuart study? I consider the methodology of that study to be flawed due to the authors' decision to use rectangular dither, and I'm not aware of any other studies that have given similar positive results (but am open to it if you know of any?).