AES meta analysis on audibility of hi-res

[NorthSky]

Paper freely available: http://www.aes.org/tmpFiles/elib/20160629/18296.pdf

The link doesn't work for me.

Not Found
The requested URL /tmpFiles/elib/20160629/18296.pdf was not found on this server.

Apache/2.2.15 (Red Hat) Server at www.aes.org Port 80

 

[NorthSky]

High Resolution Audio Perception Meta-Analysis

The June 2016 issue of the Journal of the Audio Engineering Society (JAES) includes a fascinating paper entitled, "A Meta-Analysis of High Resolution Audio Perceptual Evaluation" by Joshua D. Reiss, PhD, a Reader (professor) at Queen Mary University of London's Centre for Digital Music. In his paper, Dr. Reiss summarizes the results of his meta-analysis of 18 published experiments, with a total of over 400 participants in over 12,500 trials, intended to determine if high-res audio can be perceptually distinguished from "standard" audio that conforms to CD specs.

For those who might not be familiar with the term "meta-analysis," it's a process that compiles data from multiple studies, performs statistical analysis on the aggregate data, and draws new conclusions from this analysis. As Wikipedia puts it, "Conceptually, a meta-analysis uses a statistical approach to combine the results from multiple studies in an effort to increase power (over individual studies), improve estimates of the size of the effect and/or to resolve uncertainty when reports disagree. A meta-analysis is a statistical overview of the results from one or more systematic review. Basically, it produces a weighted average of the included study results."

The studies used in the meta-analysis were not identical. In fact, they can be divided into two main groups: auditory perception resolution and format discrimination. In the first group, most of the studies were concerned with frequency and temporal resolution—that is, the extent to which humans perceive frequencies above 20 kHz as well as time smearing caused by lowpass and anti-aliasing filters. The second group were focused on how well humans can distinguish between different formats, including CD, high-res PCM, and DSD.

Also of interest is the methodology used in these studies. They included AB (play sample A, then sample B, ask participants which is which), ABX (play sample A, then sample B, then one or the other and ask participants to identify it as A or B), and others. In addition, an analysis of the different methodologies and possible biases that could arise from them helps to understand the results. In the end, the meta-analysis sought to transform and combine the results from the various studies to determine the statistics of correct responses across all of them.

The first section of this table lists the studies included in the meta-analysis. The second section identifies the risk of various potential biases ("-" means low risk, "?" means unclear risk, and "? in a box" means high risk) along with the types of errors those risks lead to (Type I = false positive, Type II = false negative, Neutral = neither). The third section indicates the total number of trials and correct answers for each study with the associated binomial probability—assuming there is no discernable difference, this is the probability of obtaining at least that many correct answers. The numbers in boldface are statistically significant results.

The paper makes special note of the Meyer and Moran study from 2007, which has been widely discussed on AVS Forum and elsewhere—in fact, AVS is cited in Dr. Reiss' paper! It has the most participants of any study, but not all the required data was available, so it could only be included in parts of the meta-analysis. Among the problems cited in that study was that many of the test tracks might not have included high-resolution content for three different reasons: The encoding scheme of SACD obscures frequencies above 20 kHz, the mastering of SACD and DVD-Audio content might have applied additional lowpass filters, and some of the source material might not have been recorded in high resolution in the first place. Also, according to Dr. Reiss' paper, the experimental setup was not well-described, and the experiment was not well-controlled. Still, there was enough valid data to use in certain parts of the meta-analysis.

One aspect of the meta-analysis was to see if trained listeners—those who had been instructed in what to listen for, heard examples and learned the results before the test, and so on—performed significantly better than untrained listeners. (Not all studies identified trained and untrained listeners, so only those that did are included in this part of the meta-analysis.) As you might expect, trained listeners perform much better, as shown in the data plot below.

In this forest plot, the untrained listeners are clustered around 50% correct, though most of the studies are slightly above that. The trained listeners performed much better, usually between 60% and 70% correct, though the confidence intervals are much wider than for most of the studies with untrained listeners.

Several other factors were examined to see how they might have affected the results. These included the duration of each stimulus and the interval between stimuli, the test methodology, and bit depth (most of the studies were more concerned with high-frequency content) among others.

Overall, the meta-analysis concluded that, within the included studies, there was a small but statistically significant ability to discriminate between CD-quality audio and high-resolution audio with specs beyond those of CD. Also, trained listeners were substantially more successful than untrained participants at correctly distinguishing between high-res and CD-quality audio. In addition, the selection of stimuli and their duration may play an important role in the ability to discriminate, and the potential biases of the studies tended toward Type II (false negative) errors.

Several aspects of high-res audio perception could not be confirmed or denied because they were not a significant part of the studies. For example, as I mentioned, most of the studies were more concerned with high sample rates, not bit depth. Also, none of the studies used headphones, so questions about how headphones affect high-res audio perception remain open. Perhaps most important, the specific parameters of the audio systems used in the studies—such as the choice of applied filters, audio formats, and hardware components of the recording and playback systems—were not considered.

Clearly, more research is needed to quantify how high-res audio is perceived and how strongly it affects the listening experience. But this paper adds to our growing knowledge base and supports the notion that high-res audio is worthwhile—at least for those of us who care about such things.

Dr. Reiss' paper is available to download for free; to get a copy, click here. But be forewarned—you'll need some familiarity with statistical analysis to fully understand it. - Scott Wilkinson

 

[SoundAndMotion]

The link doesn't work for me.

Not Found
The requested URL /tmpFiles/elib/20160629/18296.pdf was not found on this server.

Apache/2.2.15 (Red Hat) Server at www.aes.org Port 80

I guess you didn't see my post:

But this link (click here) should always work.

I think you are immediately redirected to a temporary place, like Amir suggests.

 

[krabapple]

To me the biggest flaw in MM is that you cannot have people travel tens of kilometers, enter an unfamiliar room, with an unfamiliar system, and then do some critical listening. It doesn't work that way.

Bollocks. It 'works that way' every time an audiophile talks about veils lifted, creamier bass, etc based on immediate sighted A/B comparison.. But suddenly it doesn't work when people who already consider themselves to be 'critical listeners' agree to try it blind? Suddenly even *SACD* isn't 'hi rez' enough for a 'real' test? (another of Reiss's claims).

Bollocks.

How about taking M&M as what it intended to be: to simply take audiophiles at their own endlessly repeated word: that hi rez is '*obviously* better than Redbook. And how about taking away from M&M the obvious: that such claims are nonsense. There is no 'night and day' difference. There is nothing to 'blow you away'. Neil Young is clueless about digital audio. The emperor has no clothes...or at best, a tiny and ridiculous speedo.

Anyway I'm very please to see Archimago, who is all about the measurements, independently arrive at my main point, that this presumed 'small but important' (Reiss's words, including the unwarranted claim of 'important') finding doesn't really get to the matter of audiophile claims...which by now should be *universally acknowledged* to be nonsense.

"You know guys, the fact that we're even going through the contortions of complex statistical analysis after >15 years since the release of SACD and DVD-A clearly indicates that those who claim to hear "obvious" differences are plainly wrong. When a meta-analysis is used in science to gather data far and wide to find and declare statistical significance of this kind of tiny magnitude, it just means that the "signal to noise" ratio is poor and that the magnitude of the effect is obviously academic. The author stated just as much: "In summary, these results imply that, though the effect is perhaps small and difficult to detect, the perceived fidelity of an audio recording and playback chain is affected by operating beyond conventional consumer oriented levels." Notice the careful wording... In no way does it imply that these "small" and "difficult to detect" differences are necessarily "better" as audiophiles always desire to promote. I like this wording and think Dr. Reiss did a fantastic job putting this together. By the way, these results are of no surprise as we've been talking about this for years!"

 

[NorthSky]

I guess you didn't see my post:

Yes, I have read your post, and Amir's post too, and all the posts contained in this thread.
I still mentioned it to hvbias so that the million more eventual readers finding this thread can get a corrected/edited link.

Thanks for your contribution SoundAndMotion.

 

[NorthSky]

The original paper 404's out for me. Is everyone else still able to access it?

So I wasn't alone.

It is failing to load on my tablet. Will try later on pc where it worked before.

Same, on my laptop (Apache).

Ok, the paper is still there and he is another temporary link to it: http://www.aes.org/tmpFiles/elib/20160702/18296.pdf
The way to find it permanently is to go to the root of the page and then click download: http://www.aes.org/e-lib/browse.cfm?elib=18296
I think they put all the papers in a database and then pull them out on demand in a temporary place that eventually expires. The home page clearly says open access so should remain valid.

Thx.

I think your speculation is right. But this link (click here) should always work. It looks like this: http://www.aes.org/e-lib/download.cfm/18296.pdf?ID=18296
That is the link under the "Download Now" link. I think you are immediately redirected to a temporary place, like Amir suggests.

Confirmed.
_______-

Yesterday I was aware of all the above; I simply went to the source...first original post...the most important one.

 

[Fitzcaraldo215]

Bollocks. It 'works that way' every time an audiophile talks about veils lifted, creamier bass, etc based on immediate sighted A/B comparison.. But suddenly it doesn't work when people who already consider themselves to be 'critical listeners' agree to try it blind? Suddenly even *SACD* isn't 'hi rez' enough for a 'real' test? (another of Reiss's claims).

Bollocks.

How about taking M&M as what it intended to be: to simply take audiophiles at their own endlessly repeated word: that hi rez is '*obviously* better than Redbook. And how about taking away from M&M the obvious: that such claims are nonsense. There is no 'night and day' difference. There is nothing to 'blow you away'. Neil Young is clueless about digital audio. The emperor has no clothes...or at best, a tiny and ridiculous speedo.

Anyway I'm very please to see Archimago, who is all about the measurements, independently arrive at my main point, that this presumed 'small but important' (Reiss's words, including the unwarranted claim of 'important') finding doesn't really get to the matter of audiophile claims...which by now should be *universally acknowledged* to be nonsense.

"You know guys, the fact that we're even going through the contortions of complex statistical analysis after >15 years since the release of SACD and DVD-A clearly indicates that those who claim to hear "obvious" differences are plainly wrong. When a meta-analysis is used in science to gather data far and wide to find and declare statistical significance of this kind of tiny magnitude, it just means that the "signal to noise" ratio is poor and that the magnitude of the effect is obviously academic. The author stated just as much: "In summary, these results imply that, though the effect is perhaps small and difficult to detect, the perceived fidelity of an audio recording and playback chain is affected by operating beyond conventional consumer oriented levels." Notice the careful wording... In no way does it imply that these "small" and "difficult to detect" differences are necessarily "better" as audiophiles always desire to promote. I like this wording and think Dr. Reiss did a fantastic job putting this together. By the way, these results are of no surprise as we've been talking about this for years!"

I do not know why you, or Archimago who I normally like, are so concerned about "audiophile claims". I am not at all, really. I only care about what enhances my own listening pleasure. I want that to be based on something real, not snake oil, placebo, self delusion, etc. So, I weigh the evidence, including excellent papers like this plus measurements, and I listen as carefully as I can to reach the best conclusion for me.

I agree that many audiophiles go overboard, just like many nay sayers do, you included. So, why not go to their forums where those outlandish audiophile claims are made and bring your own cool-headed, rational and open- minded arguments to counter their outrageous claims!

I also agree with Amir, that M&M is seriously flawed and therefore its conclusions are useless. But, since you have an anti-hi-rez axe to grind, you would rather defend it to the extent it supports your beliefs.

Meanwhile, I like what hi rez does, so I buy it. To me, it is not nonsense at all. So, I hardly agree, especially after Reiss' paper, that it should be "universally acknowledged" as nonsense. I find good support in the paper that there is something of substance in hi rez. I think to read the paper your way is a gross distortion driven by your obvious bias.

Meanwhile, you hear no improvement from hi rez. So, just don't buy it. But, why the tirade?

 

[amirm]

If you like big engines and torque how does more than 5.6 million ft-lbs of torque sound (at 102 rpm no less). Thermal efficiency of 58%.
https://en.wikipedia.org/wiki/Wärtsilä-Sulzer_RTA96-C

Links with pictures:
http://www.zmescience.com/science/biggest-most-poweful-engine-world/
http://www.autoblog.com/2011/07/22/...engine-makes-109-000-horsepower/#slide-231440

And some video:

Amazing. It is like they took a normal engine and multiplied all dimensions by 1000! Wonder what kind of starter gets that going!

 

[hvbias]

Thought I would wander over to see what Hydrogen Audio thought hoping for a bit more critical analysis, and made it several posts into the first page and gave up. Audiophiles have one religion, HA has another. Somewhere there is a balance.

The devout believers don't need any research.

That is probably the most amusing part about all the posturing over a 56% success rate with a body of listeners who were specifically chosen from the general population because they are more likely to spot coloration resulting from nonlinear processing which quite possibly occurred only after the signal was converted back to analog.

I didn't read the paper, but I'm betting that jjf5 found nothing* to support his fantasy that someday he will discover unicorns in bits 17-24 of his hi-res purchases.

(*) The BS paper presented exactly *zero* evidence to corroborate the notion that trained listeners will "probably also" be able to recognize quantization effects.

 

[RayDunzl]

Wonder what kind of starter gets that going!

http://www.marinediesels.info/Basics/air_start_simple.htm

Some assembly pictures:

http://www.emma-maersk.com/engine/Wartsila_Sulzer_RTA96-C.htm

 

[krabapple]

I do not know why you, or Archimago who I normally like, are so concerned about "audiophile claims". I am not at all, really. I only care about what enhances my own listening pleasure.

I don't only care about that. I also care about how much influence such silliness has on the industry (see: MQA, Pono, etc) what things it focuses its money and R&D on, because this in the end affect the 'listener experience'. Neither you, I, Amir, nor anyone get an objectively substantial benefit (nor deficit, except in terms of storage space) from 'hi rez' per se. As opposed to, say, work on digital room/speaker correction which has potential to greatly and unambiguously affect the listener experience.


Similarly, I care about the idiotic claims made by creationists, because they tend to be politically active at the local (and sometimes national) level, and that ultimately can affect the 'citizen experience'.

I want that to be based on something real, not snake oil, placebo, self delusion, etc. So, I weigh the evidence, including excellent papers like this plus measurements, and I listen as carefully as I can to reach the best conclusion for me.

The conclusion you reach 'for you' is likely still affected by 'placebo, self delusion'. All that means , though, is that you should always qualify your claims.

I agree that many audiophiles go overboard, just like many nay sayers do, you included. So, why not go to their forums where those outlandish audiophile claims are made and bring your own cool-headed, rational and open- minded arguments to counter their outrageous claims!

This forum is not exactly free of such claims. Yet.

I also agree with Amir, that M&M is seriously flawed and therefore its conclusions are useless. But, since you have an anti-hi-rez axe to grind, you would rather defend it to the extent it supports your beliefs.

I agree that it is flawed, but flawed does not mean useless -- not even in science.

Meanwhile, I like what hi rez does, so I buy it. To me, it is not nonsense at all.

What does hi rez 'do', exactly? Make you feel good?

So, I hardly agree, especially after Reiss' paper, that it should be "universally acknowledged" as nonsense. I find good support in the paper that there is something of substance in hi rez. I think to read the paper your way is a gross distortion driven by your obvious bias.

Meanwhile, you hear no improvement from hi rez. So, just don't buy it. But, why the tirade?

You disagreed with me. Why the long response?[/QUOTE]

 

[krabapple]

Thought I would wander over to see what Hydrogen Audio thought hoping for a bit more critical analysis, and made it several posts into the first page and gave up. Audiophiles have one religion, HA has another. Somewhere there is a balance.

If that's as far as you read on that thread, then you weren't really looking for a bit more critical analysis, were you?

Btw one of the participants on that thread is AJ...who actually corresponded with Dr. Reiss starting well before the paper was published... and who is a moderator here...or at least was? Or has he been defrocked?

 

[Fitzcaraldo215]

I don't only care about that. I also care about how much influence such silliness has on the industry (see: MQA, Pono, etc) what things it focuses its money and R&D on, because this in the end affect the 'listener experience'. Neither you, I, Amir, nor anyone get an objectively substantial benefit (nor deficit, except in terms of storage space) from 'hi rez' per se. As opposed to, say, work on digital room/speaker correction which has potential to greatly and unambiguously affect the listener experience.


Similarly, I care about the idiotic claims made by creationists, because they tend to be politically active at the local (and sometimes national) level, and that ultimately can affect the 'citizen experience'.




The conclusion you reach 'for you' is likely still affected by 'placebo, self delusion'. All that means , though, is that you should always qualify your claims.




This forum is not exactly free of such claims. Yet.



I agree that it is flawed, but flawed does not mean useless -- not even in science.




What does hi rez 'do', exactly? Make you feel good?




You disagreed with me. Why the long response?

Me. I am not claiming anything except that I personally prefer hi rez, done correctly of course. I suggested to you that you not buy it if you did not see an advantage. I did not make any claims that it needs to be "universally" accepted, unlike your claim to the contrary. What "claims" do I need to qualify about my own stated listening perceptions and preferences for me in the comfort of my own home?

And, I have long used DSP room correction, for about 9 years now. Maybe we agree on that. Hi rez in Mch is what I predominantly listen to using mike calibrated DSP EQ. Hey, if it is all placebo that got me here, vive la placebo! Is that a claim I need to qualify?

I am not trying to convince you that you need to go hi rez. You will either figure out what it does or you won't. You just have to do that by listening yourself.

 

[Blumlein 88]

Nice timing on this little bit of writing about meta-analysis here.

http://blogs.plos.org/absolutely-maybe/2016/07/05/psychologys-meta-analysis-problem/

 

[Sal1950]

http://www.marinediesels.info/Basics/air_start_simple.htm

Some assembly pictures:

http://www.emma-maersk.com/engine/Wartsila_Sulzer_RTA96-C.htm

Holy Schmoly Ray, did you see the size of those main bearing journals. LOL
Interesting details into the air start system.
Thanks for the links!

 

[Blumlein 88]

Holy Schmoly Ray, did you see the size of those main bearing journals. LOL
Interesting details into the air start system.
Thanks for the links!

Think you could adapt one for motorcycle use?

 

[Sal1950]

Think you could adapt one for motorcycle use?

Nope, I only do V configurations.

 

[Keith_W]

Hi guys, so far nobody seems to have mentioned the biggest flaw of meta-analyses - publication bias. This is the bias that academic journals have towards studies that show positive results - i.e. studies that show positive results tend to get published, whereas studies that show negative results tend to get rejected. The problem is exacerbated further if most of the studies come from one journal, in which case publication bias is a function of the editorial board or the peers who review the studies and select those for publication.

I have plenty of experience reading academic papers, but I have to confess that I have not actually read this paper even though it was my intention to do so. So - apologies if it was mentioned in this thread, or the authors of the paper addressed it in their study.

 

[Fitzcaraldo215]

Hi guys, so far nobody seems to have mentioned the biggest flaw of meta-analyses - publication bias. This is the bias that academic journals have towards studies that show positive results - i.e. studies that show positive results tend to get published, whereas studies that show negative results tend to get rejected. The problem is exacerbated further if most of the studies come from one journal, in which case publication bias is a function of the editorial board or the peers who review the studies and select those for publication.

I have plenty of experience reading academic papers, but I have to confess that I have not actually read this paper even though it was my intention to do so. So - apologies if it was mentioned in this thread, or the authors of the paper addressed it in their study.

I think you will find the Reiss paper interesting. I did.

Yes, I agree, there is probably a bias toward publishing only papers with positive results. But, the most famous and widely cited one to date about hi rez audio is Meyer-Moran, which had a negative, no-difference result. But, that paper was deeply flawed according to many. Amir has a good analysis of it elsewhere in the forum.

I have not followed papers very much on other experiments involving hi rez. Its niche status in the marketplace seems not to have generated much interest in such experiments. And, perceptual experiments on human subjects with sufficient observations for credibility would seem difficult, complex and time consuming. So, this Reiss paper offers something of a useful summary of other papers I had not been aware of, whether you agree with its conclusions or not.

It does not superficially appear to jump blindly on any bandwagon. However, since it summarizes other published papers, possibly the bias toward positive results is a factor. And, spoiler alert, that summary does not point to a gee whiz breakthrough result, which is my own conclusion about the topic, as I have said often elsewhere. But, as a fan of hi rez myself, I see some support for the notion that there is something perceptable going on with it among many other listeners.

When it comes to perceptual studies of human subjects, we do tend to get away from "hard" science, like typical audio engineering measurements. We are into the realm of softer, noisier, statistical probability assessments, as in the behavioral and social sciences, medicine, etc., as is clear. That is why I strongly suggest people make their own listening assessments, but carefully to avoid common mistakes, but also to avoid rushes to judgement either way. I think the paper helps point out some common pitfalls.

 

[krabapple]

Hi guys, so far nobody seems to have mentioned the biggest flaw of meta-analyses - publication bias.

Section 3.6 of Reiss's paper is called: "3.6 Is there Publication Bias?"

So he, at least, certainly mentions it.