Was this aimed at ASR?

[amirm]

What I meant was some people might like certain set of frequency for their listening enjoyment? No?
Isn't harman test just test on your hearing not your the eq preference.

No. They just play music and you get to vote how correct the sound of one speaker is versus another. If you like more bass than a speaker has, then you will vote it less good, etc.

 

[b1daly]

This is why I made my first post. Folks are seeing us as serious competition instead of welcoming more people who are after data to help audiophiles make better decisions. The rivalry oozes in between the words.

I would say ‘rivalry’ might not capture what is going on here. To hazard a guess, ASR has shown many pieces of boutique/high dollar audio gear to be sub-par in performance, often in comparison to much cheaper gear. The companies that make this gear are the bread and butter for advertising in the “audio press”.

Testing showing poor performance of such gear is a threat to stereophile on a couple of levels. It has the possibility of lessening the chance these boutique manufacturers will stay in business.

The other issue is that the relationship between manufacturer and publisher is dependent on positive reviews. They will not be likely to advertise in a publication that gives bad reviews to their products. So this puts Stereophile in a predicamen. If ASR challenges their credibility, they can’t answer with better reviews without damaging relationships with audiophile manufacture. But if they refuse to answer the gauntlet thrown down by ASR they risk losing credibility with their readership. If this happens, readership will drop (probably). And readership is the entire reason manufacturers advertise at all.

This is further compounded by the rise of new boutique manufacturers who do focus on object performance. These can be tiny companies who might just rely on word-of-mouth and good press to sell directly to users. They will not likely spend much on advertising.

This is the dilemma that explains the ongoing absurdity found in ‘subjectivist’ audio reviews.

One of the reasons they can keep this up is that evaluating the sound quality and other performance attributes of audio gear is confusing, hard, and expensive. People want authorities they can trust.

Magazines like Stereophile are anachronisms. The editorial style remains unchanged from at least the 1980s. When you think about how many decades they have been declaring that ’this’ piece of gear outshines all that came before, it’s farcical.

The modern popular audio press like What Hi-Fi or CNET has the same problem. They review more mass produced products, but the same dynamics apply.

This problem plagues almost all reviewing outfits that rely on advertising to exist, across many different product categories.

A different business model that I’ve seen smaller review sites adopt is to buy the gear themselves and use affiliate links.

Sites like WireCutter also using affiliate links. As far as I can tell they do a reasonable job reviewing, though I suspect there are some shenanigans as I believe the gear is provided by manufacturer.

ASR doesn’t seem to have any business model at all, outside of donations for the upkeep of the panthers! Having users send equipment in to test is utterly unique and a cool thing all around.

I think Amir is building a huge amount of credibility here. The Schiit reviews were a good example. After some brutal reviews there was some grumbling about a negative bias, but I think the many positive reviews he has given them show him to be a ‘straight shooter’ which I appreciate very much. Thanks Amir!

 

[b1daly]

First off, thanks for your many thoughtful posts - I find myself in agreement with much of what you have been expressing.

There's a heck of a lot in that one post, so I won't try to react to all of it.

Could you expand on the pont above? I'm not sure how I would do that *if* the person had a true interest in audio.

That‘s a good question.

- most people listen to very poor audio. There are many affordable powered studio monitors that blow away any average consumer system. And they are relatively affordable. Almost everyone has ‘good enough’ threshold, and it doesn’t take a huge expense to pass it for most people. I’ve been less impressed with the handful of modern bookshelf speakers I’ve heard on the consumer side, but even these are still quite good.

- I’m both very picky and idiosyncratic in my preferences for audio playback. Many well designed studio monitors have very well controlled frequency response and are
biamped so they have plenty of power. Because of this they ‘sound good’ on most recordings. But to my ear they often don‘t sound ‘right’. They are just the least wrong on average.

- Since I’m picky and an audio engineer, I will fuss around a lot with different speakers, placement , and have no problem EQing speakers if I need to. Some of the older speakers I like have peaks or dips in the frequency response. What happens is that I usually have them sounding good, and then along comes a recording that has anomalies in spots that are exaggerated in the same place, causing the speakers deficiency to become appatent. For some systems I listen on, I have EQ profiles I’ve built overtime for different speakers. This takes considerable skill, technical competence, and trial and error for your average listener who just wants to hear some music. Audio is both a vocation and avocation for me, so I don’t mind working on it. I also love music, and over the years I’ve become less able to overlook sound reproduction that bugs me. Oddly, it’s often context based, or even mood based. I have a portable speaker in my garage, and I thoroughly enjoy listening to it while I’m out there!

 

[b1daly]

Nothing will change for the better with that kind of fatalism. The benefit is better sounding music and preserving the artist's intent. I would definitely say this is worth it. The music production side has so far failed miserably to fix this issue by implementing standards, so unfortunately it's now down to the reproduction side to force this change. ASR is helping with this by measuring speakers and praising those that perform well according to the 'standard' of the preference formula, which will hopefully cause people to vote with their wallets and so encourage more manufacturers to follow this standard. The question many ask is, is this preference standard accurate enough? And the answer is, it doesn't need to be 100% accurate in order to work as a standard. For example, if speakers with the same frequency response are used in the studio and at home, and the music is mastered to sound best out of that speaker, it will translate well to the home speaker, even if both speakers have say, a dip at 2kHz. The mastering engineer will simply adjust levels to account for that dip. So then the question is, why choose the particular flat anechoic on-axis response the preference formula rewards? And the answer to that is, this is the standard least disruptive to the music industry, because it is based on the average preference of speakers in controlled, double-blind tests. In effect, most of the common arguments against the use of the preference formula are moot, because ultimately it's not really about preference - it's just a tool to ease the audio industry out of the vicious cycle of confusion it's been caught up in for decades, and towards a standard that will put an end to all this nonsense. This is the bigger picture. A good enough standard needs to be chosen, stuck with, and publicised, and Dr. Sean Olive's preference formula is the best one we have that I think most people will agree follows the majority of people's preference of the sound reproduction of music produced in a variety of unstandardized ways, and so will cause the least disruption to the industry in the transition phase towards standardization.

I should qualify my anecdotal observations about the direction of the audio/music industry: what I see driving the change away from conventional notions of what constitutes good sound reproduction is not cynicism or laziness but the trends towards miniaturization and portability.

Within this realm we have seen some great improvements in audio quality within strong constraints (price, environment). The overall and low frequency response in some portable speakers is quite remarkable. Car audio is vastly improved overall. Earbuds likewise have massively improved from the time they became widely used (with the iphone). Amir has shown that many headphone dongles have respectable audio performance at a crazy low price and tiny size!

The kind of research Toole and his colleagues have done does inform much of the engineering practices used in modern audio engineering- just in ways that are orthogonal to notions of optimal audio reproduction.

Sitting in front of two good speakers to listen is just not done much anymore, never mind esoteric surround setups. I have two stepsons. One listens to no music and the other listens to exclusively rap. I have a very nice stereo system setup in our living room and he has never once listened to music on it. Outside of music found in video games, where he will use the speakers.

I‘m a recording engineer and music producer as well as a music lover and audio enthusiast. I just don‘t think your description of the production process really captures what is going on. Many practices in music and audio reproduction are emergent from cultural processes. That’s not to say science and standards don’t play a role, they do, but many other forces are at play.

The goal of mastering is not to tailor the music towards low end playback, but to release product that translates well across the world of playback systems. Everyone I know in audio production wants their products to sound good on good systems too. The things that are done to improve playback on lesser systems can also improve it on good systems. The biggest area of compromise is in dynamic range. Noisy environments require dynamic compression for effective translation, and this can sound strange on high resolution systems in quiet environments. (Dynamic compression pulls up the level of quiet sounds, keeping them from being ‘masked’ by environmental noise.)

In my experience studio control rooms sound nothing like living rooms, or any other consumer playback environment. One of the biggest differences is they are much quieter. Modern studio monitors are also much higher resolution all around than consumer systems. It is easier to get a mix to sound good in this kind of environment, it is more forgiving of anomalies in dynamic frequency response. Since virtually no consumers have such an environment, you simply cannot tailor production to this.

Even if home ‘hi-if’ started coming back as a market force, you still have to target other types of playback systems. The trend towards portability and miniaturization is probably permanent, because it has the benefit of allowing people to enjoy music in so many more places, and at lower cost. (A decent bluetooth speaker can be had for $60).

What really makes this all work is that musical communication works with much higher bandwidth than that found in audio. Essentially, much of the actual communication is ‘out of band’ and the actual audio part only has to be ‘good enough’ to work. (For example, music genre can be communicated over a low res system, and once this is done the mind will contexualize what it is hearing accordingly).

It‘s not the same, but here‘s an extreme example. Spoken voice communication can be effectively transmitted over very low bandwidth systems will little to no loss of communicative power. Music requires much more fidelity to be effectively communicated than spoken word, but it doesn’t have to be perfect to work beautifully.

 

[Blujackaal]

Sounds like Stereophile just gave away that coloured gear get hyped because it's easy $$$ not because there any good. Since when it pointed out you get a horde of audiophiles going on why sound subjective, To avoid they just been fooled the same way they bash Beats/bose fans for. lol

 

[mhardy6647]

That's not how I choose my music.

Good for you, sir!
I think that a few some many ahem audiophiles choose their recordings as "sound" rather than music. I will cop to owning a few audiophile recordings.
(and I'll also even admit that I like Jennifer Warnes, who sort of straddles the sound and music camps!)

What's this thread about again...?

 

[Robin L]

The modern popular audio press like What Hi-Fi or CNET has the same problem. They review more mass produced products, but the same dynamics apply.

Since Steve Guttenberg left, I'm not seeing much discussion of audio at CNET, only the occasional talk of headphones and soundbars. What Hi-Fi always struck me as primarily advertisement, with very little critical writing. As you point out elsewhere, the notion of a stereo as something all by itself is mostly gone, with most people using smart phones as sources, big speakers mostly an adjunct to much bigger screens, music with visuals, sound as an afterthought. It's a long way from 1970. I'm not saying everything has gone to hell, at least in the audio department. But things have changed.

 

[Kal Rubinson]

The other issue is that the relationship between manufacturer and publisher is dependent on positive reviews. They will not be likely to advertise in a publication that gives bad reviews to their products. So this puts Stereophile in a predicamen.

Earlier posted and already answered: "There are plenty of fish in the sea ."

 

[CtheArgie]

OK, I'll try to explain my issue with "preference tests". I will have to be a little long, so please bare with me.

I took a seminar in graduate school with the "inventor" of conjoint analysis. Paul Green was a systematic researcher that explored "mathematical" modeling of consumer choices. Some of his earlier work had to do with consumer choices in airplane travel. For example, consumers preferred on time arrival over seat width. He could "measure" preferences and called them utilities. You can find out about the techniques rather easily.

During the seminar, he presented probably the first project of utility (preference) measurement for the pharma industry. He was comparing the different "attributes" of two types of compounds for hypertension. One was called beta blockers (still used) and the other was for a new type of compound called ACE inhibitors. BBs were the most commonly used ($$$) at the time (unless you include diuretics but it doesn't matter now). BBs had (have) undeniable and conclusive data showing that they reduced the chances of a heart attack or sudden death in people taking them. But, one of the most "insidious" side effects was impotence. The incidence was not very high, but the popularity of it was the issue. BBs also had better BP control than the ACEi he was consumer testing. The ACEi did NOT have data on CV mortality and had a slightly less efficacious control of BP than BBs.

The testing was done with physicians and with patients. The model showed that both physicians and patients preferred ACEi over BBs because the "immediate" risk of impotence was much more important than the "eventual" risk of a heart attack. The "negative utility" or "preference" for impotence was more valuable than the "positive utility" or "preference" for improved survival. If you look at it rationally or logically, this wouldn't be the case, but this is what the study of preference showed and the actual marketing of the drugs demonstrated as well. A majority of hypertensive patients are men over the age of 50. "If your dog won't hunt, you didn't care about being alive". ACE inhibitors and other similar agents completely took over the treatment of HBP. BBs are now reserved for patients that already had a heart attack. And we have a blue pill to deal with the hunting pooch.

So, in this case, personal preference, in repeated exercises, in different groups, showed that "irrational" choices or preferences were more important to "rational" ones.

Many years later, I read Kahnemann who demonstrated that humans are not rational. I was never able to find out if Paul Green knew about Kahnemann's (and Tversky's) work. The overlap is uncanny. People like Ariely at Duke now do a lot of work based on irrationality of people's decisions. I wonder if fancy cables conclusively demonstrate what Tversky used to say: That he studied people's stupidity.....

It MAY be that Toole tested preference and not accuracy. Nothing wrong with that. But, if they used a string quartet, for example, to test speaker preference, then bass response should not be part of the decision. If you incorporate music with bass, then they demonstrated that bass will impact the overall choice or preference.

I would want to repeat Toole's choices with music that had no bass and then show if his preference stands. Clearly, it can't. But then you would know what other factors drive people to prefer one speaker over the other. In the "regression model" that is preference, the "utility" provided by bass will overwhelm almost all other parameters. At least it seems that way!

@b1daly shows exactly the problem here. Preference may not equate to accuracy. In "amplified" music, knowing how the end product was mastered is important. b1daly shows that if one listens to old masters with modern accurate monitors you may have a problem. What would happen if someone picked the Beatles original masters and remastered them using modern accurate monitors? Probably, we would have a problem with them as we would say that this is not how they "sounded"! So, for amplified music, having a standard monitoring curve seems perfectly reasonable to me. The consumer could then match their sound system to the"Harman" curve.

But for classical music, the Harman preference may not match the actual frequency distribution of the music in the playing space! I don't know if this is the case, but I am pretty sure that audience preference at the experimental site without a "live music" control can create problems. It would be the "active control" of experiments. You would not be controlling Speaker A vs B vs C but also vs "Live music".

You can have a preference choice that is not accurate or rational.

Sorry for the digressions. Hope you can think it over.

 

[amirm]

It MAY be that Toole tested preference and not accuracy.

Having taken the test, I can tell you that you vote for what you think is more accurate and liked to you. They put four speakers in the pool and you give them scores. You make the judgement of what ultimately is good in your opinion. With no reference to the real thing, you get to make that up just as you would if you were choosing a speaker to buy.

 

[RayDunzl]

I would want to repeat Toole's choices with music that had no bass and then show if his preference stands.

 

[CtheArgie]

@amirm, then "preference" may not necessarily mean "accurate". Especially if you are missing an "active control". "Good in your opinion" or "liked" is subjective. The method just systematized subjectivity.

 

[Thomas savage]

You sh***ing me. He really did that?

Spectacular insecurity about what they are doing then.

It's plain sad if true.

 

[tmtomh]

This seems to be a common allergic reaction to ASR Chris at AudiophileStyle implemented a "feature" that automatically breaks any links that point to ASR content as they get posted on that forum. Ostensibly to protect the innocent, defenseless audiophiles.

That's appalling. I haven't been active on AS since the purge of objectivists, but I did briefly look in on the "Objective-fi" subforum just the other day and was appalled - but unsurprised - to see that even that subforum has been overrun with by the site's subjectivists who appear to be trolling @pkane and other reasonable folks by repeatedly demanding "evidence" for every bit of well-established science that's mentioned or citied.

If Chris is actually blocking any links to ASR, that's unconscionable.

 

[Purité Audio]

Once advertising becomes your main revenue stream...
Keith

 

[MediumRare]

"If your dog won't hunt, you didn't care about being alive". ACE inhibitors and other similar agents completely took over the treatment of HBP. BBs are now reserved for patients that already had a heart attack. And we have a blue pill to deal with the hunting pooch.

So, in this case, personal preference, in repeated exercises, in different groups, showed that "irrational" choices or preferences were more important to "rational" ones.

Many years later, I read Kahnemann who demonstrated that humans are not rational.

Great comment. I also have a background in consumer preference testing, although it's not my main career activity. A couple points worth throwing in:

- Conjoint analysis is a great tool to tease out the various contributors to preference.
- I'm surprised we haven't seen yet any evidence of that being done by Toole and Olive; it seems they "should" have done it.
- I think you are right about the FR curve swamping the contributions of other factors (distortion is an obvious one, if measured correctly).
- I still find it implausible there are no segments of preference recognized in the Harman work; I believe headphone testing has proved otherwise and personal (anecdotal) experience supports that there are segments.
- Economists think everything revolves around money so they think humans are irrational. Well, they are, but that doesn't mean that preferring a 10% chance of dying in 10 years to a 50% chance of losing your manhood at age 50 is irrational.

 

[MediumRare]

Having taken the test, I can tell you that you vote for what you think is more accurate and liked to you.

Not trying to nitpick, but which is it? "Accurate" and "Liked" are not the same at all.

 

[CtheArgie]

Great comment. I also have a background in consumer preference testing, although it's not my main career activity. A couple points worth throwing in:

- Conjoint analysis is a great tool to tease out the various contributors to preference.
- I'm surprised we haven't seen yet any evidence of that being done by Toole and Olive; it seems they "should" have done it.
- I think you are right about the FR curve swamping the contributions of other factors (distortion is an obvious one, if measured correctly).
- I still find it implausible there are no segments of preference recognized in the Harman work; I believe headphone testing has proved otherwise and personal (anecdotal) experience supports that there are segments.
- Economists think everything revolves around money so they think humans are irrational. Well, they are, but that doesn't mean that preferring a 10% chance of dying in 10 years to a 50% chance of losing your manhood at age 50 is irrational.

Actually, classical economists think that humans are RATIONAL! This is why Kahnemann and Tversky's research was so fundamentally disruptive!

And thanks for your kind comments.

I am truly interested in Toole's work from the methodology point of view, though I am becoming convinced that their results are completely valid for the recording and reproduction of amplified music.

My work is not on consumer testing but on clinical trials design, analysis and interpretation. This is why I am interested in methodology. The way you set up the experiment (clinical trial) strongly influences results.

 

[MediumRare]

Actually, classical economists think that humans are RATIONAL! This is why Kahnemann and Tversky's research was so fundamentally disruptive!

And thanks for your kind comments.

I am truly interested in Toole's work from the methodology point of view, though I am becoming convinced that their results are completely valid for the recording and reproduction of amplified music.

Economists. So many jokes. "First, assume we have a can opener." It's a long joke so the punch line will have to suffice.

Anyway, I wrote my thesis disproving (at least I thought) what an economist had won a Nobel prize for. When I discussed my finding with him his only retort was, "You're still young. You'll see later." (He refused to even engage with me in a discussion.)

Anyway, humans are neither rational nor irrational, they are individual. But no one will pay an economist to point that out.

 

[John Atkinson]

John, I feel obliged to take advantage of this (to me) rare opportunity to communicate with you directly. I've admired your work for years and very much enjoyed your many appearances in YouTube videos. Congratulations - and thank you - for a lifetime of contributions to your industry and to our enjoyment of its products!

Thank you.

In many ways you have the most extensive experience of anyone in marrying the subjective reviews of your writers with measurements of that same equipment - and providing context for your readers to understand them both.

So, may I ask you to share with us your observations on this question of the difference (or possible gap between) between "preference" and "strict adherence to technical metrics" with respect to speakers?

I agree with Floyd Toole's summing up about what defines a good-sounding loudpeaker - or a loudspeaker that will be preferred by most listeners. See the conclusions in my 1991 article on loudspeaker measurements at https://www.stereophile.com/content/ja-loudspeaker-measurements-page-3 . So I have problems when I encounter loudpeakers like the Volti Razz - https://www.stereophile.com/content/volti-audio-razz-loudspeaker - or the Rethm Maarga - https://www.stereophile.com/content/rethm-maarga-loudspeaker - where the designer could have made a loudspeaker that adheres to the Toole model but deliberately didn't. I ask myself what they gained by sacrificing tonal accuracy and well-controlled dispersion. Sensitivity? Low distortion? In both these cases, whatever the designer was aiming for, for me - for me - the price was too high.

Sometimes the answer is that they wanted their designs to sound very different from the norm and "different" can always be perceived as "better." But more often, the designer wants a sound that to his ears and with his choice of music sounds right in his room with his associated equipment. This is what Stereophile's founder, the late Gordon Holt, used to call "my-fi" rather than "hi-fi."

John Atkinson
Technical Editor, Stereophile