Is Audio Science Review going about it all wrong? Or partly wrong? Or all right?

[beefkabob]

I read the reviews. I look at the graphs. I see definite differences. Everything is confusing online. Forum posters disagree with each other. It's total chaos. Sometimes even measurements disagree.

Then I go to the local audiophile-grade store and I listen to a bunch of different integrated amps and DACs, sometimes in combos, and all on the same speakers, Paradigm 85F. Throughout, I hear significant differences: How clear and lifelike are the vocals? How full does the guitar sound? How lifelike is the sound overall? How separate and clear are the different instruments? How wide is the soundstage? How cohesive does the music sound, like vocals that are too quiet or loud?

The Rogue tube amp had amazing sounding guitars but the vocals were almost unintelligible. The Peachtree 150 sounded pitifully thin, weak, and flat, even when fed by a Luxman DAC. The Parasound Hint 6 sounded reasonably good, a strong all-rounder, but nothing exciting. The Ayre Codex + Luxman 20 watt Class A sounded wonderful, but you definitely need to disable the digital volume control on the Codex. Turning it on and bypassing the Luxman preamp sounded as lifeless and dull as the Peachtree 150. The Ayre EX-8 sounded just great. Was it better or worse than the Codex + Luxman? Hard to say since I listened 2 weeks apart. Both of those last two setups are really better than the 85F's need, but I can see myself stepping up to better speakers later.

Get back in the car to leave and realize how incredibly mediocre the Jaguar Meridian audio sounds. Better than the Peachtree, maybe.

How does any of this translate into SINAD or THD? I don't see much of a link. I just want the music to sound good, maybe even lifelike, though that's a stretch to achieve without really big bucks.

I also tried to see if I could get the salesman to disagree with me. I listened to a setup, described to myself what I heard, and then asked the salesman to describe it. He said almost exactly what I was thinking.

Interestingly, I know the salesman from working together 20 years ago in a computer store.

 

[RayDunzl]

I actually heard exactly nothing in my main system before purchase.

I didn't buy stuff and not like it and send it back, either, just for the record.

Good luck.

 

[Webninja]

For me, I was tired of subjective reviews and ASR provided the data I was looking for.

I do some visual testing for work and it is very easy to fool the senses, so I prefer relying on measured results. But overall I think you should buy and listen to what makes you happy.

My Topping DAC's great numbers made me happy and that's why I bought it.

 

[solderdude]

I read the reviews. I look at the graphs. I see definite differences. Everything is confusing online. Forum posters disagree with each other. It's total chaos. Sometimes even measurements disagree.

Welcome in the audio world with lots of people having different opinions they formed over the years with the experiences they had.

Measurable differences may not be audible differences but could be.
You need to look at and understand a suite of measurements (that have measured all of the things needed in a correct manner) before one can pass judgement.
There thus can be differences that aren't shown in measurements but exist.
Acoustics is a real bitch to measure and understand.

So ... chaos.. indeed to most folks.

Then I go to the local audiophile-grade store and I listen to a bunch of different integrated amps and DACs, sometimes in combos, and all on the same speakers, Paradigm 85F. Throughout, I hear significant differences: How clear and lifelike are the vocals? How full does the guitar sound? How lifelike is the sound overall? How separate and clear are the different instruments? How wide is the soundstage? How cohesive does the music sound, like vocals that are too quiet or loud?

The local audiophile-grade store salesman obviously did a good job there.
Should you have been in full control of the tested equipment consider small level differences, vol pot positions, looks of equipment and bias towards brands can totally create this effect.

When you even the scores (level match and switch from the listening position) chances are the differences are smaller. Consider the tube amp changes the sound. This might be very hard to show on a test bench with resistive dummy loads. To 'measure' this real world nulling would be the way to go.

How does any of this translate into SINAD or THD? I don't see much of a link. I just want the music to sound good, maybe even lifelike, though that's a stretch to achieve without really big bucks.

Sinad is NOT the one number that tells it all. It merely says something about the deviations from the original signal measured from a certain reference point. It really says nothing about the SQ but does say something about a technical aspect.
Don't get hung up on it when it is above say 80dB.

I also tried to see if I could get the salesman to disagree with me. I listened to a setup, described to myself what I heard, and then asked the salesman to describe it. He said almost exactly what I was thinking.

Almost .... and consider you were both listening at the same levels to the same equipement and seen each others 'cues' when listening.

And... to answer the question:
ASR is heading in the right direction in general. There is always room for improvement.
It is a forum, so some people disagree all the time and others agree or partly agree. Nothing strange about that.
That doesn't make ASR right, wrong or partly wrong. It says something about the community not about the content and who posted it.

 

[BillG]

I read the reviews. I look at the graphs. I see definite differences. Everything is confusing online. Forum posters disagree with each other. It's total chaos. Sometimes even measurements disagree.

As I see it, ASR isn't really here to tell you how your gear is going to sound once you get it home, but rather how close to specifications it is (manufacturers have this annoying habit of being less-than-honest about that) and how audibly transparent it should be based on what the performance analyses find. And in that regard, Amir, and the rest of the pros here on ASR, are doing a fantastic job.

A lot of how something is going to sound, when being fed with reasonably well performing and transparent electronics, is determined by the speakers and room they're playing in. (i.e. if you've got crap speakers, and a really reflective room, it's going to sound bad.)

Kind regards

P.S. Another point I wanted to add: Please don't be afraid to apply some EQ to your system to tune it to a sound that you like.

 

[decoRyder]

I think audiosciencereview provides an invaluable service by actually measuring the the reviewed devices, and by doing that showing the engineering quality, or lack thereof, of the reviewed devices. There's no other site that takes this diligent, systematic approach. Finally we don't have to rely on the subjective 'audiophile happy talk' that is used more often than not in almost all other publications on the web.

I needed a new DAC for the garage, cheap. I bought the Topping D30 based on the measurements from this site, and this site enabled me to make an INFORMED buying decision, rather than one based on strictly subjective reviews. That's a major benefit, and thanks to Amir for providing this service.

Another 'emergent benefit' from which we as consumers will benefit over time is that fact that audio manufacturers will notice the reviews and measurements on this site, and will - hopefully - improve the quality of their products (I'm looking at you, Shiit).

 

[Purité Audio]

You must hear equipment in your own room in your own system, compare unsighted if there isn’t an immediately apparent difference/improvement.
To go further if there isn’t a significant improvement then don’t change anything, the largest gains are speakers and room.
Keith

 

[Shadrach]

You have to start somewhere and there are imo things you should avoid.
There is absolutely no point in paying any attention to a single subjective reviews unless you have the same equipment as the reviewer, the same listening environment and the same ears.
You could try and aggregate a lot of subjective reviews but even that has pitfalls. For some reason, flavour of the month hysteria seems to be infectious.
Stay out of audio retail and consultants dens. They are there to take money off you. Any reviewer, salesman, consultant of any experience will know that whatever he sells you you will probably be delighted with as long as it works.
Objective measurement reports will at least give you an idea of the standard of performance. You don't really need to be an expert in technical reports to understand many of the measurements. A simple approach is to investigate what is and what isn't believed to be audible and anything that falls below that threshold is good enough for you to not worry about.
Set up some ABX tests with a music player and be stringent in the tests. You will probably be horrified at what you find out about your hearing acuity.
Try the same test with some equipment with some friends and then you can all be horrified together and start a disillusioned ex audiophiles group.
I used to be in such a club.
Finally you have to decide whether you want something that is measurably transparent, something that sounds good to you with your equipment in your home, or something that looks impressive and will make you feel like you're one of the boys on the audio forums.
Once you follow this path the anxiety will slowly diminish. You may even end up enjoying the music you play.

 

[Habu]

 

[garbulky]

Looks like this gentleman is talking about subjective non blind impressions. I think that's where I come in as that's what I do too. I think you are going about it the right way. Your subjective impressions are subject to bias and errors due to a lot of factors. But then again, if you use the gear like you would in real life, you would be exposed to the same thing. You may be influenced by actually great sound, perhaps you are influenced by the fantastic looks or lack thereof, who knows. But whatever you may be influenced by that's what you hear and that's what matters - what is your experience with the gear going to be like. So listen away and go with what you think would sound best to ya.

That siad reviews like these aren't useless at all. If you are interested in measurements like I am, you can use it to weigh against what you hear. For instance, I care most about the sound, but if I see a measurement like distortion peaks in to -50 db, I'm probably going to be swayed away from a peice of gear that I actually do like - unless I really like it. Sounds contradictory? Sure, but that's my choice. You do get to see if a manufacturers stated claims of distortion do add up to reality.
So for instance, I like what Schiit does but despite all the claims on this forum about their Ygdrassil measuring like crap - it doesn't actually do so if you look at the graph. It measures fine. In real world music listening, it's likely the distortion produced may be completely negligible.

It just doesn't measure as well as stuff that's a whole lot of much cheaper stuff measured here. This doesn't make it a bad dac. Just not one that will win awards on measurements.
So just from looking at that, I can see their claim of a dac with incredible resolution "21 bits, no guessing" and "mission critical" is ....eh...not really accurate. .They also talked about pitch accuracy, but looking at the measurements I see that the clock is a little more off than delta sigma designs - so little you won't hear it in the least. But still is off. So at least I know those claims don't really stand up. But in terms of sound, I still don't know if it sounds any good to me as i haven't heard it. I have heard the Gugnir Multibitl, their lower end model and I thought it sounded really nice, but lacking some specific things. So I am still interested in hearing the Yggy, but I'll also take it with a grain of salt.

Meanwhile I've also seen measurements of laptop soundcards which appear to measure better than the Ygdrassil. Now I know from mmy experience that matters not a whit because I've never liked the sound of onboard audio. Biased? Who cares. I didn't like it. So it let me know how much that measurement mamttered to me. The Oppo 205 measured extremely well. But having listened to it extensively, you couldn't pay me to take that mess. Did I not like the sound of "perfection". No I didn't like the sound - and no it certainly didn't sound like "perfection" even if it measured well, unless the real world I hear is completely distorted too. So that's where I got use from here in the measurements. What I'm saying is, you'll figure out what measurements will matter to you and what don't by going out and listening and comparing your experience to the measurements.

On the other hand, Woo Audio has really burned their bridges wit hme. I always thought Woo Audio would be a fantastic headphone amp. They had a separate chassis for the power supply and everything. The measurements said the thing distorted so badly it was well into audible territory. Now if that's their "signature sound" that's fine. But....you don't need a separate power supply to produce such a massively distorted signal. At that point, unless the thing sounds like soft velvet to me, I wouldn't be interested in shelling out the dough to buy it.

 

[beefkabob]

I want the system to sound good to me. Looking good is fine but not vital. I think my wife cares more about that. She saw a photo of the speakers and insisted that, if I get them, the grills have to stay on. My favorite for looks is McIntosh, but nobody local sells them, and they don't seem to get much respect anymore. Luxman looks good and sounds darn good too, but so much $. The Rogue amp looks like art. It's a conversation piece. I just don't like how it sounds.

I am replacing my mostly 20-year-old system. McIntosh bookshelves, Carver power amp, Chase preamp, and LG CD player. I have a $15 Amazon DAC that sounds mediocre, converting from the TV to the preamp. I have an SMSL Q5 Pro that sounds equally terrible. I think the equipment I have is old and tired. Muddy. Flabby bass. Frequencies that just seem to not be there so much. Not much detail. In many ways, but not all ways, worse than the best speakers at Frys.

I bought the Chase because it was internet flavor of the month many years ago. I bought the Q5 Pro for a similar reason and I wanted it to be small with my computer setup. I think I was going about it all wrong.

As for blind ABing or testing it all at home before purchase, that's just not possible. At least I can listen to it all in the store.

As for sound levels, through all of this, I did have volume control. Could the salesman have been adjusting sound levels somehow? I guess. He'd have to have been pretty damn slick to do it without me noticing.

 

[JJB70]

If you like it then you like it. However if looking at things rationally then measurement is a much more reliable method of comparing performance than using your ears. What measurement won't offer is anything on tactile feel and quality but that isn't a sound quality issue. For the source, DAC and amplifier you can reach the point of diminishing returns and audible transparency for peanuts these days. I think that by providing objective reviews using measurement and demonstrating how well a lot of low cost gear performs ASR is performing a great service.

 

[MrGoodbits]

I want to start a crusade that everyone interested in high quality audio reproduction should try a level-matched double blind listening test sometime in their lives. For me it was very enlightening and really helped my thinking about audio electronics. And made me a more relaxed and happier listener.

I set up these kinds of tests for myself as a young adult using an amplifier switchbox I made. I took my switchbox to a couple of friendly audio salons that had very good speakers that were set up well. I compared a 70 watt JVC receiver and an 80 watt Sansui integrated against the TOL high-end preamps and amps they had at the salons. I couldn't hear any differences (a long as we kept volumes below clipping). The salespeople thought they could but they didn't do any better than random guessing and gave up pretty quickly. To me the lack of difference was definite and, well, liberating. I had heard very clear differences in sighted tests.

tl:dr I heard all sort of differences in electronics until I did fair blind comparisons. After that, it was clear that speakers, acoustics, and source material were a better place to focus.

 

[beefkabob]

Is music rational? Not for me. My son can't play piano as quickly as a lot of people, but he plays with more emotion than most.

Also it's amazing how much better a real, tuned baby-grand sounds compared to a recording of a concert grand.

 

[JJB70]

Making music is art, designing and manufacturing audio equipment is science and engineering.

 

[Hybride]

Great post.
For over 10 years i visit a DAC designer who spend most of his lifetime designing DAC circuits. It is incredible difficult to correlate what we hear to what we are measuring. There seems even to be a unmeasurable area. Changing a single component in the circuit causes no change in measurement but can changed sound impression. It happens. Building two exact same dac's and they sound different, due tolerance in components. Our ears are incredible measuring instruments. Measuring a dac with a single frequency tells not everything about how the DAC performs playing complex music. There is still al lot to learn in design.

 

[beefkabob]

Great post.
For over 10 years i visit a DAC designer who spend most of his lifetime designing DAC circuits. It is incredible difficult to correlate what we hear to what we are measuring. There seems even to be a unmeasurable area. Changing a single component in the circuit causes no change in measurement but can changed sound impression. It happens. Building two exact same dac's and they sound different, due tolerance in components. Our ears are incredible measuring instruments. Measuring a dac with a single frequency tells not everything about how the DAC performs playing complex music. There is still al lot to learn in design.

I wouldn't think that certain things are necessarily unmeasurable. I think they're probably just terribly hard to measure. How do you measure soundstage or likelike? It's like measuring the difference between good and bad writing. Some things are more organic for sure. It's like measuring the uncanny valley.

Making music is art, designing and manufacturing audio equipment is science and engineering.

My cousin designs lenses for ground and space-based telescopes. Nothing consumer oriented. There's a tremendous amount of math involved, and he even managed to create a groundbreaking, patented model for lens design. He also said that there's still a lot of trial and error and intuition in the process. In education, my field, there's plenty of research that matches my own experience, but the research also misleads constantly. This is clear in, among other things, the textbooks written for common core. Science and engineering have bias in them too.

I want to start a crusade that everyone interested in high quality audio reproduction should try a level-matched double blind listening test sometime in their lives.

I did some A/B testing with my LG CD player going through a $15 amazon DAC vs its own built in DAC. Not blind but switching back and forth. I couldn't tell a consistent difference. It sounded mediocre both ways. Maybe the rest of the setup hid all differences? Maybe both DACs sucked?

 

[andymok]

Great post.
For over 10 years i visit a DAC designer who spend most of his lifetime designing DAC circuits. It is incredible difficult to correlate what we hear to what we are measuring. There seems even to be a unmeasurable area. Changing a single component in the circuit causes no change in measurement but can changed sound impression. It happens. Building two exact same dac's and they sound different, due tolerance in components. Our ears are incredible measuring instruments. Measuring a dac with a single frequency tells not everything about how the DAC performs playing complex music. There is still al lot to learn in design.

Too I realise there are two types of serious electronic engineer exist (forget those audiophiles), those who makes a living in music and those who doesn't, and somehow they share a radically different views in perception vs measurements.

and they call the others deaf or blind.

I do wish to listen more stories from those who making a living in music and design stuff for music making (again, serious professional music production), hear what the other end has to say about the magic.

 

[SIY]

Great post.
For over 10 years i visit a DAC designer who spend most of his lifetime designing DAC circuits. It is incredible difficult to correlate what we hear to what we are measuring.

With electronics, it's incredibly easy.

Changing a single component in the circuit causes no change in measurement but can changed sound impression. It happens. Building two exact same dac's and they sound different, due tolerance in components. Our ears are incredible measuring instruments. Measuring a dac with a single frequency tells not everything about how the DAC performs playing complex music. There is still al lot to learn in design.

You're mixing up a lot of things here and nice straw man at the end (as looking at any review here will show). There is no, and I mean NO examples of anything demonstrated to be audible which is not also easily measurable.

 

[amirm]

How do you measure soundstage or likelike?

Funny you mention that. A number of times when testing headphone amplifier I have found very large differences in soundstage. Then I realized in *every case* the channels were swapped! Remarkable how the brain interprets soundstage differently just because the channels are reversed.

Outside of that, I have listened and tested dozens and dozens of headphone amplifiers. Not once have I found a difference in soundstage. Yet every subjective review claims differences in soundstage between headphone amplifiers.

The difference between my listening tests and theirs is that I match levels and am trained in critical listening.

All of this aside, soundstage is in the music. For sound reproduction, we can modify that with different levels in each channel or delay. Both of these are very easy to measure.

On your larger question, again, all of us hear what you heard when tests are not controlled. Even knowledge of what can happen is insufficient from drawing the wrong conclusions. Our hearing is adaptive and pick what we want out of what we are hearing. So run to run variations are normal even when nothing has changed.

When I measure high amounts of distortion, I can hear their effect as psychoacoustics predicts. High frequencies are distorted the most, and details are lost.

When the differences are very small, then the audible differences are not there. Again, psychoacoustics and listening tests confirm the same.

All in all, there is no inconsistency at all. When it doubt, please doubt what you are perceiving first. Don't jump at the conclusion that the measurements must be wrong. Subjective effects that people describe cannot escape the suite of measurements I subject the gear to.

What I state is the conclusion of audio science and engineering after decades of testing. So your heading might as well say "is audio engineering/science going about it all wrong." Needless to say, research is performed for real. No way do companies and research organizations want to conclude the wrong things and mislead themselves.

Ask a loved one to test you blind. Run the test at least 10 times and see if you get 8 times right. That is the only way you know if your audio conclusions are correct.