• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

In DAC, Anything audible but unable to be measured(so far)?

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
Following the rabbit-hole links in that post (on Drop) and subsequently in SBAF there is an obvious problem with Amir's measurements... it's just that no one actually had a problem with this aspect. Before butts are hurt, let me say I'm not saying the measurements were incorrect - and certainly not that Amir doesn't know what he's doing. He, and many others here, know a hell of a lot more than I do - that's for sure.

With subjective reviews there is a level of inherent confirmation bias that is selection dependent - i.e. once 20 reviewers have said a particular device sounds exceptional (or horrid), anyone expressing the complete opposite opinion is risking the very kind of online lynching seen in the linked posts. Sure there might be a few mavericks, but the assumption is that if everyone else hears something you don't... you must be wrong. Since there is usually no shortage of subjective reviews, this bias presents itself rapidly to anyone searching for information on a product they're considering, often before the actual product even ships in volume. :rolleyes:

With objective, measurement-driven reviews - the opposite problem exists. In many (if not most) cases, the manufacturer doesn't even provide a full set of AP measured charts. Even if they do, was the sample device the same as the one reviewed on ASR? What about the rest of the chain? Couple that with a dizzying array of products, variety of testing methodologies, and poor QC seen in even high-end brands... and you are often left with results in total isolation. With nothing to compare them to, can these results be considered much more than anecdotal?

The truly optimal situation is that there are 10 reviewers like Amir, all with similar test configurations, all being sent multiple samples of a given product (one from the manufacturer, and a couple from consumers pulled from actual retail samples). In that fantasy world, there would be a fairly simple way of eliminating edge cases where a single bad unit of an otherwise stellar product was tested. It would also be trivial to determine if a particular test (or reviewer) failed in some aspect of configuration or execution - if a single reviewer consistently produced results outside the mean.

Unfortunately, objective reviews are very hard to come by - and consistently are limited to one or two samples of a product. So although reproducible results can be achieved... without the ability to confirm that the product you as a consumer purchased measures the same - you are left with many of the same doubts regarding the review and/or source.

When those doubts lead to fanaticism - positive or negative - problems can and will arise. I would still much rather have even partially flawed measurements to include in a purchasing decision, than nothing but subjective platitudes regarding how "sublime and compelling" some audio device is to someone. Just because I will allow that it's possible (though unlikely) for something to measure well but not sound good, doesn't mean the measurements are the problem. If anything it likely just means there weren't enough measurements taken, or that the sample size was too small for it to be meaningful in comparison.

While everyone is quick to yell how someone's subjective opinion is meaningless without full documentation of their ABX methodology... in other cases, a metric without confirmation - on a sample size of one - isn't necessarily viewed with the same skepticism. For some at least.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,386
Likes
24,752
Location
Alfred, NY
FWIW, Amir has always made project files available to any of us who want to try to replicate or do something similar. I've received a few of them- they're solid. In the two cases where we tested the same device, we got essentially the same results. I would not worry much about the repeatability of Amir's results.
 

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
FWIW, Amir has always made project files available to any of us who want to try to replicate or do something similar. I've received a few of them- they're solid. In the two cases where we tested the same device, we got essentially the same results. I would not worry much about the repeatability of Amir's results.

I don't worry "too much" about any of it - however your statement makes my point. Sample size is too small to be meaningful.

I'm not saying that I believe anything deceptive or incompetent has been published (and certainly not deliberately as many of the SBAF users implied). Conversely, I feel a great effort has been spent on providing transparency where possible. However, statistically it's only slightly more meaningful than no confirmation at all, surely you admit that?
 

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,588
Likes
38,291
Location
Gold Coast, Queensland, Australia
The truly optimal situation is that there are 10 reviewers like Amir, all with similar test configurations

Consider that in the heyday of HiFi, there were many highly comprehensive technical reviews of equipment all around the world being published monthly in over the counter HiFi magazines.

It was trivial to obtain multiple, independent reviews of the same piece of gear to compare. UK, US, EU and Australia were common.

I remember Pioneer would do all the hard work for us and send stapled photocopies of reviews for gear we were selling, to give or show to customers.
 

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
Consider that in the heyday of HiFi, there were many highly comprehensive technical reviews of equipment all around the world being published monthly in over the counter HiFi magazines.

It was trivial to obtain multiple, independent reviews of the same piece of gear to compare. UK, US, EU and Australia were common.

I remember Pioneer would do all the hard work for us and send stapled photocopies of reviews for gear we were selling, to give or show to customers.

Add to that the fact that there was far fewer manufacturers and products back then and it's even worse. A much different landscape than what we have today with so many boutique shops and DIY implementations available - often with one shop operating under multiple names or production facilities for the same basic device.

Another problem is that the objective measurements are less significant when they are positive than negative. If a supposedly well-engineered product measures well, it's less likely that a "lucky" sample was reviewed. In the case of a poor result, but where there aren't obvious problems in design, it's as likely that a manufacturing (or QC) failure is to blame. That's still good to know as a consumer, but it runs the risk of throwing the baby out with the bathwater in some cases.
 

JohnYang1997

Master Contributor
Technical Expert
Audio Company
Joined
Dec 28, 2018
Messages
7,175
Likes
18,292
Location
China
Add to that the fact that there was far fewer manufacturers and products back then and it's even worse. A much different landscape than what we have today with so many boutique shops and DIY implementations available - often with one shop operating under multiple names or production facilities for the same basic device.

Another problem is that the objective measurements are less significant when they are positive than negative. If a supposedly well-engineered product measures well, it's less likely that a "lucky" sample was reviewed. In the case of a poor result, but where there aren't obvious problems in design, it's as likely that a manufacturing (or QC) failure is to blame. That's still good to know as a consumer, but it runs the risk of throwing the baby out with the bathwater in some cases.
More simple to put, if it measures badly, why bother. There are many more that measure well. If that's unfair to the manufacturer, then the manufacturer can always provide a separate sample just like benchmark did.
 

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,588
Likes
38,291
Location
Gold Coast, Queensland, Australia
If a supposedly well-engineered product measures well, it's less likely that a "lucky" sample was reviewed.

The entire 'lucky sample' thing is much less of an issue that people think in my opinion.

Back in the analog days, where a huge amount of manual setup was required (think tape decks, CD players, turntable and cartridge variations etc) of course sample to sample variation and performance was a thing.

Amplification has been virtually two pot adjustment (on the whole) for decades- (bias and offset) and variations are rare. Even less so now, with much of Class D being adjustment free (excluding hypex).

I've taken (multiple over the years) samples of the same model integrated amplifiers made in the early 1970s and they have tested identically with original published reviews from the time.
 

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,588
Likes
38,291
Location
Gold Coast, Queensland, Australia
Add to that the fact that there was far fewer manufacturers and products back then and it's even worse.

I would say the complete opposite. We had fewer manufacturers, but those manufacturers had absolutely enormous ranges of product.

In a given year, for instance, Technics (Matsushita) may have had around 30 different turntables, perhaps 20 amplifiers, 15 receivers, 20 cassette decks, 20 CD players. Sony may have had the same, as did Akai, Denon, Sansui, Luxman, Teac, Sanyo, Hitachi, Pioneer, Yamaha, Accuphase etc.

Now we have myriad manufacturers making a few products each. No comparison.

As a random example, I just pulled a 1983 Technics range brochure for Oceania (Aust):
24 pages
3 power amplifiers
3 preamplifiers
11 integrated amplifiers
1 CD player (it was 1983)
1 digital tape recorder
1 digital video tape recording system (audio)
7 tuners
3 receivers
5 graphic eqs
1 audio frequency analzyer
22 turntables
2 turntable bases
3 dedicated arms
16 cartridges
15 cassette decks
3 open reel decks
4 microphones
21 pairs of speakers
8 complete component systems
3 audio racks
4 pairs of headphones

Phew. One manufacturer. One region. One year. And none of that was the really expensive home market stuff or the gear that went to Germany and Europe that year.
 
Last edited:

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
The entire 'lucky sample' thing is much less of an issue that people think in my opinion.

You misunderstand (or I poorly expressed it). I'm saying it's almost completely a non-issue. The issue is with an 'unlucky sample' instead.
Given the state of manufacturing today - and excluding DIY assemblies - I find it extremely unlikely for a company with good engineering to have horrible manufacturing. However, mistakes still occur even in the best facilities - especially where QC is lacking or spot-checking is too sparse.

The odds of a bad company making a lucky sample are miniscule in comparison to a good company having a single defect falling through the cracks and winding up on the reviewer's table. The Benchmark example is a perfect one @JohnYang1997
 

JohnYang1997

Master Contributor
Technical Expert
Audio Company
Joined
Dec 28, 2018
Messages
7,175
Likes
18,292
Location
China
Spatial imaging

When a musical performance is recorded, the instruments/ voices are recorded either with one mic (with each having a different position relative to that mic) or several mics. A hypothetical listener listening to that performance live will be located away from the mics and will have a different perception of the music spatially based on his/ her position relative to the sound emitters.

The challenge then would be for the sound engineer to master the recording so that it can render the most pleasing sound spatially for the listener; and for the audio equipment that plays the sound to reproduce that 3d sound field with fidelity.

How do you measure that?
There are measurements. And there are solutions. It's just it's not common.
It's all inside the impulse response. It's just we don't know how to process the impulse response the way our brain does but the impulse response is there. And we can use it.
About the recording, tho Paul in psaudio is not well regarded here and he does believe something that isn't necessarily true but according to him all the information prior to the microphone are already recorded including the spatial information. Then rotate the microphones and you get speakers to reproduce what's recorded. In good systems there are depth and good imaging.
And a different solution is to use dummy head and earphones. The results are really good. Just no one wants to do it or it's not beneficial in general.
There is difference between interpretation and measurement. We can already measure everything. Issue is do we really do that or how to interpret. There are many many different tests. Do we do them all? Where is the point that no more measurements are necessary. That's the question.
 

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
I would say the complete opposite. We had fewer manufacturers, but those manufacturers had absolutely enormous ranges of product.

In a given year, for instance, Technics (Matsushita) may have had around 30 different turntables, perhaps 20 amplifiers, 15 receivers, 20 cassette decks, 20 CD players. Sony may have had the same, as did Akai, Denon, Sansui, Luxman, Teac, Sanyo, Hitachi, Pioneer, Yamaha, Accuphase etc.

Now we have myriad manufacturers making a total of half a dozen products. No comparison.

That's not the opposite... it's what I was saying. Despite there being "30 turntables" in a single product line... it was still a single company with (likely) a single set of engineers designing for it. Much different than 30 companies making 1 turntable each - with 30 separate designers. The "more products" simply refers to the fact that there are not only more specific products (ex: "amp" or "speaker") on the market today than 40 years ago... there are more categories of audio products - many new ones were created (ex: "network streamer"), yet with few exceptions - all the old ones still remain.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,386
Likes
24,752
Location
Alfred, NY
You misunderstand (or I poorly expressed it). I'm saying it's almost completely a non-issue. The issue is with an 'unlucky sample' instead.
Given the state of manufacturing today - and excluding DIY assemblies - I find it extremely unlikely for a company with good engineering to have horrible manufacturing. However, mistakes still occur even in the best facilities - especially where QC is lacking or spot-checking is too sparse.

If "qc is lacking or spot-checking is too sparse" then this is not one of the "best facilities." At the price of much gear these days, there's no excuse for less than 100% outgoing checks.

As a matter of prioritizing getting things right, if components I measure for review give unexpectedly poor results, I'll ask the manufacturer if this is normal.
 

maxxevv

Major Contributor
Joined
Apr 12, 2018
Messages
1,872
Likes
1,964
Hi, I came across a post claiming ASR is a "cult" just now Cult of Audio 'Science' Review - Amir's Faithful

Amir's response to this guy was great. Being a guy who has some Audiophile friends but usually find their way of evaluating equipments unconvincing ---- I'm quite used to hearing things just like in the link above. however... I've been thinking about this question for a year or two now. Let me make a hypothetical example for you.

If there are two DAC and both have perfect measurement, say 0% THD and also good score in everything else that's been used to test DACs.

Is it possible that there are ANY kind of difference in how they sound like?

Is it possible that some audible difference only occurs under certain circumstances?(such as when amps clip, manufacturers usually don't give measurements about amps in Clipping Status, and I guess there's no widely accepted way to measure this anyway) Like we know amps MAY sound differently when they clip. But for DAC, anything like this?

Are the methods we have been using to test audio equipments a GOOD, THOROUGH and EXHAUSTING way of simulating these equipments' normal working condition?

Discussion welcome.


In practice its possible. IF the output performance in areas such as power response are different.

You will pick up differences in output if one unit can do say 2W output and the other say 5W and you are loading the system with a 2.5W transducer at close to maximum power required and with a very low efficiency.

So if you need the transducer to operate at very close to its rated power , unit with the 2W output will likely experience some sort of clipping at different times of outs output compared to the 5W unit.

If the transducer was one that required 1W of power and that both units are operating at a level where their power output curves are linear across the whole frequency spectrum, then there is not likely that you will pick up any differences.

Most of the time, we "hear"differences between different DACs not because that they are different in their signals at the chip level, but rather the output level is not linear and not output level matched between different units. Differences are usually a result of power clipping or different output levels.

Let me put it this way. Everything can be measured, but not everything are interpretable yet. Though we can use things we can understand and measure to overcome the things that are not yet interpretable. If we shoot for perfection, we can achieve greatness.

The physicists working at CERN and FERMI Labs would certainly not agree with that.

"Dark Mark" has been interpreted with observation data, but they have not been able to quantifiably measure it with instrumentation.

"Quantum Entanglement" has been instrumentally demonstrated and observed but they have not been able to quantify it either.
 

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
If "qc is lacking or spot-checking is too sparse" then this is not one of the "best facilities." At the price of much gear these days, there's no excuse for less than 100% outgoing checks.

As a matter of prioritizing getting things right, if components I measure for review give unexpectedly poor results, I'll ask the manufacturer if this is normal.
So by that logic would you say that Benchmark is not a "best facility" in light of the DAC3 review here? Certainly I would have to put Denon/Marantz in that bucket as I received a problematic unit from them before (non-functional network connection).

I'm confused regarding your second point... how is that possible if your first point is immutable? Either it's not right - in which case, they are provably a "bad manufacturer" by your first point... or it is, in which case they intentionally released a bad product.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,524
Likes
37,057
I don't worry "too much" about any of it - however your statement makes my point. Sample size is too small to be meaningful.

I'm not saying that I believe anything deceptive or incompetent has been published (and certainly not deliberately as many of the SBAF users implied). Conversely, I feel a great effort has been spent on providing transparency where possible. However, statistically it's only slightly more meaningful than no confirmation at all, surely you admit that?
No.

What is the standard deviation of any measurement in manufactured gear? In general it is very very small. Is it possible for there to be outliers or for poorly QC'd gear to vary more than normal? Yes. In general the difference between particular samples is terribly small. Knowing that, knowing how modern manufacturing works we don't need to test 30 of everything. There is the chance to get tripped up, but it is very small.

If we had wide experience, and ample data that anecdotal reports by audiophiles were 99.9% the same and in some instances this had been confirmed more formally we could get by without being skeptical of those reports. It is the wide variance and the numerous interfering factors that means we have to be highly skeptical of individual human reports.

I've measured some things Amir has, and gotten within the ability of my gear identical results. There are at least a few others here who have as well. Amir has tested several pieces of gear that have been tested elsewhere like Stereophile and again differences were way down in the weeds.

To paint this idea that his measuring one bit of gear is no better than a single anecdotal report is disingenuous and misleading in the extreme.
 

FireLion

Active Member
Joined
May 25, 2018
Messages
243
Likes
98
Can measurements tell you of bass is more weighty, soundstage is wider or imaging is sharper?
I believe different circuit topology, proprietary technology, fabrication processes, composition, chips use can make for different flavors of sound.

Is it measurable? Not sure but I think the human ear can discern much more than a measurement rig.
 
Last edited:

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,386
Likes
24,752
Location
Alfred, NY
So by that logic would you say that Benchmark is not a "best facility" in light of the DAC3 review here?
No, that was yours. The words I quoted came out of your post, your assumptions. I know nothing about Benchmark's QC or manufacturing systems.

I have no idea what you're talking about with respect to me checking with manufacturers.
 

digicidal

Major Contributor
Joined
Jul 6, 2019
Messages
1,981
Likes
4,838
Location
Sin City, NV
No.

I've measured some things Amir has, and gotten within the ability of my gear identical results. There are at least a few others here who have as well. Amir has tested several pieces of gear that have been tested elsewhere like Stereophile and again differences were way down in the weeds.

To paint this idea that his measuring one bit of gear is no better than a single anecdotal report is disingenuous and misleading in the extreme.

You are correct. However, once again your defense is to indicate correlation to other reviews and your own testing results. That is the exact opposite of the circumstance I described - i.e. one in which there is no other data available. In each specified example you indicate multiple samples are involved.

I'll just give up already. Single unit sample measurements are obviously without reproach here... even in cases where Amir retests and subsequently updates his measurements or methodologies after some feedback. I believe Amir is both competent and highly intelligent... but I don't believe that necessarily translates to infallible or omniscient.

I'm starting to see some of the cult here after all I think.

No, that was yours. The words I quoted came out of your post, your assumptions. I know nothing about Benchmark's QC or manufacturing systems.

I have no idea what you're talking about with respect to me checking with manufacturers.
Right back at ya.
...I'll ask the manufacturer if this is normal.
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
We are not claiming nor is it our mission to invent new science , amir just tests hifi gear.

It's all well documented, reliable and repeatable if one has the equipment.

If one thinks its value is so limited for whatever reason they best seek other indicators of what might suit them for a bit of hifi. In a ideal world all manufacturers would provide this data and then it could be cross checked by individuals in the field.

As a aside , my brother in law was recently convicted of driving while unfit. Convicted on a pee sample take in non sterile surroundings and tested by just one lab... The horror!

That was good enough for the court, as is VADA testing in sport.

I guess my Bother in laws legal council missed a trick by not arguing the illegitimacy of a single sample size.
 
Top Bottom