• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Why Audio Science Review's measurements are so different compared to other sources?

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
Sorry for not being specific, but I found something with a quick Google search (if I am not exact or make a mistake in regards to something, just let me know)

This is what Amir got from the Yggdrasil in regards to FR:
View attachment 71202

This is what Jude got:
View attachment 71203

In term of harmonic distortion and (possible) mains hum:

Amir:
View attachment 71204

prr1n from SBAF:
View attachment 71206

I would like to reiterate that if I make a mistake, I wrongly interpret something, or I miss some information, I would like to be corrected where necessary and thanks for your time.
How familiar are you with measurement setup?

At minimum, you have to ask:
  • What tools/fixtures?
  • What input/output?
  • What electrical/digital level?
  • What test signal?
  • What test bandwidth?
  • What FFT size?
For example, in the final graph the analog level is -10dBu. That's 0.25V. There is no question that distortion would be lower in that case. FFT size is also enormous seeing how low the graph goes.

Between Jude and Amir, there is complete concurrence in the balanced FR graph as well.

Also, have you seen this article? https://www.audiosciencereview.com/...ts-of-schiit-yggdrasil-dac-inconsistent.3812/
 

Inner Space

Major Contributor
Forum Donor
Joined
May 18, 2020
Messages
1,285
Likes
2,938
I also have the tendency of not taking any view at face value.

Healthy skepticism is essential, of course. But some folks (not you, necessarily) use a facade of skepticism to avoid engagement with reality. Like the inverse of "everything matters". Truth is, not everything matters, and sooner or later skepticism will be satisfied if a person digs in and does the work.
 

Jimbob54

Grand Contributor
Forum Donor
Joined
Oct 25, 2019
Messages
11,066
Likes
14,697
When researching for measurements I have seen multiple websites and sources that provide these. This is one of those websites directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there has been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations.

Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in Head-Fi Sound Science subforum to know their opinion about this issue.
PS(2): I want to reiterate that this is not an attack on the forum and its creator, this is an honest question that has been bugging me since I discovered this website.
PS(3): Do not direct me to the thread he explains how he measures and with what equipment, I already read it multiple times. I want to know the reasons the differences arise from different subjects and their measurements.

Question for you. Did you see measurements of the Asgard 3 DAC module before you bought it? If so, do those measurements reconcile to Amirs very recent review that showed the DAC to be , shall we say, outdated at 90ish dB SINAD?
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
Healthy skepticism is essential, of course. But some folks (not you, necessarily) use a facade of skepticism to avoid engagement with reality. Like the inverse of "everything matters". Truth is, not everything matters, and sooner or later skepticism will be satisfied if a person digs in and does the work.
Ever read Wittgenstein? Paraphrasing very loosely: We are familiar for examining our grounds for certainty, but not for doubt. Grounds for doubt may be lacking, too.

/philosophy
 
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
Healthy skepticism is essential, of course. But some folks (not you, necessarily) use a facade of skepticism to avoid engagement with reality. Like the inverse of "everything matters". Truth is, not everything matters, and sooner or later skepticism will be satisfied if a person digs in and does the work.
Exactly, I try to find the truth, and if not possible, the closest we can get to it by the use of repeatable evidence and methods.

By the way, after Amir published the measurements for the Asgard 3 and its AK4490 implementation, this is the new DAC I am using:
51aBovrC1rL._AC_SL1000_.jpg


Rather not taking the risk of distortion. The included DAC (CX21988) should not measure bad since this is not a bespoke audiophile DAC. Using my Asgard as amplifier only now (with the benefit of a longer cable run).
 
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
Question for you. Did you see measurements of the Asgard 3 DAC module before you bought it? If so, do those measurements reconcile to Amirs very recent review that showed the DAC to be , shall we say, outdated at 90ish dB SINAD?
Actually not, they were not available. I saw Schiit's AP report on it (which does not include measurements for both cards), but this report only addressed the amplifier (which is good).
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
Exactly, I try to find the truth, and if not possible, the closest we can get to it by the use of repeatable evidence and methods.

By the way, after Amir published the measurements for the Asgard 3 and its AK4490 implementation, this is the new DAC I am using:
View attachment 71213

Rather not taking the risk of distortion. The included DAC (CX21988) should not measure bad since this is not a bespoke audiophile DAC. Using my Asgard as amplifier only now (with the benefit of a longer cable run).
Aahh. Remember that DAC chip /= full DAC device. There's a lot of analog stuff to consider.

Go here and filter DACs by chip: https://www.audiosciencereview.com/forum/index.php?pages/Audio_DAC_Performance_Index/

The results are interesting to say the least.
 
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
Aahh. Remember that DAC chip /= full DAC device. There's a lot of analog stuff to consider.

Go here and filter DACs by chip: https://www.audiosciencereview.com/forum/index.php?pages/Audio_DAC_Performance_Index/

The results are interesting to say the least.
I know it isn't a full DAC, but compared to Schiit previous record, I think it can work until I get a cheap full DAC. Also, I don't even find the spec sheet for that chip, but I think it is somewhat safe to assume that it measures reasonably well (or that's what I want to believe for now).

Edit: The CX21988 was THX-certified and tested at their facilities, so it is safe to assume that measures well.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825

LTig

Master Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
5,760
Likes
9,442
Location
Europe
How familiar are you with measurement setup?

At minimum, you have to ask:
  • What tools/fixtures?
  • What input/output?
  • What electrical/digital level?
  • What test signal?
  • What test bandwidth?
  • What FFT size?
  • What load for the output?
  • What source impedance for the input
  • What number of averages for FFT?
 
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
  • What load for the output?
  • What source impedance for the input
  • What number of averages for FFT?
How many questions? It seems that it is hard to troubleshoot measurements.
 

Inner Space

Major Contributor
Forum Donor
Joined
May 18, 2020
Messages
1,285
Likes
2,938
Ever read Wittgenstein? Paraphrasing very loosely: We are familiar for examining our grounds for certainty, but not for doubt. Grounds for doubt may be lacking, too.

Yes, I have, and I just read a book about Frank Ramsey, a math prodigy who was commissioned to do the English translation of the Tractatus, at the age of 18. (Ramsey died at 26, but his work lives on.) I hadn't realized the Wittgenstein family was immensely rich. Good old Ludwig gave his portion away and lived very simply.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
Yes, I have, and I just read a book about Frank Ramsey, a math prodigy who was commissioned to do the English translation of the Tractatus, at the age of 18. (Ramsey died at 26, but his work lives on.) I hadn't realized the Wittgenstein family was immensely rich. Good old Ludwig gave his portion away and lived very simply.
Continuing that line of thought, I always liked the fact that Ravel wrote his Piano Concerto for Left Hand for Ludwig's brother Paul, who lost in right arm in WWI.
 

Inner Space

Major Contributor
Forum Donor
Joined
May 18, 2020
Messages
1,285
Likes
2,938
Continuing that line of thought, I always liked the fact that Ravel wrote his Piano Concerto for Left Hand for Ludwig's brother Paul, who lost in right arm in WWI.

Yes ... one of several sad, awful, tragic compositions for maimed musicians of that era. What a world.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
How many questions? It seems that it is hard to troubleshoot measurements.
It depends on the specific measurement being made.

The last simple one is weighting (A, B, C, Z, and so on), which is like a filter applied to a regular measurement:
1593465983046.png
 

bogart

Active Member
Forum Donor
Joined
Sep 10, 2019
Messages
248
Likes
304
How many questions? It seems that it is hard to troubleshoot measurements.

May be a rough choice of words when you say "troubleshooting" the measurements. Amir's documented test parameters that have been mentioned in the above posts will generally push devices to the edges of their performance envelope. By testing closer to limits, the tests attempt to document performance vs. claimed specification, vs. testing standards, or vs. other tested devices.

As @pozz and @LTig point out, each of these small choices add up, but an advantage of the ASR approach is that Amir's tests have been heavily criticized by a knowledgeable community that doesn't shut up until they're generally satisfied. Paraphrasing, of course :) Enough other folks on here are knowledgeable and experienced that deviations from good practice get called out and debated to death. The test protocol has therefore been effectively peer reviewed, and although some dissent remains on a very few points (e.g. where to measure the ultimate output power of an amp - the 'knee' of the chart or a fixed % distortion, etc.), the tests have been tested well across lots of devices.

For that reason, and I'd say somewhat uniquely here, is that 'open science' is being done. You don't necessarily need to become independently expert in all the tests to understand that they are appropriately designed tests with full transparency to test design.
 
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
I just bought an S.M.S.L Sanskrit 10th MK II, saw the measurements, and some usability reviews, and it was enough to give it a try. Measures beautifully and it is small enough. The DAC ranking @pozz provided is really useful for filtering results by multiple categories. Thanks for your help.
 

Ron Texas

Master Contributor
Joined
Jun 10, 2018
Messages
6,075
Likes
8,908
I don't see anything wrong with the measurements here which are done on state of the art measurement devices. If they are different elsewhere, other people have a problem.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
May be a rough choice of words when you say "troubleshooting" the measurements. Amir's documented test parameters that have been mentioned in the above posts will generally push devices to the edges of their performance envelope. By testing closer to limits, the tests attempt to document performance vs. claimed specification, vs. testing standards, or vs. other tested devices.

As @pozz and @LTig point out, each of these small choices add up, but an advantage of the ASR approach is that Amir's tests have been heavily criticized by a knowledgeable community that doesn't shut up until they're generally satisfied. Paraphrasing, of course :) Enough other folks on here are knowledgeable and experienced that deviations from good practice get called out and debated to death. The test protocol has therefore been effectively peer reviewed, and although some dissent remains on a very few points (e.g. where to measure the ultimate output power of an amp - the 'knee' of the chart or a fixed % distortion, etc.), the tests have been tested well across lots of devices.

For that reason, and I'd say somewhat uniquely here, is that 'open science' is being done. You don't necessarily need to become independently expert in all the tests to understand that they are appropriately designed tests with full transparency to test design.
It's really instructive to look up the oldest reviews and read forwards if anyone wants to see how it was: https://www.audiosciencereview.com/forum/index.php?pages/Audio_Equipment_Reviews/

It's always given me a chuckle that in the very first review (the "ground box") the discussion is very skeptical on every single point, even similar in tone/vocabulary to what was found on other forums. Measurement specifics were not used to be being analyzed. That quickly progressed.

Edit: For reference: https://www.audiosciencereview.com/...ignal-grounding-preliminary-measurements.476/
 
Last edited:
Top Bottom