• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). Come here to have fun, be ready to be teased and not take online life too seriously. We now measure and review equipment for free! Click here for details.

Why Audio Science Review's measurements are so different compared to other sources?

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
#1
When researching for measurements I have seen multiple websites and sources that provide these. This is one of those websites directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there has been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations.

Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in Head-Fi Sound Science subforum to know their opinion about this issue.
PS(2): I want to reiterate that this is not an attack on the forum and its creator, this is an honest question that has been bugging me since I discovered this website.
PS(3): Do not direct me to the thread he explains how he measures and with what equipment, I already read it multiple times. I want to know the reasons the differences arise from different subjects and their measurements.
 

GXAlan

Senior Member
Forum Donor
Joined
Jan 15, 2020
Messages
468
Likes
648
#2
1) 2V and 4V for unbalanced and balanced inputs.
This is like testing 0 to 60 or 1/4 mile performance. Why 0-60 instead of 0-55 or 1/4 mile instead of 1 mile? The rationale for 2V and 4V is well documented HOWEVER a lot of companies choose not do to this and criticize Amir for being too strict. The latest reviews show performance at multiple voltages. The Denon X3500 actually does better with lower voltage. The Onkyo pre-amp did not.

This also leads to testing the Denon at 82.5 instead of 80 or 70 volume. This matters for the SINAD chart only.

2) Testing with more sensitive equipment and more difficult equipment
His AP is more sensitive which makes the very best (like Mola Mola DACs) measure even better than the standard gear.
He also uses HDMI from a HTPC which is probably noisier or worse than something like a HDMI test generator or perhaps a Panasonic UB9000 or Oppo 205.

3) Apples vs. Oranges comparisons*
Testing DACs vs. AVRs
Testing $5000 pre-amps against $1000 receivers.

The X3600h beats several ultra-expensive HT pre-amps, so it's not an unfair comparison and new Chinese made DACs outperform vintage Mark Levinson DACs. But this is a much higher standard than others IMHO.
 

SIY

Technical Expert
Technical Expert
Joined
Apr 6, 2018
Messages
4,606
Likes
9,133
Location
Phoenix, AZ
#5
When researching for measurements I have seen multiple websites and sources that provide these. This is one of those websites directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there has been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations.

Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in Head-Fi Sound Science subforum to know their opinion about this issue.
PS(2): I want to reiterate that this is not an attack on the forum and its creator, this is an honest question that has been bugging me since I discovered this website.
PS(3): Do not direct me to the thread he explains how he measures and with what equipment, I already read it multiple times. I want to know the reasons the differences arise from different subjects and their measurements.
I can’t speak for others, but I’ve had no trouble reproducing Amir’s measurements when I’ve had a shot at the same gear. And as I’ve noted before, Amir is the only reviewer besides me who offers to share his AP project files.

One other observation: the one time I posted honest measurements with full disclosure of methods at Head Fi, my posts were edited and removed because of embarrassment to an advertiser. I never posted there again.
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #6
Sorry to not be able to provide specific at this moment, I am at work right now and not able to look for examples.
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #7
I can’t speak for others, but I’ve had no trouble reproducing Amir’s measurements when I’ve had a shot at the same gear. And as I’ve noted before, Amir is the only reviewer besides me who offers to share his AP project files.

One other observation: the one time I posted honest measurements with full disclosure of methods at Head Fi, my posts were edited and removed because of embarrassment to an advertiser. I never posted there again.
Weird, it seems that there are interests between parties, but I am not able to conclude who is involved in this and what interests they may have. Many posts with divergent views in different sites (especially the Yggdrasil thread) have created some distrust of all measurements.
 

Frank Dernie

Major Contributor
Forum Donor
Joined
Mar 24, 2016
Messages
3,443
Likes
6,770
Location
Oxfordshire
#10
When researching for measurements I have seen multiple websites and sources that provide these. This is one of those websites directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there has been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations.

Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in Head-Fi Sound Science subforum to know their opinion about this issue.
PS(2): I want to reiterate that this is not an attack on the forum and its creator, this is an honest question that has been bugging me since I discovered this website.
PS(3): Do not direct me to the thread he explains how he measures and with what equipment, I already read it multiple times. I want to know the reasons the differences arise from different subjects and their measurements.
I think any web site publishing measurements but accepting sponsorship from manufacturers has less incentive to be unbiased than Amir.
 

Canuck57

Active Member
Forum Donor
Joined
Jan 20, 2019
Messages
123
Likes
138
Location
Fergus, ON Canada
#11
I think one has to consider if there's a conflict of interest with head-fi. As far as I know they receive advertising revenue from many vendors and therefore it's in Head-fi's (Jude's) interest not be be fully transparent with measurement. Don't bite the hand that feeds you; similar to Stereophile.

Amir is very open with regard to his testing philosophy and has no financial interest related to his measurements/reviews, and besides who can't trust a Pink Panther!
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #12
Without specific examples it's just a vage impression from your side.
Sorry for not being specific, but I found something with a quick Google search (if I am not exact or make a mistake in regards to something, just let me know)

This is what Amir got from the Yggdrasil in regards to FR:
a1.png


This is what Jude got:
Ju1.jpg


In term of harmonic distortion and (possible) mains hum:

Amir:
a2.png


prr1n from SBAF:
Yaggy 1kHz BAL-1.jpg


I would like to reiterate that if I make a mistake, I wrongly interpret something, or I miss some information, I would like to be corrected where necessary and thanks for your time.
 

Attachments

Hemi-Demon

Active Member
Joined
Jan 17, 2019
Messages
188
Likes
168
#13
Talk about a disingenuous well written thread, simply meant to pit forum against forum. This has been done many times over, and explained so many times. I swear the Yggy threads always create such a fit. You really have to sit and read BOTH reviews and then see Jude's measurements and you can see how the scales on the graphs are different and what each reviewer finds as "ok" is just personal slant. What is wrong with that. If you want a device that can't do DSD, is barely 20 bit capable, has high distortion by design (shown in all reviews), has power main issues (shown by each reviewer) and allegedly requires a week to stabilize, then buy it. Most subjective reviews seem to like the $2,400 device. The Yggy is designed to be warm, and Schiit and most of the users, WANT the elevated 2nd order harmonics.

Input voltage, measuring equipment, testing methodology, hidden software upgrades (see Head-fi on how certain review units were upgraded-wink wink), paid bias (do you know how much money Head-fi makes per month for sponsorship) and the skill of the person using the same equipment will always vary. All of this has been explained over and over again. Amir is one person, just like all the other reviewers. No one person makes the standard. If you disagree, that is fine, just take the time to either read all the threads, or at least produce your own counter with detailed science.

When Amir has made a mistake, he gets mad, but guess what he has engaged with people, changed things MULTIPLE times, or at minimum allowed people to continue to explain why they think he got it wrong, instead of banning them (see SBAF and Head-fi).

I mean c'mon he bought a AP out of his own pocket. measured the Yggy multiple times, from multiple owners, different devices. Guess what Schiit did, bought an AP and started improving their manufacturing standards? Now look at the last few devices reviewed here, and how improved they are? We all win.

I bet the next Yggy, will measure far better than the current one, I mean look at the Modius, Asgard 3 and Heresy. No way these devices would have existed a few years ago.

Nobody is perfect, but no need to try and create fights on the internet. Audio forums are not gangs.
 

tmtomh

Senior Member
Joined
Aug 14, 2018
Messages
427
Likes
1,015
#14
Sorry for not being specific, but I found something with a quick Google search (if I am not exact or make a mistake in regards to something, just let me know)

This is what Amir got from the Yggdrasil in regards to FR:
View attachment 71202

This is what Jude got:
View attachment 71203

In term of harmonic distortion and (possible) mains hum:

Amir:
View attachment 71204

prr1n from SBAF:
View attachment 71206

I would like to reiterate that if I make a mistake, I wrongly interpret something, or I miss some information, I would like to be corrected where necessary and thanks for your time.
Those measurements do not seem generally inconsistent with each other. The 2kHz distortion spike is about 90dB down from the 1kHz signal in both sets of measurements, and the balanced frequency response is similar. There is a difference in the measured balanced frequency response from about 18kHz to 20kHz, and in one measurement the 3rd harmonic (3kHz) is higher than the 2nd whereas in Amir's it is lower. But I don't see any major contradictions.
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #15
no need to try and create fights on the internet
I am with you up to this part. I am not trying to create fights. I used the Yggdrasil because it was the start of my doubts regarding not just Amir's measurements, also everyone's. I have an actual question and inquiry that I want to answer by asking others with more knowledge than me what they think about the evidence presented by multiple parties. Actually, there are various other devices that I don't recall of my head right now that the measurements not just done by Amir, also done by others, have been in disagreement and the deltas are sufficiently large thus cannot be attributed to manufacturing variations.

TLDR: I actually have an honest question and I don't want to bash anybody. I know there a lot of factors and I would like to know which could be affecting the different results.
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #16
Those measurements do not seem generally inconsistent with each other. The 2kHz distortion spike is about 90dB down from the 1kHz signal in both sets of measurements, and the balanced frequency response is similar. There is a difference in the measured balanced frequency response from about 18kHz to 20kHz, and in one measurement the 3rd harmonic (3kHz) is higher than the 2nd whereas in Amir's it is lower. But I don't see any major contradictions.
Well, to my inexperienced eye there are some differences compared to the scale of precision you get from the measurement equipment. These are not optimal examples, thus, please forgive my ignorance.
 

Inner Space

Active Member
Joined
May 18, 2020
Messages
206
Likes
317
#17
Those measurements do not seem generally inconsistent with each other.
I agree. When I worked in studios, I would meet manufacturers socially now and then, and polite conversation would often include questions like, "When is the Mk 2 coming out?" More often than not, the answer would include, "Man, we'll get to it. But it's hard enough to make today's Mk 1 the same as yesterday's." Therefore unit-to-unit sample variation is always an issue, perhaps (speculating here) more pronounced with manufacturers like the one referenced, which doesn't strike me as the most organized or meticulous in the world.
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #18
I agree. When I worked in studios, I would meet manufacturers socially now and then, and polite conversation would often include questions like, "When is the Mk 2 coming out?" More often than not, the answer would include, "Man, we'll get to it. But it's hard enough to make today's Mk 1 the same as yesterday's." Therefore unit-to-unit sample variation is always an issue, perhaps (speculating here) more pronounced with manufacturers like the one referenced, which doesn't strike me as the most organized or meticulous in the world.
Maybe it's just me not knowing how to read graphs correctly. I also have the tendency of not taking any view at face value.
 

LTig

Major Contributor
Forum Donor
Joined
Feb 27, 2019
Messages
2,011
Likes
2,956
Location
Europe
#19
Sorry for not being specific, but I found something with a quick Google search (if I am not exact or make a mistake in regards to something, just let me know)

This is what Amir got from the Yggdrasil in regards to FR:
View attachment 71202

This is what Jude got:
View attachment 71203
The second graph does not specify which output was measured.
In term of harmonic distortion and (possible) mains hum:

Amir:
View attachment 71204

prr1n from SBAF:
View attachment 71206
There are several differences how the test has been setup and the results are presented:
  • @amirm measured at an analog output voltage of 4.2 V rms which is 12.5 dBV (as shown in the FFT). SBAF measured at -10 dBU which is 0.25V rms and -12.2 dBV, about 17 times lower than @amirm's choice. Of course distortion products are lower then, but still far from SOTA.
  • The noise floor of the SBAF graph is some 10 to 15 dB lower than @amirm's graph. SBAF must have used a higher FFT size and probably also more averages. Both settings hide the noise.
  • The SBAF graph starts at 500 Hz so it does not show the bulk of the PS hum (starting at 50/60 Hz). The 600Hz peak (H 10 of 60 Hz or H5 of 120 Hz) is not there, but we know that PS hum may result from the specific setup (cables, grounding, ...).
 
OP
KeithPhantom

KeithPhantom

Active Member
Forum Donor
Joined
May 8, 2020
Messages
209
Likes
142
Thread Starter #20
The second graph does not specify which output was measured.

There are several differences how the test has been setup and the results presented:
  • @amirm measured at an analog output voltage of 4.2 V rms which is 12.5 dBV (as shown in the FFT). SBAF measured at -10 dBU which is 0.25V rms and -12.2 dBV, about 17 times lower than @amirm's choice. Of course distortion products are lower then, but still far from SOTA.
  • The noise floor of the SBAF graph is some 10 to 15 dB lower than @amirm's graph. SBAF must have used a higher FFT size and probably also more averages. Both settings hide the noise.
  • The SBAF graph starts at 500 Hz so it does not show the bulk of the PS hum (starting at 50/60 Hz). The 600Hz peak (H 10 of 60 Hz or H5 of 120 Hz) is not there, but we know that PS hum may result from the specific setup (cables, grounding, ...).
Thank you for the clarification. I really appreciate it.
 
Top Bottom