Should ASR bother testing stuff with mediocre manufacturer specs?

[pwjazz]

Every now and then I come across a review like this AVR review which essentially proves that a device with mediocre specs produces mediocre measurements. Honest question--is there any value to this kind of review? Seeing measurements for devices with missing or incomplete manufacturer specs seems useful because it fills in missing information, and seeing measurements for devices with good looking manufacturer specs is useful because it confirms whether the device is in fact something good and whether the manufacturer's specs are to be trusted. I personally don't have any curiosity to learn more about devices whose own manufacturer states they won't perform well.

 

[pkane]

I vote "YES". Specs are not the same as measurements, even if the numbers agree. Harmonic distortion plot, power vs distortion output, multi-tone IMD charts, etc. provide a whole lot more information than seeing a single published number. And, of course, independent testing is always needed to confirm or to invalidate manufacturer's claims. So, yes.

 

[flowjm]

Amir's time is limited and therefore, given that excellent DACs are now cheap and readily available, it makes sense to be more selective about what gets reviewed. However, I'd say that the specs of a given piece of equipment not being SOTA doesn't automatically make it uninteresting.

 

[sergeauckland]

Yes, I think it is worth measuring equipment even though it doesn't have superlative specifications. Firstly, superlative specifications are not necessary for transparency, so provided a product is adequate, then it's worth measuring if the product has some interesting features, or is attractively priced.

Frankly, I find it boring to read of yet another DAC with SINAD of -110dB, I'd much rather read a review of a 100w into 8Ω amplifier with -80dB distortion which can also drive 2 ohms happily, and/or costs under $200. More comprehensive reviews of a smaller number of interesting products rather than lots of reviews of yet another DAC.
I know loudspeaker reviews are a pain to do, but that would be really interesting.

 

[Frank Dernie]

Yes, I think it is worth measuring equipment even though it doesn't have superlative specifications. Firstly, superlative specifications are not necessary for transparency, so provided a product is adequate, then it's worth measuring if the product has some interesting features, or is attractively priced.

Frankly, I find it boring to read of yet another DAC with SINAD of -110dB, I'd much rather read a review of a 100w into 8Ω amplifier with -80dB distortion which can also drive 2 ohms happily, and/or costs under $200. More comprehensive reviews of a smaller number of interesting products rather than lots of reviews of yet another DAC.
I know loudspeaker reviews are a pain to do, but that would be really interesting.

I couldn't agree more.

 

[pwjazz]

Yes, I think it is worth measuring equipment even though it doesn't have superlative specifications. Firstly, superlative specifications are not necessary for transparency, so provided a product is adequate, then it's worth measuring if the product has some interesting features, or is attractively priced.

Frankly, I find it boring to read of yet another DAC with SINAD of -110dB, I'd much rather read a review of a 100w into 8Ω amplifier with -80dB distortion which can also drive 2 ohms happily, and/or costs under $200. More comprehensive reviews of a smaller number of interesting products rather than lots of reviews of yet another DAC.
I know loudspeaker reviews are a pain to do, but that would be really interesting.

I think that's a great way of looking at it, with special emphasis on the ideas of "adequate" performance and "interesting" products. To bring it back to my original example, the RX-A1080 is a more than $1000 AVR with a stated spec of 0.06% THD (-64dB) at 110 W into 8 ohms between 20-20,000Hz. Would you consider that adequate and/or interesting?

 

[sergeauckland]

I think that's a great way of looking at it, with special emphasis on the ideas of "adequate" performance and "interesting" products. To bring it back to my original example, the RX-A1080 is a more than $1000 AVR with a stated spec of 0.06% THD (-64dB) at 110 W into 8 ohms between 20-20,000Hz. Would you consider that adequate and/or interesting?

The THD spec is certainly adequate, and if that's conservatively specified, it could be better than that. Interesting would be what the AVR does into more severe loads, whether performance is maintained or falls apart. In other words, what's the performance envelope under a wider range of conditions than just 8 ohms resistive.

S

 

[amirm]

I do turn down fair number of gear both public and in private which has mediocre specs, especially DACs.

 

[Cahudson42]

Yes, I think it is worth measuring equipment even though it doesn't have superlative specifications. Firstly, superlative specifications are not necessary for transparency, so provided a product is adequate, then it's worth measuring if the product has some interesting features, or is attractively priced.

Frankly, I find it boring to read of yet another DAC with SINAD of -110dB, I'd much rather read a review of a 100w into 8Ω amplifier with -80dB distortion which can also drive 2 ohms happily, and/or costs under $200. More comprehensive reviews of a smaller number of interesting products rather than lots of reviews of yet another DAC.
I know loudspeaker reviews are a pain to do, but that would be really interesting.

+1

 

[Xulonn]

I personally don't have any curiosity to learn more about devices whose own manufacturer states they won't perform well.

I have a brilliant idea that is an easy, zero cost solution to your problem - don't read anything that doesn't interest you...

But on a more serious note, Amir does a lot of testing - a lot more than Stereophile, and does indeed turn down offers to test "mediocre spec" gear, apparently unless there is somethng otherwise interesting about it. Much of what he tests and reviews is predicated by what ASR members and a few manufacturers people send to him for that purpose. That simple model has lead to a wonderful mix of tested and reviewed audio electronics, ranging from old to new, Chi-Fi to audiophile brands, and super-cheap through budget and up to some pretty high-end products.

My curiosity is obviously much greater in breadth than yours. I am fascinated by the wide range of audio equipment that is marketed around the world to a wonderful and broad spectrum of music lovers from many cultures audio gear hobbyists so let me engoy my wide range of curiosity and academic interest, while you can simply ignore that which you do not care about.

Also, if you want something tested, how about sending an example of that product to Amir, or convincing someone - consumer or manufacturer - to send in examples for testing? That's how ASR works, and helps to avoid a dependency on advertising revenue and the ethical problems associated with such a model.

I really like the unique, science-based ASR audio forum model because it also tolerates subjectivist opinions, but combats falsehoods, bad logic, fraud and snake oil audiophilia. Although there is always room for growth, change, and modification of details, I hope "we" can keep this forum website as interesting, active, and fun as it is today.

 

[pwjazz]

I have a brilliant idea that is an easy, zero cost solution to your problem - don't read anything that doesn't interest you

To be clear, I'm not complaining about this site's content and I'm grateful for the work that @amirm does. It was mostly just a question of whether his limited bandwidth could be better utilized, and the related observation that confirming that something with mediocre specs meets those specs isn't necessarily the most interesting result.

It's funny that you mention cost, because opportunity cost is the heart of my question (note that this was a question, not a complaint or directive or even request, just a question). I have enough time to read every review that Amir puts out. Amir on the other hand doesn't have time to review everything under the sun, so it's just a question of making the best use of his time (which I'm sure he already puts a lot of thought into, and before anybody's mind goes in that direction I'm not trying to tell him how to go about his business).

Also, if you want something tested, how about sending an example of that product to Amir, or convincing someone - consumer or manufacturer - to send in examples for testing?

I have, see https://www.audiosciencereview.com/...rements-of-nobsound-ns-dac3-pro-dac-amp.5293/

I hope "we" can keep this forum website as interesting, active, and fun as it is today

I hope you're including "me" in that "we". I think process discussions like this one can be interesting, active and fun if we try to keep them that way.

 

[q3cpma]

I'm with you, mate. No point in proving that mediocre stuff is mediocre.

 

[amirm]

The industry provides so little data that I rarely know how bad something is until I test it.

 

[Xulonn]

I hope you're including "me" in that "we".

Absolutely!

 

[DonH56]

I vote "no" based upon Amir's limited time (he's already overwhelmed with requests) and if he has more time I'd rather see more in-depth tests of what he's got than more tests of bad gear. That said, some of the gear that gets rave audiophile reviews (like NP's little kit circuit), is worth testing just to see how it stacks up IMO.

 

[VintageFlanker]

The industry provides so little data that I rarely know how bad something is until I test it.

Unsurprising. Audiofools don't care (or mostly don't understand) about specs so manufacturers don't need to show much of these....

Keep measuring bad specs-products is a way to highlight the truth to all. The poor Panther has to be beheaded more often.

So yes, it is relevant to me.

 

[Frank Dernie]

Probably the only bad bit of a review of mediocre kit, for me, is the huge traffic from new members who have bought the thing and wish to contest the results...
I end up not reading the whole thread then.

 

[restorer-john]

Specifications and test results can be wildly distant from one another. Specifications can be incomplete, very optimistic, or very conservative.

Where there is likely reader interest in the testing of a given product, I see no problem in testing anything, unless it is clearly seriously flawed from the outset. Amir already pulls the plug on 'broken' gear so as to not waste his or the readership's time.

Sometimes there is great joy in uncovering a product that flies (or flew) under the radar and offers excellent performance not hinted at in the specs.

 

[Thomas savage]

Probably the only bad bit of a review of mediocre kit, for me, is the huge traffic from new members who have bought the thing and wish to contest the results...
I end up not reading the whole thread then.

It's tiresome I agree.

 

[bravomail]

Every now and then I come across a review like this AVR review which essentially proves that a device with mediocre specs produces mediocre measurements. Honest question--is there any value to this kind of review? Seeing measurements for devices with missing or incomplete manufacturer specs seems useful because it fills in missing information, and seeing measurements for devices with good looking manufacturer specs is useful because it confirms whether the device is in fact something good and whether the manufacturer's specs are to be trusted. I personally don't have any curiosity to learn more about devices whose own manufacturer states they won't perform well.

I wouldn't measure old (no longer sold) devices and the devices that specced low. Life is too short. There are so many new devices on Aliexpress every month!