Feedback on "quick review" process

[amirm]

Apologies if already suggested above, but you could provide a list of upcoming reviews with your suggested review level, and allow people to vote on whether it should be adjusted up or down. That way you could get a sense of what people are interested in, and how much effort you should expend on each.

The process of what gets reviewed next is highly complex and doesn't lend itself to group effort or any kind of democratic process. Often there are deadlines for equipment. Or it is something I value more than others. Or it is unannounced product. Or it is an item from a company I have never reviewed so I am interested in getting it done.

The few times people have know what is coming has actually added stress to the whole process with a stream of question as to when it is coming out. Mind you, I do want to hear from you as that impacts the cadence of such reviews. But it comes at a cost.

 

[617]

@jhaider I agree with you on speakers. I think the power compression chart is more useful than distortion.

Speakers have the greatest variety of audible nonlinearities. I would not omit directivity graphs. The sonogram is the defacto standard for comparing speaker dispersion; the spinorama is very useful for evaluating how good a speaker is, but it doesn't show what the speaker is doing.

I would omit the CSD charts. They are only useful in comparison, and are not a good way to assess sound quality. Really, their main use is to compare resonances in individual drive units, not complete systems.

I personally think the DAC/Headphone amp reviews could be streamlined a bit, so many of them perform so incredibly well, and almost every test is dealing with infinitesimals. Have we ever seen a DAC which performed well but which had a bad multi-tone? Or vice-versa?

 

[Rja4000]

My suggestion a while back was to present the objective review measurements in a locked thread with a link to an accompanying discussion thread on the same. That locked thread review could be the mini review.

That way the reviews can be short, concise and to the point. They also won't get polluted with infighting and ultimately derailed or locked (as was a recent review). @amirm , you can then choose to participate and at what level. You could add content if you wanted to this thread.

The reviews are then clean and easy to search. People can choose "just the facts" or go down the rabbithole of discussion.

I think the open discussion below the review is of highest added value.

Publishing measured figures, anyone can do that.
But letting members question and challenge it is very important and scientific in essence.
It helps improving content quality and make reviews more robust to critics.
That's one of the reasons I'm here, personnaly.

I may be wrong, but I think Amir also likes it.

Where you're right is this may become very time consuming in some cases.
Which now makes me think that shortening review of crap devices may end up costing more time than saving it...

 

[Fluffy]

The process of what gets reviewed next is highly complex and doesn't lend itself to group effort or any kind of democratic process. Often there are deadlines for equipment. Or it is something I value more than others. Or it is unannounced product. Or it is an item from a company I have never reviewed so I am interested in getting it done.

The few times people have know what is coming has actually added stress to the whole process with a stream of question as to when it is coming out. Mind you, I do want to hear from you as that impacts the cadence of such reviews. But it comes at a cost.

I totally get that. Some processes suffer when they're democratized.

 

[Koeitje]

Proposed quick formats:

Speakers

1. Spinorama
2. Distortion Spectrum (86dB vs. 96dB)
3. Heatmap directivity

Electronics

1. Dashboard
2. SNR
3. Distortion vs. frequency
4. IMD vs. level
5. Distortion vs. power (if applicable)
6. Peak/burst power (if applicable, for power amps)
7. Channel balance vs. volume (if applicable)

No comparison charts. No subjective section or listening tests.

I'd like to add that I like the subjective listening tests for speakers, but not for the general audio quality but more for the ability to play loud. How easy is it to bottom out the speaker? When does low frequency distortion become a clear problem? The Distortion spectrum at 96dB can do a lot, but its not quite te same.

But watch your ears .

 

[Frank Dernie]

I've mentioned it before but e.g. for speaker reviews, how interesting are poor performing speakers? I don't know the best process for selection, but being more selective towards potentially good sounding speakers would be one way.

I think this is a difficult one since there has not been total agreement with the expected result when Amir has listened, both to ones with similar spins and widely varying. So it is difficult to be 100% sure from the measurement at present to give a totally reliable benchmark to discard a "poor" speaker.
Also there is a big enough minority of listeners who prefer more directional speakers to wide directivity choices for the single preference rating to be universally applicable (IMO).
I consider wide directivity with even frequency response off axis to add too much of my listening room acoustics to the recording, which I consider a type of euphonic colouration IME.

 

[Frank Dernie]

I think the power compression chart is more useful than distortion.

I think both are important.
I know some people hear differences between "High res" and 320 bit rate MP3s that I either do not or find inconsequential, but I discovered low bass distortion in speakers which many people don't seem to mind has been important to me decades ago, and has been a key aspect of speaker choice for me for many years. I would agree power compression is important.
OTOH maybe my opinion is irrelevant since I am not likely to buy any more speakers, or certainly not of the size Amir can lift into his klippel device.

 

[restorer-john]

OTOH maybe my opinion is irrelevant since I am not likely to buy any more speakers, or certainly not of the size Amir can lift into his klippel device.

Frank, your opinion is never irrelevant, far from it. Amir may buy a small forklift and the Klippel "stage" may end up larger. Perhaps he may even extend his roof space to allow decent sized speakers to be measured.

 

[Frank Dernie]

Frank, your opinion is never irrelevant, far from it. Amir may buy a small forklift and the Klippel "stage" may end up larger. Perhaps he may even extend his roof space to allow decent sized speakers to be measured.

Thanks John, but I feel less and less typical of the general membership as time goes on.

 

[nemanja_t]

I would be totally non typical member, DACs do not interested me at least, if integrated amplifier does not have good enough DAC, then it is not worth considering. Also headphone amps and such, just a rabbit whole for money and never ending discussions. If only few things could be tested, then good integrated and power amps with speakers. This small electronics devices makes just noise in system and takes time for testing.

 

[sergeauckland]

I think both are important.
I know some people hear differences between "High res" and 320 bit rate MP3s that I either do not or find inconsequential, but I discovered low bass distortion in speakers which many people don't seem to mind has been important to me decades ago, and has been a key aspect of speaker choice for me for many years. I would agree power compression is important.
OTOH maybe my opinion is irrelevant since I am not likely to buy any more speakers, or certainly not of the size Amir can lift into his klippel device.

As much as I appreciate the effort going into loudspeaker measurements, I have very little interest in yet another two-way 'speaker, whether active or passive. My interest in loudspeakers tends to be for the larger ones, especially active, like PMC MB3 or larger. A possible exception might be for the Devialet Phantom or Dutch & Dutch 8C. I would also find interesting some of the larger horn systems, as I've never heard one I've liked.

As to electronics, very little interest in yet another DAC or headphone amplifier, and none at all in headphones unless something special like the Smyth system.

My main interest is in vintage equipment, so would be interested in vintage loudspeakers like the Empire 9000 or IMFs, Pro reel to reel machines like Ampex ATRs and Studers, Scullys, MCIs etc etc.

As to reviewing only current equipment available to buy, I have to ask why? We're not a commercial site and have no need for manufacturers' support, so whether one can buy the equipment new or not is, to me, irrelevant. It just needs to be interesting.

Finally, as to short or long reviews, I think it should be decided by how interesting the equipment is. If it's just another 2-way loudspeaker as short review is fine, if it's a special design, like the Phantom or 8C, then more in depth. Similarly with electronics. Is it an interesting design or another 'me-too' product.

S.

 

[Alexanderc]

Amir, I would tend to give you the benefit of the doubt regarding the depth of a review. We see frequent comments along the lines of, “the next few graphs tell us what we already know....” Perhaps that could be a cue to stop. If you see something interesting or unusual, maybe you’ll want to dig deeper. People will always have additional and specific questions, but sometimes these don’t contribute to the general understanding and don’t “need” to be included. You have to draw the line somewhere.

I might also suggest, as others have, that items for review be more selectively curated.

 

[SKBubba]

All I look at is the name of the product, the price, and the panther.

 

[restorer-john]

Thanks John, but I feel less and less typical of the general membership as time goes on.

You and me both.

Actually, I reckon you are more 'progressive' in many ways than me.

 

[Midwest Blade]

Lighter reviews in general for both speakers and electronics would be okay for me, as a non-technical (blasphemy) audio scientist. While I learn more and more each day on ASR, I would probably gain more from a less technical and more focused report.

 

[Thomas savage]

Thanks John, but I feel less and less typical of the general membership as time goes on.

Well you don't live in ya mums basement so yes your a rare bread around here these days .

All the more valued for being so.

 

[GXAlan]

I've sent @amirm lots of quirky stuff. Here's how the #s for the stuff I sent in look

PS Audio Noise Harvester: 11 pg discussion
Panasonic DP-UB9000: 8 pg discussion
AudioQuest PowerQuest 3: 7 pg discussion
JBL 4319: 7 pg discussion
Canon S-50: 5 pg discussion
JBL XPL90: 3 pg discussion

Vintage Denon DA-500: Popularity TBD

The XPL90 had the lying down panther and the least amount of discussion. It's not readily available. For me, it was useful to see the "Best of Harman" right before they embarked on the science-based re-engineering, in part because it added to the story of the Revel M22, Studio 530, HDI 1600, and Revel M106, etc. But I think this shows that iconic vintage speakers are worth testing, but that's about it. The Canon being an iconic one. The Noise Harvester and PowerQuest 3 have high levels of discussion even though no one is going to buy either. Some of it is discussion about how much audio-foolery there is, but I think this also shows the value of testing things for science sake. That's one of the reasons I submitted the DA-500.

I've asked Amir to do a bit of extra work in that it's one of the only DACs that let us turn on/off ALPHA processing (which is marketed as a feature on today's AL32 X4700H and up) and I've sent him the test tracks that Denon uses for their pretty pictures even to this day.
http://mobile.denon.com/pages/GlossaryDetail.aspx?GId=13

The real questions are
a) Does ALPHA give you that smooth sine wave?
b) How does a modern ESS or AKM DAC reproduce the waveform? Is it blocky like the "conventional" output or is is smooth?
c) How does a modern ESS or AKM DAC reproduce -60dB music recorded at 16-bit/44 compared to a design with ALPHA

There are some other questions too. The digital roll off is supposed to be dynamic based upon the signal,

How does the readership feel about Amir spending extra time studying Denon ALPHA technology? Since I don't need the DAC back right away, my bias is to let Amir keep until he has time to review it -- but if no one else is interested in Alpha, then it's fair for him to spend less time on it.

 

[Alexanderc]

I've sent @amirm lots of quirky stuff. Here's how the #s for the stuff I sent in look

PS Audio Noise Harvester: 11 pg discussion
Panasonic DP-UB9000: 8 pg discussion
AudioQuest PowerQuest 3: 7 pg discussion
JBL 4319: 7 pg discussion
Canon S-50: 5 pg discussion
JBL XPL90: 3 pg discussion

Vintage Denon DA-500: Popularity TBD

The XPL90 had the lying down panther and the least amount of discussion. It's not readily available. For me, it was useful to see the "Best of Harman" right before they embarked on the science-based re-engineering, in part because it added to the story of the Revel M22, Studio 530, HDI 1600, and Revel M106, etc. But I think this shows that iconic vintage speakers are worth testing, but that's about it. The Canon being an iconic one. The Noise Harvester and PowerQuest 3 have high levels of discussion even though no one is going to buy either. Some of it is discussion about how much audio-foolery there is, but I think this also shows the value of testing things for science sake. That's one of the reasons I submitted the DA-500.

I've asked Amir to do a bit of extra work in that it's one of the only DACs that let us turn on/off ALPHA processing (which is marketed as a feature on today's AL32 X4700H and up) and I've sent him the test tracks that Denon uses for their pretty pictures even to this day.
http://mobile.denon.com/pages/GlossaryDetail.aspx?GId=13

The real questions are
a) Does ALPHA give you that smooth sine wave?
b) How does a modern ESS or AKM DAC reproduce the waveform? Is it blocky like the "conventional" output or is is smooth?
c) How does a modern ESS or AKM DAC reproduce -60dB music recorded at 16-bit/44 compared to a design with ALPHA

There are some other questions too. The digital roll off is supposed to be dynamic based upon the signal,

How does the readership feel about Amir spending extra time studying Denon ALPHA technology? Since I don't need the DAC back right away, my bias is to let Amir keep until he has time to review it -- but if no one else is interested in Alpha, then it's fair for him to spend less time on it.

I find it interesting. A manufacturer is making a claim about a technology that is in current products, so let’s see if it works.
Edit: and if it matters.

 

[Beershaun]

Here is my opinion on metrics in general.
1) you have a set of top level metrics that are standard across all your devices in the test category. They should be the best representation of meaningful customer facing performance you can provide within the limits of the required precision and standardization to make them comparable across devices.
2) you have a set of deeper metrics that would change device over device depending on that devices measured strengths or weaknesses. Meaning you are only posting and commenting on areas where that device stood out in it's class as performing particularly well or particularly poorly. Based on your judgment. To help highlight the comparative strengths and weaknesses.

This would help you standardize on a format to deliver the most relevant information to people without being too Ridgid such that it forces you back to showing everything or intentionally hiding a metric of interest.

 

[Thomas_A]

I think this is a difficult one since there has not been total agreement with the expected result when Amir has listened, both to ones with similar spins and widely varying. So it is difficult to be 100% sure from the measurement at present to give a totally reliable benchmark to discard a "poor" speaker.
Also there is a big enough minority of listeners who prefer more directional speakers to wide directivity choices for the single preference rating to be universally applicable (IMO).
I consider wide directivity with even frequency response off axis to add too much of my listening room acoustics to the recording, which I consider a type of euphonic colouration IME.

Difficult perhaps but having a minimum of listening test and spinorama gives a choice for Amir to continue with more masurements. A choice that would give less work. There are other values in life.