• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Peer review...a bad thing....a failure....a waste of time.........

MCH

Major Contributor
Joined
Apr 10, 2021
Messages
2,581
Likes
2,199
well.... it is a very complex problem that the author treats in the most simplistic way: blame the peer review system! but in reality it is only one small part of the real broader problem. If i would have to name, like that, in two minutes, without thinking much, the main reasons for the evident problems that scientific production is facing, i would say (this is just my opinion):
- Link funding to literature production
- the business model of a few big publishing houses
the consequences of these are many, examples: millions of papers that need to be published to obtain or justify the funding, so many that nobody will ever read or even search and all the ramifications of this you can imagine: very low quality, fraud...
The peer review system? yes, it could work better, but it is just a collateral small piece of the problem, i don't see how someone can make it the main reason.

The problem starts with institutions (universities, governments, agencies, companies....) that don't have the capacity (knowledge, time, desire) to evaluate the science they are funding. The rest is just a consequence of that, and the peer review system is just a small fix in the engine that fails to do its job properly.

Going back to the article, it seems to try to find the reason for a supposed failure of the science of the last 50 years. Really? why has research in the last decades been a failure and why the peer review system is the responsible of that? either isn't clear to me.

my 2c.
Now lets go back to the real most relevant problem of humanity: why Topping and SMSL keep on releasing 2 channel DACs with no DSP, no HDMI and no multichannel capability?????
 

Marc v E

Major Contributor
Joined
Mar 9, 2021
Messages
1,106
Likes
1,606
Location
The Netherlands (Holland)
well.... it is a very complex problem that the author treats in the most simplistic way: blame the peer review system! but in reality it is only one small part of the real broader problem. If i would have to name, like that, in two minutes, without thinking much, the main reasons for the evident problems that scientific production is facing, i would say (this is just my opinion):
- Link funding to literature production
- the business model of a few big publishing houses
the consequences of these are many, examples: millions of papers that need to be published to obtain or justify the funding, so many that nobody will ever read or even search and all the ramifications of this you can imagine: very low quality, fraud...
The peer review system? yes, it could work better, but it is just a collateral small piece of the problem, i don't see how someone can make it the main reason.

The problem starts with institutions (universities, governments, agencies, companies....) that don't have the capacity (knowledge, time, desire) to evaluate the science they are funding. The rest is just a consequence of that, and the peer review system is just a small fix in the engine that fails to do its job properly.

Going back to the article, it seems to try to find the reason for a supposed failure of the science of the last 50 years. Really? why has research in the last decades been a failure and why the peer review system is the responsible of that? either isn't clear to me.

my 2c.
Now lets go back to the real most relevant problem of humanity: why Topping and SMSL keep on releasing 2 channel DACs with no DSP, no HDMI and no multichannel capability?????
I wholeheartedly concur. I would give you two thumbs up if I could.
 
Last edited:

MaxwellsEq

Major Contributor
Joined
Aug 18, 2020
Messages
1,630
Likes
2,430
It's an interesting perspective, but doesn't hit the real target. The current business model of the journal publishers looks like a scam to me. Individuals can't afford to subscribe, so it comes down to libraries (e.g. in colleges), to take the periodicals. So the publishers produce more and more variations with an almost guaranteed income, because the colleges have to subscribe!
Meanwhile, to get and maintain tenure, scientists have to publish regularly. To handle so many articles, there needs to be a large number of journals (if all we had was Nature, it might take years for your paper to be scheduled) so the publishers create more journals. See above.
Meanwhile, if there's a paper I want to read in a subject I specialise in, the publishers try and charge me $250. I can't afford that, so there's a chilling effect on non-academically funded peer to review.
 

nerdstrike

Active Member
Joined
Mar 1, 2021
Messages
258
Likes
311
Location
Cambs, UK
The opening of the article is flat wrong about peer review. Maybe they didn't do it via parasitic publishers, but they sure did challenge people in places like the Royal Institution when work was presented. Many a person was branded a nutter.

Yes, peer review is flawed and corruptible, and "publish or perish" has produced an extraordinary mess. What has been happening however, is that there has been a 20 year push slowly in the direction of reproducibility, and that's what's showing up the charlatans for opinion-pieces like this.

The publishing houses are heavily culpable, because they and the funding agencies are creating environments where cheating flourishes.
 

Wmusic

New Member
Joined
Oct 1, 2022
Messages
2
Likes
2
We are in a pandemic proliferation of scientific papers but peer review is still an embankment against floods. In ASR you are experimenting a sort of peer review system applied to hobbyist sound reproduction. In my opinion it works.
 

tuga

Major Contributor
Joined
Feb 5, 2020
Messages
3,984
Likes
4,281
Location
Oxford, England
The problem starts with

Humans. All areas of society are affected and this one is no different.
It's a bit like digital filtering and Redbook: the theory is perfect but ultimately cannot be implemented in practice.
 

Geert

Major Contributor
Joined
Mar 20, 2020
Messages
1,938
Likes
3,526

Article about the pitfalls of peer review in science. What do you guys think?

Doesn't really surprise me. Review work is often boring. It's not creative or innovative, you might even not be interested in the topic. So often mistakes surface only when people use your work to build on and they don't get the expected results, or when someone makes a meta-study. As mentioned above, it's human.
 

killdozzer

Major Contributor
Joined
Aug 2, 2020
Messages
1,615
Likes
1,628
Location
Zagreb
Ooooooorrrrr... Someone was smart enough to harness the power of professional jealousy and ego-fueled competitiveness? You're easier to see faults in others.
 

nerdstrike

Active Member
Joined
Mar 1, 2021
Messages
258
Likes
311
Location
Cambs, UK
professional jealousy and ego-fueled competitiveness
These traits do a tonne of harm to genuine publications, when Reviewer 2 takes a dump all over your work and it gets rejected out of hand despite obvious incomprehension in the criticism.

Some of the egotists are really good at suppressing rival work. It's a thankless business to be utterly dependent on publication metrics in a hotly contested field.
 

tuga

Major Contributor
Joined
Feb 5, 2020
Messages
3,984
Likes
4,281
Location
Oxford, England
I think that in many scientific journals peer reviewing is expected to be done for free.
Is this a good thing?
 

Keith_W

Major Contributor
Joined
Jun 26, 2016
Messages
2,533
Likes
5,808
Location
Melbourne, Australia
Meanwhile, if there's a paper I want to read in a subject I specialise in, the publishers try and charge me $250. I can't afford that, so there's a chilling effect on non-academically funded peer to review.

Do what I do. Call your University library that is likely to have the paper that you need. Tell them that you are a student and you need this paper for a research project. Ask them to email or fax it to you.
 

Colonel7

Addicted to Fun and Learning
Joined
Feb 22, 2020
Messages
616
Likes
876
Location
Maryland, USA
The silly medieval tenure system is a big part that requires more and more pubs no matter the field or type of institution. One thing that the article doesn’t talk about is that the freely available and open peer-reviewed journals or articles can sometimes cost more to the university or dept than journals by the big publishers if they have high research output. You pay or contribute by article. It’s been awhile since I’ve been in govt but at the time we were working on having govt funded research free and open to the public within 4 or 6 months of journal publication date. Not sure where that stands. Still doesn’t mean anyone will read them or that the big publishers don’t have workarounds to extend the delay in availability by offering pre-publication online before the clock starts ticking.
 

LuvTheMusic

Active Member
Joined
Jun 7, 2020
Messages
111
Likes
137
The author, when not making unsupported assumptions, cherry picks data and relies on anecdotes.

Where to start? Let's try these:

"From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church." -- Not quite: how about the fact that until quite recently, only a very small economically and socially elite class had any ability whatsoever to communicate anything other than orally, much less "publish" scientific information.

"research productivity has been flat or declining for decades, and peer review doesn’t seem to have changed that trend." The author fails to understand the point of the article that he cites, which is that research productivity has diminished because the ideas are becoming more difficult to find. (Consider that Einstein could do thought experiments alone in his office; finding the Higgs boson took billions of dollars and hundreds of scientists.)

"New ideas are failing to displace older ones." First, why should new ideas displace older ones? Isn't the robustness of old ideas a sign that the latter likely are correct? That's a cause for celebration! Moreover, this point is related to the one above: in a mature science, it is really difficult to generate new ideas -- and that is precisely as it should be.

"you can’t even ask them to rate the Nobel Prize-winning physics discoveries from the 1990s and 2000s because there aren’t enough of them." It does not seem to occur to the author that Nobel prizes tend to be awarded for research that has had time (as in decades) to demonstrate its importance in a field. And how do we judge where there are "enough" Nobel prizes to rank order them?

I gave up after a few paragraphs. Peer review may indeed be imperfect, and perhaps we would be better off without it. (Or maybe not: think hydroxychloroquine and invermectin for Covid.) And rants about frustrating processes are fun to write and to read. But as a piece of reasoning, well.....
 

oivavoi

Major Contributor
Forum Donor
Joined
Jan 12, 2017
Messages
1,721
Likes
1,935
Location
Oslo, Norway
I'm a tenured social scientist at a university, and I agree with almost everything in this article. This is not the first time such criticism has been voiced, there have been many criticisms of the peer review system in recent years. Here's a longer and more detailed scholarly exposition of why peer review doesn't work as it should: https://www.liamkofibright.com/uploads/4/8/9/8/48985425/is_peer_review_a_good_idea_.pdf

Regarding audio science, for example, the main challenge as I see it is not a lack of peer review. The main challenge is that there is not enough funding and not enough researchers working on such topics. Therefor there haven't been many attempts at replicating some of the canonical findings in the field. As a result I regard many postulates in audio science as "probable, but not settled science". This has nothing to do with the quality of peer review, but with the lack of real scholarly competition and debate around several key issues.
 

NiagaraPete

Major Contributor
Forum Donor
Joined
Jun 23, 2021
Messages
2,185
Likes
1,953
Location
Canada
I don't think Einsteins peer reviews went very well, but now ....
 
Top Bottom