On Peer Reviewed Science

[Wombat]

The OP was to inform about peer-review in science. It was not about defining science. That is where some members have unfortunately taken it in order to involve themselves in ego-driven engagement.

Twitter?? Being able to engage in less than 280 characters of type.

 

[Cosmik]

@Cosmik , you make good examples to illuminate your point. And I like your open stance on trying to understand other parties’ positions.

What you described is what can be called measurement without theory. I have already talked about the theorization and commensurability problem, which is related.

If you dig deeper into the term measurement without theory, you’ll see that this is an old controversy. More intelligent men than we are have discussed this at length many years ago (and the debate will never end!). So I don’t have one answer to your thoughts. But reading about this controversy - and how people have dealt with this debate previously, decades and decades ago - will make us all a bit more knowledgable and humbler.

OK. Hints taken on board!

But I get cognitive dissonance between the high level intellectual stuff you are suggesting everyone should have read about for years before entering into this discussion, and the level of discussion about audio science in forums!

If 'measurement without theory' is a thing - and a thing I have been rejecting in this forum for years using my own primitive reasoning - how is it that no one has raised the issue in discussions about listening tests? It is so very obvious to me that listening tests are measurement without theory, and if there is, at least, controversy over the matter, then listening tests are, at least, controversial. Yet I am literally the only person in this forum who seems to suggest that listening tests might be fundamentally unscientific.

 

[Cosmik]

Twitter?? Being able to engage in less than 280 characters of type.

As far as I can tell, your comments *always* fit into way less than 280 characters, and you never really engage, just 'snipe'.

 

[Wombat]

I don't babble, usually.

Engagement: I don't tend to engage with self-opinionation, navel gazing, contrarianism and pseudointellectual musings unless something rational is put forward.

Snipe?? Prick unfounded beliefs, you mean.

You were free to start a thread on the definition of Science.

Have you engaged with those who can test/take advantage of your propositions and thoughts. Universities, audio companies, and enlightened entrepeneurs ? Would be more beneficial than the repeated posits, here, don't you think?

I am way past confidently stated imaginings impressing me.

 

[svart-hvitt]

OK. Hints taken on board!

But I get cognitive dissonance between the high level intellectual stuff you are suggesting everyone should have read about for years before entering into this discussion, and the level of discussion about audio science in forums!

If 'measurement without theory' is a thing - and a thing I have been rejecting in this forum for years using my own primitive reasoning - how is it that no one has raised the issue in discussions about listening tests? It is so very obvious to me that listening tests are measurement without theory, and if there is, at least, controversy over the matter, then listening tests are, at least, controversial. Yet I am literally the only person in this forum who seems to suggest that listening tests might be fundamentally unscientific.

I guess it’s kudos to you for having an intuitive sense of BS and the fact that you’re skeptical towards measurements without theory.

In my view, this issue is delicate, fragile and complex. It takes some insight and wisdom to separate BS from real science (i.e. knowledge). In my area I may be able to separate the good from bad; in other areas I wish (even though I may be able to find methodological flaws or weaknesses).

PS: Your remark on «hints taken»...I don’t know if I gave any...but this may be two cultural biases meeting each other

 

[andreasmaaan]

@Cosmik it looks like the discussion has gone in a different direction since you quoted my post, so I'll keep it brief.

To your first example, you gave the hypothesis: a neural network can be programmed to judge good vs bad art.

What's lacking here is empiricism. The hypothesis is not empirical because "good" and "bad" are value judgements and therefore cannot be falsified empirically. You need to revise the hypothesis to something more empirical; perhaps "a neural network can be programmed to mimic human judgements of art."

Secondly, your prediction (the specific thing you try to falsify) does not fit your hypothesis, because it introduces a ranking system (1-10) whereas your hypothesis posits a binary system (good/bad). Thirdly, the conclusion you draw ("I speculate that I have mimicked the way the human brain views art...") does not relate to your hypothesis, which contained no statement about the functioning of the human brain or how humans perceive anything.

Finally, a positive result in one experiment does not prove a hypothesis. It just means that the prediction was not falsified under the experimental conditions and therefore the hypothesis has not been disproven. The experiment needs to be repeated and (more importantly) additional empirically falsifiable predictions that follow from the hypothesis need to be posited and tested, before you can start to become more confident in your hypothesis.

Science can never prove anything is "true". It is, rather, a rigorous way of arriving at the best explanation subject to current limitations.

You may tell me: science isn't meant to be useful or to have a point. Science just 'is' at this moment and only this moment. It does not claim to be the truth nor timelessness nor universality.

If so, it begins to sound a bit like definitions of art... slippery, self-referential, circular. I'm sure it's not meant to be like that!

The scientific method itself is rigorous and unchanging, but the validity of the theories it leads to are certainly not absolute; these are always limited by both the intelligence and creativity of those who develop them, and their capacity (e.g. technologically) to test them. The most a scientific theory can hope to be is the current best explanation.

So science really does make only rather modest claims compared to what your expectations seemed to be

 

[Cosmik]

@Cosmik it looks like the discussion has gone in a different direction since you quoted my post, so I'll keep it brief.

To your first example, you gave the hypothesis: a neural network can be programmed to judge good vs bad art.

What's lacking here is empiricism. The hypothesis is not empirical because "good" and "bad" are value judgements and therefore cannot be falsified empirically. You need to revise the hypothesis to something more empirical; perhaps "a neural network can be programmed to mimic human judgements of art."

Secondly, your prediction (the specific thing you try to falsify) does not fit your hypothesis, because it introduces a ranking system (1-10) whereas your hypothesis posits a binary system (good/bad). Thirdly, the conclusion you draw ("I speculate that I have mimicked the way the human brain views art...") does not relate to your hypothesis, which contained no statement about the functioning of the human brain or how humans perceive anything.

Finally, a positive result in one experiment does not prove a hypothesis. It just means that the prediction was not falsified under the experimental conditions and therefore the hypothesis has not been disproven. The experiment needs to be repeated and (more importantly) additional empirically falsifiable predictions that follow from the hypothesis need to be posited and tested, before you can start to become more confident in your hypothesis.

Science can never prove anything is "true". It is, rather, a rigorous way of arriving at the best explanation subject to current limitations.



The scientific method itself is rigorous and unchanging, but the validity of the theories it leads to are certainly not absolute; these are always limited by both the intelligence and creativity of those who develop them, and their capacity (e.g. technologically) to test them. The most a scientific theory can hope to be is the current best explanation.

So science really does make only rather modest claims compared to what your expectations seemed to be

Let's not forget where this came from! I earlier said

Is there anything in your opinion that could not be tested by science? It seems to me that a scientific procedure could be followed with anything, such as investigations of beauty, art, religion, and it might even give repeatable 'results'.

But can't you just look at those fields and declare that science does not cover them? You cannot use science to determine good and bad art, for example, even if your procedure looks like science. The warning sign example is very close to that which is why I would have said it was not actually science beforehand...

To which you replied

Could you give a hypothetical example of a study concerning beauty, art or religion that you believe follows a procedure that looks like science but is not science?

So I had already claimed that science cannot determine good and bad art. Basically because they are value judgements of another non-falsifiable judgement - what constitutes art in the first place. I was implying that anything involving value judgements is not science and can be dismissed beforehand regardless of the rigour of the procedure.

And I think you are putting words in my mouth. I didn't say anywhere that science establishes 'the truth' nor 'proves' anything.

And on the details, in my experiment I was careful to specify a 'pseudo-objective' score (the time spent looking at the painting), and a subjective score (asking people to give a score of 1-10). I didn't say how I was combining them, nor what threshold I was using to define 'good' and 'bad'.

The whole point of my 'experiment' was to create something that looks like a scientific procedure but isn't science. The conclusions and speculations I draw are not part of the procedure but they do resemble the unwarranted conclusions that people every day draw from science - including in the audio world.

Definitions of 'value judgement' include:

...a subjective assessment based on one's own code of values or that of one's class

...an opinion about [something] based on your principles and beliefs and not on facts which can be checked or proved.

...an estimate made of the worth, goodness, etc. of a person, action, event, or the like, esp. when making such judgment is not called for or desired

You say

The hypothesis is not empirical because "good" and "bad" are value judgements and therefore cannot be falsified empirically. You need to revise the hypothesis to something more empirical; perhaps "a neural network can be programmed to mimic human judgements of art."

An infinitesimal difference, I would say. Pre-defining my version of 'good' and 'bad' as being the scores humans give, is almost precisely the same as saying "I hypothesise I can mimic humans' judgement of good and bad art if I ask them to score art as good or bad".

And surely you are just focusing on 'good art' and 'bad art' as value judgements but missing that what constitutes 'art' in the first place is the biggest non-falsifiable judgement of all. It could be viewed inherently as a good/bad judgement because creating something that we acknowledge as 'art' implies that it has, at some level, enough 'merit' to be given the title, rather than all the other things in the world. But if you insist, we can say that there is no fixed definition of art and we can only define one ourselves and make non-falsifiable judgements as to whether that is the 'true' definition.

Care to define what 'art' is ? Well, just as I made up a scoring mechanism for 'good' and 'bad' we could make one up for the purposes of the experiment. If the experimenter says, for example "We define art in our experiment as any exhibit found in an art gallery" then that would have no more merit than my saying "We define good art as something in an art gallery that a person stares at for longer than the mean time".

If what constitutes art is a 'value judgement' - or some other kind of judgement that is not falsifiable empirically - then I claim there is *nothing* regarding art on which you can perform a scientific experiment. And that's where all this came from: I suggested there are areas in which science is not valid no matter how rigorous the procedure. Judgement of art is one of them.

And I look forward to the same levels of rigour whenever anyone is recommending listening tests and statistics to establish which ethernet cable sounds best. Which speaker sounds best. Whether second harmonic distortion is 'preferred' over third, etc. All 'value judgements' based on another thing we cannot define without making another non-falsifiable judgement: music. If the claim is that if we use the word 'preference' instead of 'good', 'bad', 'best', 'worst', then this renders the experiment scientific, then I maintain that this is 'slipperiness' of the first order.

 

[Cosmik]

We are often told that science does not discover 'the truth'. Yet the headline in The Guardian the other day says:
Our job as scientists is to find the truth.
It's one viewpoint - that seems to be as widely held as the opposite one. If, like grammar, enough people start to think of it in one way, does that change the definition de facto?

If someone (like me) tries to suggest that science is stepping beyond its bounds if straying into judgements of 'art' I am told that it can all still be science as long as we draw valid conclusions - and those conclusions can be infinitesimally narrow with no generality or universality whatsoever: e.g. these people on this day at this time in this place under these conditions said this when shown these things - probably. It's still science!

But one reading of the UK Science Council definition of science says there must be 'general rules or conclusions' drawn from facts or examples:

Science is the pursuit and application of knowledge and understanding of the natural and social world following a systematic methodology based on evidence.​

Scientific methodology includes the following:

• Objective observation: Measurement and data (possibly although not necessarily using mathematics as a tool)
• Evidence
• Experiment and/or observation as benchmarks for testing hypotheses
• Induction: reasoning to establish general rules or conclusions drawn from facts or examples
• Repetition
• Critical analysis
• Verification and testing: critical exposure to scrutiny, peer review and assessment

Cleverly, they don't stipulate 'the' methodology, just 'a' systematic methodology - you could read it either way. Also, they say scientific methodology 'includes the following'. Again, you could read it as 'must include all the following' or 'pick and mix from the following and add some of your own if you like as long as it's systematic'.

Someone a few posts back stated convincingly that science was not invented with Karl 'falsifiability' Popper, and so the criterion of falsifiability is just a recent fashion. I like Popper's definition, but maybe it's not a condition of science. Or maybe it is.

Great store is held by 'value judgements' not being falsifiable empirically. But if we gracefully slip into 'preference' (a.k.a. 'liking'), or 'mimicking human judgements of good and bad', etc. we seem to be able to sidestep that issue.

We are told science does not establish truth. That science doesn't prove anything. Does this mean there is no such thing as a 'fact' that is not mathematical? How do we establish what facts are? We need 'facts' in order to use them in our scientific methodology, apparently (see above).

A fact is a statement that is consistent with reality or can be proven with evidence. The usual test for a statement of fact is verifiability — that is, whether it can be demonstrated to correspond to experience.

So not falsifiable, but 'verifiable'. The word 'proven' is used, but science doesn't 'prove'. 'Can be demonstrated' sounds like a small part of the scientific method, and upon that basis we can apparently use the 'fact' in our science to draw conclusions from? (see above).

I am perceiving science as 'elusive', slippery, 'subjective', 'circular'. It is whatever you want it to be. Substantive issues of principle can be papered over simply by using different terms. Don't change the experiment; just keep changing the words until you've made it 'scientific'.

Some people define art as "Whatever an artist says is art". I am beginning to think that science is very similar...

 

[svart-hvitt]

Which subject(s) (field(s) of study) is the team focusing on?

One could say it’s social sciences and the use and construction of huge data bases.

 

[svart-hvitt]

We are often told that science does not discover 'the truth'. Yet the headline in The Guardian the other day says:
Our job as scientists is to find the truth.
It's one viewpoint - that seems to be as widely held as the opposite one. If, like grammar, enough people start to think of it in one way, does that change the definition de facto?

If someone (like me) tries to suggest that science is stepping beyond its bounds if straying into judgements of 'art' I am told that it can all still be science as long as we draw valid conclusions - and those conclusions can be infinitesimally narrow with no generality or universality whatsoever: e.g. these people on this day at this time in this place under these conditions said this when shown these things - probably. It's still science!

But one reading of the UK Science Council definition of science says there must be 'general rules or conclusions' drawn from facts or examples:

Cleverly, they don't stipulate 'the' methodology, just 'a' systematic methodology - you could read it either way. Also, they say scientific methodology 'includes the following'. Again, you could read it as 'must include all the following' or 'pick and mix from the following and add some of your own if you like as long as it's systematic'.

Someone a few posts back stated convincingly that science was not invented with Karl 'falsifiability' Popper, and so the criterion of falsifiability is just a recent fashion. I like Popper's definition, but maybe it's not a condition of science. Or maybe it is.

Great store is held by 'value judgements' not being falsifiable empirically. But if we gracefully slip into 'preference' (a.k.a. 'liking'), or 'mimicking human judgements of good and bad', etc. we seem to be able to sidestep that issue.

We are told science does not establish truth. That science doesn't prove anything. Does this mean there is no such thing as a 'fact' that is not mathematical? How do we establish what facts are? We need 'facts' in order to use them in our scientific methodology, apparently (see above).

So not falsifiable, but 'verifiable'. The word 'proven' is used, but science doesn't 'prove'. 'Can be demonstrated' sounds like a small part of the scientific method, and upon that basis we can apparently use the 'fact' in our science to draw conclusions from? (see above).

I am perceiving science as 'elusive', slippery, 'subjective'. It is whatever you want it to be. Substantive issues of principle can be papered over simply by using different terms. Don't change the experiment; just keep changing the words until you've made it 'scientific'.

Some people define art as "Whatever an artist says is art". I am beginning to think that science is very similar...

@Cosmik , did you read this?

(I think the material is relevant for casting light on your questions).

https://poseidon01.ssrn.com/deliver...3025065001022003011094084002017078083&EXT=pdf

Tell me what you think

 

[Cosmik]

@Cosmik , did you read this?

(I think the material is relevant for casting light on your questions).

Tell me what you think

I did start to read it. It seems to contradict @andreasmaaan when he says:

The scientific method itself is rigorous and unchanging

In the paper you link to, the author seems to be saying that you can have science even when using a methodology you 'make up yourself' based on 'pragmatism'..?

If so, I am still left thinking that science is somewhat 'malleable' even though the usual perception is that its usefulness is based on the opposite: 'rigour'.
(I know, it isn't meant to be useful, just as it doesn't find the truth nor prove anything. But appeals for funding are almost always based on science's usefulness and its ability to prove things and establish the truth Ditto when anyone refers to some science to back up a point they are making).

 

[SIY]

Sigh. More conflation of social "science" with actual science.

 

[svart-hvitt]

Sigh. More conflation of social "science" with actual science.

Your position is a marginal one. And because audio is physics, psychoacoustics and psychology - you are also on the marginal side of audio scientists.

 

[SIY]

I'm not an audio scientist, my specialties are more molecular.

I have done non-audio sensory science (haptics, organoleptics) professionally, and there's actual rigor there. Social science, not so much that I can see.

edit: I was inaccurate. What I did was NOT sensory science, but technology. We used the methods and insights from sensory scientists to get objective and quantitative results guiding product development. My apologies for my inaccuracy.

 

[Cosmik]

Putting it another way: science wants to have its cake and eat it.

It lays it on pretty thick when it (informally) claims or implies importance due to its rigour, its implied usefulness, its ability to discover the truth; to prove what it true. Even its ability to lend objective credence to 'value judgements'.

But at the same time, 'in the small print' it says:
"Disclaimer: we do not discover the truth or prove anything. In fact, it is up to you to decide whether to rely on anything we conclude - and our 'conclusions' are made of jelly with so many disclaimers on them you may as well toss a coin. 'Consensus' is meaningless, but you are all too stupid to spot the sleight of hand. It's all just probability, really - and the figures we give are bound to be wrong. And that's actually a fact. Probably. Science isn't even 'a thing' that can be defined so you can't hold us to anything. Don't blame us if you use what we've come up with and it all turns to schiit. It wasn't us, guv".

 

[Wombat]

I did start to read it. It seems to contradict @andreasmaaan when he says:

In the paper you link to, the author seems to be saying that you can have science even when using a methodology you 'make up yourself' based on 'pragmatism'..?

If so, I am still left thinking that science is somewhat 'malleable' even though the usual perception is that its usefulness is based on the opposite: 'rigour'.
(I know, it isn't meant to be useful, just as it doesn't find the truth nor prove anything. But appeals for funding are almost always based on science's usefulness and its ability to prove things and establish the truth Ditto when anyone refers to some science to back up a point they are making).

 

[andreasmaaan]

But at the same time, 'in the small print' it says:
"Disclaimer: we do not discover the truth or prove anything. In fact, it is up to you to decide whether to rely on anything we conclude - and our 'conclusions' are made of jelly with so many disclaimers on them you may as well toss a coin. 'Consensus' is meaningless, but you are all too stupid to spot the sleight of hand. It's all just probability, really - and the figures we give are bound to be wrong. And that's actually a fact. Probably. Science isn't even 'a thing' that can be defined so you can't hold us to anything. Don't blame us if you use what we've come up with and it all turns to schiit. It wasn't us, guv".

Ahahahaa! Mostly don't agree with this, but well put.

EDIT: I actually agree with the substance of this but not the tone. Science is the most rigorous process we have for understanding the natural world. It, like all sources of knowledge, is imperfect, but it is rigorous and controlled to a degree that other forms of investigation are not.

 

[andreasmaaan]

And I look forward to the same levels of rigour whenever anyone is recommending listening tests and statistics to establish which ethernet cable sounds best. Which speaker sounds best. Whether second harmonic distortion is 'preferred' over third, etc. All 'value judgements' based on another thing we cannot define without making another non-falsifiable judgement: music. If the claim is that if we use the word 'preference' instead of 'good', 'bad', 'best', 'worst', then this renders the experiment scientific, then I maintain that this is 'slipperiness' of the first order.

Sure! This is why listening tests can be valid scientifically only when the aim is to determine audibility thresholds, not when the aim is to determine preferences.

 

[svart-hvitt]

I'm not an audio scientist, my specialties are more molecular.

I have done non-audio sensory science (haptics, organoleptics) professionally, and there's actual rigor there. Social science, not so much that I can see.

edit: I was inaccurate. What I did was NOT sensory science, but technology. We used the methods and insights from sensory scientists to get objective and quantitative results guiding product development. My apologies for my inaccuracy.

It seems like you were involved in «techne», which is one of the pillars of science, but not the whole edifice?

 

[Wombat]

Sure! This is why listening tests are only valid scientifically when the aim is to determine audibility thresholds, not when the aim is to determine preferences.

Really? DBT can't measure preference? Tell us more.