• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Opus codec and why its magic

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
Yeah, I can calculate PSNR, SNR, SegSNR, and lots of other SNR's on audio too.

But what do they tell us?

There is a world of difference between a video codec whose error pattern is, oh, say +1+1+1+1-1-1-1-1 and one that is just +1 or -1, but the PSNR remains the same.

YES?
You missed my edit. ;)

We use a raft of technical measures to assess performance (audio or video) , as of course you know. :) . They help paint an overall picture of performance. That is not suggesting one measurand is the answer to all questions or a single number describes quality.

A few years ago I spent a fascinating day at BBC research and development performing controlled picture quality testing. Part of a complaint about their HD encoder picture quality. The new encoder had specific picture quality issues despite notionally being technically better than the old. Very much an objective technical and controlled subjective exercise.

BBC have an interesting range of papers and articles available

https://www.bbc.co.uk/rd

https://www.bbc.co.uk/rd/publications

https://www.bbc.co.uk/rd/topics/quality

https://www.bbc.co.uk/rd/projects/video-coding
 
Last edited:

q3cpma

Major Contributor
Joined
May 22, 2019
Messages
3,060
Likes
4,416
Location
France
You can but it's almost useless (especially PSNR). Even when using "advanced" metrics like dSSIM, mSSIM and VMAF, developing encoders while using them as goals leads to failures like VP9 (or more accurately, libvpx). Only VMAF seem good to me, especially since it was made for video first (meaning that temporal components aren't an afterthought).

x264, for example, is known for having completely disregarded these metrics in the later part of its developement, allowing it the become the reference AVC encoder over hundreds of metrics-tuned commercial encoders. This is also why it's still superior or equal to x265 when targeting transparency at 1080p.

I follow the developement of rav1e, and I think that their approach of using multiple metrics to spot regressions is a good way of using them.
 
OP
Y

yuki980e

Member
Joined
Aug 3, 2019
Messages
16
Likes
4
Yeah ABX is useful but the easiest way to stress a lossy audio codec. Is feed it complex/noisy music to me Opus/Musepack are okay in the 128kbps area but for safety i would stick with 192kbps. I've noticed HA users get angry when noticed i artifacts in the 80 - 192kbps area in that style without needing to do 8 trials just the demo mode, but MPC is transparent at Q5 on all them. lol
 

Soniclife

Major Contributor
Forum Donor
Joined
Apr 13, 2017
Messages
4,500
Likes
5,417
Location
UK
Yeah ABX is useful but the easiest way to stress a lossy audio codec. Is feed it complex/noisy music
I've found the same, stay way from sparse well recorded stuff, bombard it with a wall of sound to encode.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
You can but it's almost useless (especially PSNR). Even when using "advanced" metrics like dSSIM, mSSIM and VMAF, developing encoders while using them as goals leads to failures like VP9 (or more accurately, libvpx). Only VMAF seem good to me, especially since it was made for video first (meaning that temporal components aren't an afterthought).

x264, for example, is known for having completely disregarded these metrics in the later part of its developement, allowing it the become the reference AVC encoder over hundreds of metrics-tuned commercial encoders. This is also why it's still superior or equal to x265 when targeting transparency at 1080p.

I follow the developement of rav1e, and I think that their approach of using multiple metrics to spot regressions is a good way of using them.
To state that technical objective measures are "useless" is nonsense.

Just as a simple start.

https://www.bbc.co.uk/rd/blog/2016-...lash-avc-50-percent-bit-rate-savings-verified
 
OP
Y

yuki980e

Member
Joined
Aug 3, 2019
Messages
16
Likes
4
I've found the same, stay way from sparse well recorded stuff, bombard it with a wall of sound to encode.

I've had dark ambient artifact with 128kbps AAC needed +224kbps to fix it, Lossy still struggles with classical instruments and industrial sounds too. I find it bit rich few still hype lossy in 2019, But still no guarantee on it working on heavy music without it being 256kbps. A +200GB micro SD card is cheaper and nearly everything now supports Flac.

With Musepack some noisy stuff avg 350 - 790kbps on the Q5 setting, may as well use flac.
 

q3cpma

Major Contributor
Joined
May 22, 2019
Messages
3,060
Likes
4,416
Location
France
To state that technical objective measures are "useless" is nonsense.

Just as a simple start.

https://www.bbc.co.uk/rd/blog/2016-...lash-avc-50-percent-bit-rate-savings-verified
"Technical objective measures" isn't a magic word for "measures that relate well to perceptual quality". The only thing you can deduce from your paper is that there's correlation in these examples, that's all; if you read your paper in detail, you'll see that MOS and PSNR aren't consistent in their linearity at all.
Also, why did you ignore the rest of my post about x264?

By the way, that paper is just a joke:
* Video encoder comparison using only one quite bad (biaised toward denoising and spatial only) metric and ignore SSIM/VQM was unheard of in 2016
* BBC having stakes in HEVC is a strong clue for why they chose an outdated metric conveniently favouring a PSNR tuned codec
* Using the reference AVC encoder and not x264, which is the real world leader
* Using unrealistically low bitrates
* The encoder exact command lines aren't given

Honestly, I'm not trying to start a fight, but this paper was ridiculed on doom9 (http://forum.doom9.org/archive/index.php/t-173071.html) when it came out and that for good reasons. It just shows that papers and marketing aren't always distinct.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
"Technical objective measures" isn't a magic word for "measures that relate well to perceptual quality". The only thing you can deduce from your paper is that there's correlation in these examples, that's all; if you read your paper in detail, you'll see that MOS and PSNR aren't consistent in their linearity at all.
Also, why did you ignore the rest of my post about x264?

By the way, that paper is just a joke:
* Video encoder comparison using only one quite bad (biaised toward denoising and spatial only) metric and ignore SSIM/VQM was unheard of in 2016
* BBC having stakes in HEVC is a strong clue for why they chose an outdated metric conveniently favouring a PSNR tuned codec
* Using the reference AVC encoder and not x264, which is the real world leader
* Using unrealistically low bitrates
* The encoder exact command lines aren't given

Honestly, I'm not trying to start a fight, but this paper was ridiculed on doom9 (http://forum.doom9.org/archive/index.php/t-173071.html) when it came out and that for good reasons. It just shows that papers and marketing aren't always distinct.
.. Mmmmmm... Now let me see.... Internationally technically renowned public broadcaster or random guy on the internet.......

You lose....;)

That was just one simple example, clearly not intended to be taken as exhaustive. I have no intention of getting into this and wasting my time.

If you beleive objective technical measurements are "useless" so be it. BTW that opinion won't get you very far in this forum.

Also, you mentioned marketing. You do realise that the BBC is not a commercial organisation?
 
Last edited:

BillG

Major Contributor
Joined
Sep 12, 2018
Messages
1,699
Likes
2,266
Location
Auckland, New Zealand
I personally encode all my FLAC downloads to -q 0.95 these days within NeroAAC (*.ma4) - This averages out to ~380 kbps. It's indistinct from the original to me, and offers me ~50% space saving as well.

Oh, but it's lossy! Yeah? So is my hearing... :p Oh, but storage is cheap these days! So? I paid for my storage, and will manage it as I see fit... :D
 

q3cpma

Major Contributor
Joined
May 22, 2019
Messages
3,060
Likes
4,416
Location
France
Ignoring all the arguments, appeal to authority and still using "objective technical measurements" as a magical mantra. Nice.
>If you beleive objective technical measurements are "useless" so be it.
Where did I say that? I said this particular set of metrics in this application are almost useless. I'm very fond of Genelec and Neumann on the speaker front.
>Also, you mentioned marketing. You do realise that the BBC is not a commercial organisation?
You know that the BBC is part of HEVC's patent pool, right?
 
  • Like
Reactions: DDF

j_j

Major Contributor
Audio Luminary
Technical Expert
Joined
Oct 10, 2017
Messages
2,267
Likes
4,759
Location
My kitchen or my listening room.
Ignoring all the arguments, appeal to authority and still using "objective technical measurements" as a magical mantra. Nice.
>If you beleive objective technical measurements are "useless" so be it.
Where did I say that? I said this particular set of metrics in this application are almost useless. I'm very fond of Genelec and Neumann on the speaker front.
>Also, you mentioned marketing. You do realise that the BBC is not a commercial organisation?
You know that the BBC is part of HEVC's patent pool, right?

I'll have to chime in here. "Normal" technical measurements like SNR, THD, and the like are "mostly harmless", much after the thought about "Earth" in the Hitchhiker's guide to the Galaxy. That includes PSNR as well.

Complex, carefully worked machine-based measurements using hearing (or vision) models are better. They are, however, quite limited in their accuracy because they must have some model of perception built in. Things that match their idea of perception will measure better, things that do not measure worse. I won't say they are useless, but they are not terribly accurate unless one measures among similar impairments. Both PEAQ and PESQ can be accidentally broken by some inputs and coding methods, often leading to an inappropriate dismissal of particular techniques.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
Ignoring all the arguments, appeal to authority and still using "objective technical measurements" as a magical mantra. Nice.
>If you beleive objective technical measurements are "useless" so be it.
Where did I say that? I said this particular set of metrics in this application are almost useless. I'm very fond of Genelec and Neumann on the speaker front.
>Also, you mentioned marketing. You do realise that the BBC is not a commercial organisation?
You know that the BBC is part of HEVC's patent pool, right?

ME
"We use a raft of technical measures to assess performance (audio or video)"

YOU
"You can but it's almost useless (especially PSNR). "

The BBC is publicly funded and independently scrutinised. Their purpose is to deliver public service broadcasting. Technical research and development is part of that remit, which is to deliver the best technical quality, production innovation and cost. They have a very long track record of technical innovation that obviously includes patented technology. You will need to try much harder than that to justify your implication of bias or inappropriate technical preference/steering.

The appeal to authority was actually an invitation to YOU to take a look through the BBCs numerous R&D papers which show that use of "objective technical measures" are clearly not "useless".
 
Last edited:

j_j

Major Contributor
Audio Luminary
Technical Expert
Joined
Oct 10, 2017
Messages
2,267
Likes
4,759
Location
My kitchen or my listening room.
The appeal to authority was actually an invitation to YOU to take a look through the BBCs numerous R&D papers which show that use of "objective technical measures" are clearly not "useless".

Your assumption that I haven't is kind of insulting, really. Furthermore, you're mixing several issues here, and then trying to frame my position quite out of the context it was given.

Technical measures are useful indeed, when you have a way to relate them to what you know is important.

For many things, like modems, transmission, etc, they are the way to go. They are the germane thing to measure. But that's not what we are talking about here, we are talking about the perceived quality of a perceptual codec. That, frankly, is a very different thing.

As I mentioned, there are PEAQ and PESQ and the surrounding standards, which work "ok" when you have a consistent distortion mechanism across the things you are testing. If you use two algorithms that use different designs for a perceptual model, though, you're not quite, but almost completely sunk, and comparisons, except at "very very good" and "very very bad" are not very meaningful at all.

For evaluating unknown systems on a perceptual basis, purely mechanical measurements are not so good. Not for audio, not for video. Once you have a perceptual model involved, testing is "listen, listen, and listen". Now, if you're calling BS1116 a "measurement", I'd agree, but I don't think mean and distribution of a subjective test using the ITU impairment scale is what you meant.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
Your assumption that I haven't is kind of insulting, really. Furthermore, you're mixing several issues here, and then trying to frame my position quite out of the context it was given.

Technical measures are useful indeed, when you have a way to relate them to what you know is important.

For many things, like modems, transmission, etc, they are the way to go. They are the germane thing to measure. But that's not what we are talking about here, we are talking about the perceived quality of a perceptual codec. That, frankly, is a very different thing.

As I mentioned, there are PEAQ and PESQ and the surrounding standards, which work "ok" when you have a consistent distortion mechanism across the things you are testing. If you use two algorithms that use different designs for a perceptual model, though, you're not quite, but almost completely sunk, and comparisons, except at "very very good" and "very very bad" are not very meaningful at all.

For evaluating unknown systems on a perceptual basis, purely mechanical measurements are not so good. Not for audio, not for video. Once you have a perceptual model involved, testing is "listen, listen, and listen". Now, if you're calling BS1116 a "measurement", I'd agree, but I don't think mean and distribution of a subjective test using the ITU impairment scale is what you meant.

Sorry JJ but I think you have misunderstood the posts. Always a danger with text (email etc) communication.

My primary responses have been to other posters which are saying technical objective measurements are "useless". My comments (including those specific ones you quoted) have not been aimed at you or your input.

Also my post above said "As of course you know :)" . I know full well your qualification and experience, so I dont understand your offended response unless it were due to a misunderstanding, or that we have some fundamentally different definition of "useless".

I have not attempted to "frame" your position in any way whatsoever.

I have however mentioned quite clearly above that these technical objective measures do not paint the whole picture, and that subjective assessment is an intrinsic part of the process. A process which I have been involved with at the BBC (all be it in a limited way). In fact that involvement was due to a formal complaint about their HD broadcast picture quality (subjectively observable problems) even though technical measurements would lead you to believe it was an overall improved encoder. So I really do "get it" :)

So I dont think we are at odds at all here. :)
 
Last edited:

q3cpma

Major Contributor
Joined
May 22, 2019
Messages
3,060
Likes
4,416
Location
France
ME
"We use a raft of technical measures to assess performance (audio or video)"

YOU
"You can but it's almost useless (especially PSNR). "
You're obviously being dishonest. Your "If you beleive objective technical measurements are "useless" so be it" sentence can only mean that I believe that ALL "objective technical measurements" are useless, which I denied because it's simply wrong (and the equivalent of sceaming heresy, in here).

The BBC is publicly funded and independently scrutinised. Their purpose is to deliver public service broadcasting. Technical research and development is part of that remit, which is to deliver the best technical quality, production innovation and cost. They have a very long track record of technical innovation that obviously includes patented technology. You will need to try much harder than that to justify your implication of bias or inappropriate technical preference/steering.
YOU will have to try much harder than that (which is not hard since you ignored all technical arguments since the beginning) to explain why they used the useless PSNR ALONE for an oversmoothing codec (cf. SAO) while almost everyone else uses at least SSIM in addition or instead. No, your diatribe won't replace technical arguments.

The appeal to authority was actually an invitation to YOU to take a look through the BBCs numerous R&D papers which show that use of "objective technical measures" are clearly not "useless".
Looks like to me that you can only paste papers and can't have any technical discussion about the subject. I hope that's why you consistently ignored the technical side of this conversation.

How about you read more papers about why PSNR doesn't relate well to image quality (like https://arxiv.org/abs/1711.00693) and even less to video quality.
 

q3cpma

Major Contributor
Joined
May 22, 2019
Messages
3,060
Likes
4,416
Location
France
By the way, one should wonder a bit why a "public interest" entity like the BBC doesn't support a more open, not patent based and royalty-free codec like AV1.
 

Krunok

Major Contributor
Joined
Mar 25, 2018
Messages
4,600
Likes
3,065
Location
Zg, Cro
By the way, one should wonder a bit why a "public interest" entity like the BBC doesn't support a more open, not patent based and royalty-free codec like AV1.

In broadcasting business codec is not something that is replaced easilly as it is incorporated into a complex chain of production. Codec royalties are not significant part of production expenses.
 

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
You're obviously being dishonest. Your "If you beleive objective technical measurements are "useless" so be it" sentence can only mean that I believe that ALL "objective technical measurements" are useless, which I denied because it's simply wrong (and the equivalent of sceaming heresy, in here).


YOU will have to try much harder than that (which is not hard since you ignored all technical arguments since the beginning) to explain why they used the useless PSNR ALONE for an oversmoothing codec (cf. SAO) while almost everyone else uses at least SSIM in addition or instead. No, your diatribe won't replace technical arguments.


Looks like to me that you can only paste papers and can't have any technical discussion about the subject. I hope that's why you consistently ignored the technical side of this conversation.

How about you read more papers about why PSNR doesn't relate well to image quality (like https://arxiv.org/abs/1711.00693) and even less to video quality.
No dishonesty, just responses to your comments which were incorrect.

I have not ignored the technical conversation, I have referred you to an organisation that will know more about the subject than you do ;) That means I dont have to waste my time arguing with random people on the internet.
 
Last edited:

March Audio

Master Contributor
Audio Company
Joined
Mar 1, 2016
Messages
6,378
Likes
9,317
Location
Albany Western Australia
By the way, one should wonder a bit why a "public interest" entity like the BBC doesn't support a more open, not patent based and royalty-free codec like AV1.


Pure comedy. :) It absolutely does.

If you had actually bothered to read the links I posted you would have seen

In addition to this work, the video coding team is also looking into even newer and more advanced solutions to ensure the BBC is future-proof in terms of delivering content to audiences. BBC R&D is now researching and testing new and emerging video compression standards, like the royalty free AV1, and the VVC standard, which will replace HEVC in the coming years. In doing so, we are always striving to make our findings public, to share knowledge and to help move the industry in the right direction.

OR

BBC R&D has released the first version of the Turing codec, an open source software HEVC video encoder that allows highly efficient compression of video content with low computational complexity.
https://www.bbc.co.uk/rd/blog/2016-09-turing-codec


Anyway, you are one for the ignore button
 
Last edited:

j_j

Major Contributor
Audio Luminary
Technical Expert
Joined
Oct 10, 2017
Messages
2,267
Likes
4,759
Location
My kitchen or my listening room.
Also my post above said "As of course you know :)" . I know full well your qualification and experience, so I dont understand your offended response unless it were due to a misunderstanding, or that we have some fundamentally different definition of "useless".


...



So I dont think we are at odds at all here. :)

Well, good.
 
Top Bottom