Audio Quality Measuring question

[dalbert02]

First, let me apologize if this is the wrong forum, mods feel free to move to where it would be most appropriate. I have a question about audio quality measuring, and General Audio seemed like the best place for this question.

I need suggestions on how to best measure audio quality. I work for one of the largest fire departments in the country and we currently test our radio system with a simple radio signal strength measurement followed by a "Can you hear me now?" test with a subjective test of signal strength and clarity. Obviously, each individual has a different opinion of signal strength on a scale of 1-5 and of clarity on a scale of 1-5. One person's 5 by 5 may be another person's 3 by 3. Even the same person may judge the same system differently on a different day. I heard of STIPA (Standardized Test of Intelligibility for Public Address) but I would like something like what we have here on this website. I'd like to compare a given signal to the same signal after it has been transmitted and received by our radio system. Somewhat surprising I have found that we can have poor radio signal strength but excellent audio quality or great radio signal strength and poor audio quality. I'd imagine we have spent tens of millions on the system and nearly nothing on analysis. I'd imagine an Audio Precision analyzer or similar analyzer would be a drop in the bucket. I'd love a simple frequency response and SNR value for each tested location so we can map out where we have good coverage and where we have poor coverage...not measured by radio signal strength but by audio clarity. Any suggestions, please?

 

[wwenze]

Are you testing for audibility? Are you sending out words on your system?

Similar testing exists for visual quality:

Just say a predetermined string of words at your target and tell him to repeat back.

Alternatively just record it with a mic and decide yourself. There are also differencing software that give a number for how close the new signal is to the original, but only human can decide what the audibility of words are.

 

[dalbert02]

Yes, we communicate with spoken words on our life safety radio system.
I'd like a repeatable measure of quality and not rely on the human factor. Audio quality can be subjective and different testers may be more skilled at hearing through noise and of course others may be less skilled.

 

[Chrispy]

If you have sufficient signal is one thing, too high a level/clipping, another. Don't know what your gear choices are particularly to test, tho.

 

[dougi]

Maybe take a look at the ITU-T Telephony subjective quality recommendations such as ITU-T P.800 and ITU-T P.830. Some parts of these may have suitable subjective opinion scales and the like.

 

[wwenze]

Put an Alexa in the room. See if it understands. No human factor.

 

[kemmler3D]

If you have the ability to record the output of the radio receiver (is there a line out?) you can use Deltawave to compare the output and the original using many metrics. The audio precision analyzer that Amir uses is needed because the gear being tested is often very hi-fi and the analyzer should be much more discerning than the gear being tested.

In your case it sounds like the audio is fairly obviously degraded once it gets to the receiver, so you don't need an ultra sensitive device to measure that. Just recording the audio with (say) a MOTU m2 and analyzing it on the computer should be useful.

That said, as @dougi points out, there are already standards for this kind of thing, so I would use those as a starting point.

Gear wise I don't think you need a lot.

In your case audio quality is basically tantamount to intelligibility, so mic technique and enunciation probably matter too?

 

[dougi]

Oh, I see you want to automate coverage quality analysis? This should be readily possible with one of the land mobile radio (LMR) drive test systems available. So talk to some of them and see if they can do what you want. Either manufacturers or drive test service providers.

PCTEL seems one US provider, but there will be others. Whats available may depend on what standard you are using (e.g. analog FM vs Tetra vs APCO P.25 etc).

 

[janbth]

Speaking as an amateur here: With a system like REW you can play test signals and measure the frequency response of your system at any listening position.

REW room acoustics and audio device measurement and analysis software

REW is free software for room acoustic measurement, loudspeaker measurement and audio device measurement and analysis

www.roomeqwizard.com

A relevant graph to look at is Clarity C50. Quoting from the help pages of REW: "C50 is most often used as an indicator of speech clarity".

Clarity Graph

 

[Gary_G]

For "one of the largest fire departments in the country" I am curious as to why this was not part of the purchase, installation and/or maintenance of the radio system. I would think final payment would be predicated on such a test.

It must be a complex system of radios, repeaters, etc. and there must be an existing contract for repair and maintenance of such an important communication system.

 

[Speedskater]

You might look into hearing aid Spoken Word tests. There are so many different Real World noises, that are nothing like the electronic noise measured in audio equipment.

 

[Pareto Pragmatic]

The audio quality will not be determined solely by the system, but also by the use. For example, how far the mic is held from the mouth, does that change in a truck with sirens blaring compared to being on scene? What is the "too close" distance to the mic, what is the "too far" distance? And should you go "too close" when a lot of noise is present, or will that make things worse?

Seems like a training issue to me. It also seems like a situation where the more a user uses it, the better they get at picking out meaning. So testing with experienced users and naive users seems like a good idea.

By all means max out the signal to noise ratio in the gear! But the users can make good signal still be hard to hear and comprehend.

 

[617]

Metrics such as linear and nonlinear distortion are not really relevant to your task. You need to look up metrics for intelligibiliIy. I believe there are methods by which you can get an intelligibility score without using human listeners to repeat novel sentences and rate how accurate they are.

Google 'dpa microphones intelligibility ' and you should find a helpful article.

Good luck.

 

[dougi]

There does also seem to be audio domain measurement systems as well. ABC-MRT, designed for first responder systems, and the STI/STIPA systems designed for room/PA intelligibility. An interesting topic this is!

 

[amirm]

Somewhat surprising I have found that we can have poor radio signal strength but excellent audio quality or great radio signal strength and poor audio quality.

This is quite common with digital/spread spectrum communication. You can have strong signal but if there is multipath, there will be data corruption. For this reason, such systems usually also have a metric of signal integrity.

 

[amirm]

Any suggestions, please?

This is a hard problem as creating a static RF situation is extremely hard. Run to run variations will exist for any objective metric.

There are objective measurements for speech. See: https://www.ap.com/analyzers-accessories/apx-overview/perceptual

But they are only good for first order approximation. Authoritative results are only achieved using humans. You could play phrases, record them and then conduct blind tests of intelligibility.

 

[617]

I believe the most important frequency band is that containing formants. Your mouth and tongue emphasize different harmonics which in differing proportions make vowel sounds. Preserving the integrity and relative magnitude of this frequency range, around 300 to 3k, seems to be critical for speech intelligibility.

Griesinger has some interesting observations about how the reflections from a blackboard situated about two feet from an instructor disturb precisely this frequency range, which is why intelligbility of a speaker facing a wall is poor.

If I was looking for a transducer for recording or playing back speech with maximal intelligbility I'd be looking for a broad peak from 300 to 3k and probably not much else, but I'd bet there is a ton of research out there on the subject from people designing aircraft headsets, hearing aids, emergency pa systems, alarm systems etc.

 

[dalbert02]

For "one of the largest fire departments in the country" I am curious as to why this was not part of the purchase, installation and/or maintenance of the radio system. I would think final payment would be predicated on such a test.

It must be a complex system of radios, repeaters, etc. and there must be an existing contract for repair and maintenance of such an important communication system.

You are correct. There is a measurement called DAQ - Delivered Audio Quality that the vendor must meet. Unfortunately, although sounding like a helpful parameter, it has little resemblance to real-world intelligibility.

 

[EERecordist]

First, let me apologize if this is the wrong forum, mods feel free to move to where it would be most appropriate. I have a question about audio quality measuring, and General Audio seemed like the best place for this question.

I need suggestions on how to best measure audio quality. I work for one of the largest fire departments in the country and we currently test our radio system with a simple radio signal strength measurement followed by a "Can you hear me now?" test with a subjective test of signal strength and clarity. Obviously, each individual has a different opinion of signal strength on a scale of 1-5 and of clarity on a scale of 1-5. One person's 5 by 5 may be another person's 3 by 3. Even the same person may judge the same system differently on a different day. I heard of STIPA (Standardized Test of Intelligibility for Public Address) but I would like something like what we have here on this website. I'd like to compare a given signal to the same signal after it has been transmitted and received by our radio system. Somewhat surprising I have found that we can have poor radio signal strength but excellent audio quality or great radio signal strength and poor audio quality. I'd imagine we have spent tens of millions on the system and nearly nothing on analysis. I'd imagine an Audio Precision analyzer or similar analyzer would be a drop in the bucket. I'd love a simple frequency response and SNR value for each tested location so we can map out where we have good coverage and where we have poor coverage...not measured by radio signal strength but by audio clarity. Any suggestions, please?

You might contact Biamp Labs. All the people I knew there either retired of moved to other audio companies. They migrated over the years in their product line then were merged.

They make distributed PA systems, for instance in hospitals. They have DSPs to implement your question, so they would have the measurement expertise. I would find their VP of engineering on LinkedIn or someone in the Beaverton, Oregon metro in engineering and ask.

I managed part of a 4G specification. In digital radios, the voice is compressed and encoded to a spec. Then that data stream is passed to the transport layer, then decoded back at the other end from the transport layer. What happens between the transmitter and receiver transport sockets is very complex. There is a lot of error correction, forward error correction, and cross-layer dynamic optimization to and from the radio channel, which in itself is complex depending on noise floor, multipath, as mentioned, vegetation, topography, and antennas.

What is the operating frequency of the uplink and downlink? Does the radio vendor have sufficient technical resources in house or contracted to have measured what you are asking? I would also ask your vendor if there is a diagnostic mode on your receivers of received dBm you can see. That would be the numerical versions of your bars of signal strength. From memory, it might range from -140 to -80. There are also probably technical resources at First Net.