Is there any syntetic tests for audio (like SSIM/PNSR for video) exists?
If yes is it possible to chech same samples with this tests in order to compare how accurate such syntetic tests are?
It's really VERY interesting!
Thanx a lot!
Full Version: Automated quality testing
QUOTE(dimzon @ Mar 22 2006, 06:18 PM)
Is there any syntetic tests for audio (like SSIM/PNSR for video) exists?
If yes is it possible to chech same samples with this tests in order to compare how accurate such syntetic tests are?
It's really VERY interesting!
Thanx a lot!
If yes is it possible to chech same samples with this tests in order to compare how accurate such syntetic tests are?
It's really VERY interesting!
Thanx a lot!
PEAQ is well-known. It will fail grossly for HE-AAC. People did correlation tests of PEAQ results with our listening test results before.
Generally the tool isn't bad, but if you depend on it blindly or take it as the last word you are making a big mistake.
QUOTE(Garf @ Mar 22 2006, 08:22 PM)
PEAQ is well-known. It will fail grossly for HE-AAC. People did correlation tests of PEAQ results with our listening test results before.
Generally the tool isn't bad, but if you depend on it blindly or take it as the last word you are making a big mistake.
Generally the tool isn't bad, but if you depend on it blindly or take it as the last word you are making a big mistake.
Where to get it?
There was EAQUAL but it was shut down due to patent issues... it incorporated basic model of PEAQ.
Also, there was another independent implementation by McGill university - maybe you could look it up. It was also basic model.
There is commercial implementation, having also Advanced PEAQ model from Opticom.
Basically, both basic and advanced PEAQ fail miserably at low bit rates (PEAQ was designed to measure small impairments as specified in BS.1116 guidelines for subjective testing), and they fail even more miseralby if parametric tools are used, like in SBR and PS - furthermore, PEAQ does not have any binaural model implemented, so it would rank PS files as absolutely horrible.
This has been acknowledged - and there might be some improvements to PEAQ in the years to come, especially as there is a lot of new advances in audio coding since the PEAQ was standardised.
Also, there was another independent implementation by McGill university - maybe you could look it up. It was also basic model.
There is commercial implementation, having also Advanced PEAQ model from Opticom.
Basically, both basic and advanced PEAQ fail miserably at low bit rates (PEAQ was designed to measure small impairments as specified in BS.1116 guidelines for subjective testing), and they fail even more miseralby if parametric tools are used, like in SBR and PS - furthermore, PEAQ does not have any binaural model implemented, so it would rank PS files as absolutely horrible.
This has been acknowledged - and there might be some improvements to PEAQ in the years to come, especially as there is a lot of new advances in audio coding since the PEAQ was standardised.
Ok, sorry for offtopic (moderator, please, split thread ), does anybody interested in such metrics at all? Maybe we can create our own metrics wich will be close to listening tests?
What does You thing?
I belive such tool will be very useful for automatic codec tuning in dvelopment stage...
What does You thing?
I belive such tool will be very useful for automatic codec tuning in dvelopment stage...
Well, such tools are quite complex and take a lot of domain specific knowledge. The area is full of patents as well.
Consider that making one amounts to making an advanced psychoacoustic model.
Given that nobody could even be arsed to make one for FAAC, I think the odds aren't good that something decent will pop up.
The SoundExpert thing seems to be a bit in the middle between this and listening tests. (Sharing the good and bad sides of both, too).
Consider that making one amounts to making an advanced psychoacoustic model.
Given that nobody could even be arsed to make one for FAAC, I think the odds aren't good that something decent will pop up.
The SoundExpert thing seems to be a bit in the middle between this and listening tests. (Sharing the good and bad sides of both, too).
QUOTE(Garf @ Mar 22 2006, 08:22 AM)
PEAQ is well-known. It will fail grossly for HE-AAC. People did correlation tests of PEAQ results with our listening test results before.
It will fail grossly when handed the output of several of the more advanced codecs. Hard problem.
QUOTE(Garf @ Mar 23 2006, 02:18 AM)
Consider that making one amounts to making an advanced psychoacoustic model.
It's actually even harder than that because you can't just use the error signal and consider it as the noise because of the vocoder-like algorithms. You need to sort of compare the perception of the waveform.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.