BS.1387: Difference between revisions

From Hydrogenaudio Knowledgebase
VisualWikitext
(EAQUAL is not part of PEAQ. It is exactly based on PEAQ the standard.)
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''ITU-R recommendation BS.1387''' is the document that defines '''Perceptual Evaluation of Audio Quality''' (PEAQ), an ''objective'' measurement technique used to measure the quality of encoded/decoded audio files. It acts in contrast to more the common place ''subjective'' testing methodology deployed using [[ABX]] and [[ABC/HR]] reference testing -- frequently preferred by hydrogenaudio. PEAQ returns an "ODG" rating, which is intended to match the difference in subjective (1–5) scores between the two input samples.
'''ITU-R recommendation BS.1387''' is the document that defines '''Perceptual Evaluation of Audio Quality''' (PEAQ), an ''objective'' measurement technique used to measure the quality of encoded/decoded audio files. It acts in contrast to more the common place ''subjective'' testing methodology deployed using [[ABX]] and [[ABC/HR]] reference testing -- frequently preferred by hydrogenaudio. PEAQ returns an "ODG" rating, which is intended to match the difference in subjective (1–5) scores between the two input samples.


'''EAQUAL''' (''Evaluation Of Audio Quality'') is an open-source software that implements PEAQ. Several tests have been performed using EAQUAL most notably using numerous [http://www.hydrogenaudio.org/forums/index.php?showtopic=20264 AAC encoders] to determine via a [http://en.wikipedia.org/wiki/Pearson_correlation Pearson Correlation] the linear relationship between human ratings and EAQUAL ratings on a given set of test samples. The results, however when using objective testing methodologies are still inconclusive and mostly only used by codec developers and researchers.  
== Structure ==
PEAQ has two versions: basic and advanced. The basic version only uses an FFT-based ear model and is easier to compute. The advanced version uses both FFT and filter bank and is expected to be more accurate.


===Testing Using EAQUAL===
== History ==
As of version 0.1.3alpha, the ''-h'' argument can be used to find out how to use eaqual (ex: ''eaqual -h'').<br>
To compare a test wave file to a reference wave file, one can use for example: ''eaqual -fref ref.wav -ftest test.wav''.<br>


===Interpreting the Results Output by EAQUAL===
BS.1387 was initially published in 1998. It was updated to BS.1387-1 in 2001 and BS.1387-2 in 2023.
The results of EAQUAL are interpreted using what is known as an ODG score.  
''Note: this information needs to be expanded.''<br>


===Status of the EAQUAL Project===
* BS.1387-1 includes important technical corrections -- ones that are important to reach the standard's own conformance criteria.<ref>https://www.opticom.de/download/CorrectionstoBS1387.pdf</ref>
* BS.1387-2 seems to have no real change, except for removal of references to BS.1115, addition of a table of contents, and extensive reformatting.


===External links:===  
== EAQUAL ==
'''EAQUAL''' (''Evaluation Of Audio Quality'') is an open-source software that implements PEAQ's basic model ''only''.


* [http://www.mp3-tech.org/programmer/sources/eaqual.tgz EAQUAL Sourcecode] linux archive of c code used to implement EAQUAL provided by Gabriel Bouvigne.  
=== Invoking EAQUAL ===
As of version 0.1.3alpha, the ''-h'' argument can be used to find out how to use eaqual (ex: ''eaqual -h'').
 
To compare a test wave file to a reference wave file, one can use for example: ''eaqual -fref ref.wav -ftest test.wav''.
 
=== Interpreting EAQUAL output ===
EAQUAL outputs one score, the PEAQ "ODG" rating. This ODG (Objective Difference Grade) rating is designed by ITU to match an SDG (Subjective Difference Grade) rating, which is the difference between the subjective (1&ndash;5) scores between the two input samples. Assuming the HydrogenAudio subjective scoring system, where the reference sample is always scored as a perfect 5, adding 5 points to ODG should produce an approximation of the subjective score.
 
=== Status of the project ===
Development of EAQUAL was halted in 2002 due to patent concerns. This is not a problem for PEAQ compilance, however, considering the 2001 BS.1387-1 does not differ substantially from the 2023 version.
 
The ITU patent declaration system does not list any specific PEAQ patent by number. However, no new patents have been added since 1998, so any patent should have expired by 2018.<ref>https://www.itu.int/en/ITU-R/study-groups/Pages/itu-r-patent-information.aspx</ref>
 
Versions of EAQUAL include:
* [http://www.mp3-tech.org/programmer/sources/eaqual.tgz EAQUAL Sourcecode] linux archive of c code used to implement EAQUAL provided by Gabriel Bouvigne, mirrored on github by [https://github.com/spxnn/eaqual spxnn]
* [http://www.rarewares.org/others.html EAQUAL Tools] zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.
* [http://www.rarewares.org/others.html EAQUAL Tools] zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.
* [https://github.com/ivan-codelegs/eaqual ivan-codelegs] github fork, adds macOS support
== GstPEAQ ==
[https://github.com/HSU-ANT/gstpeaq GstPEAQ] is an implementation of PEAQ, ''both'' basic and advanced, in GStreamer. In addition to the ODG, it also outputs the distortion index (DI), which is not clipped at extremes and not fitted to score anchors. On the HA multiformat dataset:
* The advanced model gives a correlation improvement of ~0.2 over basic;
* DI is slightly better at predicting subjective scores than ODG, with a correlation improvement of ~0.03.
== Comparison with subjective listening tests ==
* [http://www.hydrogenaudio.org/forums/index.php?showtopic=20264 EAqual results for the AAC@128v2 listening test] - fair Pearson correlation (0.699) among higher-quality samples: all AAC
* [https://hydrogenaud.io/index.php/topic,124607.msg1031323.htmlGstPEAQ: PEAQ done right, allegedly || Multiformat correlation] - great Pearson correlation (0.924, DI Adv): samples of three quality groups
HA comparisons between PEAQ and human raters remain inconclusive. PEAQ is considered useful for an approximation of human senses in codec development and research, but concrete results still need human participation.
== Other implementations ==
PEAQ-Basic is simple enough to have many implementations.
* [https://sourceforge.net/projects/peaqb/ peaqb] is another implementation of PEAQ. Last updated 2003.
* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.
== Other objective metrics ==
PEAQ is not the end. There are other metrics:<ref>https://github.com/jonnor/machinehearing/blob/09b5060bd03b8a49fc1d0afd8eedba4babca83ca/audio-quality/README.md</ref>
* ITU also has PESQ and POLQA, both designed for speech.
* [https://github.com/google/visqol VISQOL] is Google's open-source metric. It works for both speech and music, but the neural network (don't worry, it runs fast enough on a CPU) is trained for short clips only. Maybe someone can write a tool to one file into many clips and see individual segment scores.
* CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.
There are also much more primitive methods that don't attempt anything perceptual, preferred by peddlers of Bluetooth codecs (they make more sense for analogue systems like DACs):
* SNR
* THD+N


== External links==
* [[Wikipedia:Perceptual Evaluation of Audio Quality]]
* [https://www.itu.int/rec/R-REC-BS.1387 ITU BS.1387] download -- free full text of the standard, straight from the official site.
<references />


[[Category:Software]]
[[Category:Software]][[Category:Quality measurement]]

Latest revision as of 00:07, 25 August 2023

ITU-R recommendation BS.1387 is the document that defines Perceptual Evaluation of Audio Quality (PEAQ), an objective measurement technique used to measure the quality of encoded/decoded audio files. It acts in contrast to more the common place subjective testing methodology deployed using ABX and ABC/HR reference testing -- frequently preferred by hydrogenaudio. PEAQ returns an "ODG" rating, which is intended to match the difference in subjective (1–5) scores between the two input samples.

Structure

PEAQ has two versions: basic and advanced. The basic version only uses an FFT-based ear model and is easier to compute. The advanced version uses both FFT and filter bank and is expected to be more accurate.

History

BS.1387 was initially published in 1998. It was updated to BS.1387-1 in 2001 and BS.1387-2 in 2023.

  • BS.1387-1 includes important technical corrections -- ones that are important to reach the standard's own conformance criteria.[1]
  • BS.1387-2 seems to have no real change, except for removal of references to BS.1115, addition of a table of contents, and extensive reformatting.

EAQUAL

EAQUAL (Evaluation Of Audio Quality) is an open-source software that implements PEAQ's basic model only.

Invoking EAQUAL

As of version 0.1.3alpha, the -h argument can be used to find out how to use eaqual (ex: eaqual -h).

To compare a test wave file to a reference wave file, one can use for example: eaqual -fref ref.wav -ftest test.wav.

Interpreting EAQUAL output

EAQUAL outputs one score, the PEAQ "ODG" rating. This ODG (Objective Difference Grade) rating is designed by ITU to match an SDG (Subjective Difference Grade) rating, which is the difference between the subjective (1–5) scores between the two input samples. Assuming the HydrogenAudio subjective scoring system, where the reference sample is always scored as a perfect 5, adding 5 points to ODG should produce an approximation of the subjective score.

Status of the project

Development of EAQUAL was halted in 2002 due to patent concerns. This is not a problem for PEAQ compilance, however, considering the 2001 BS.1387-1 does not differ substantially from the 2023 version.

The ITU patent declaration system does not list any specific PEAQ patent by number. However, no new patents have been added since 1998, so any patent should have expired by 2018.[2]

Versions of EAQUAL include:

  • EAQUAL Sourcecode linux archive of c code used to implement EAQUAL provided by Gabriel Bouvigne, mirrored on github by spxnn
  • EAQUAL Tools zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.
  • ivan-codelegs github fork, adds macOS support

GstPEAQ

GstPEAQ is an implementation of PEAQ, both basic and advanced, in GStreamer. In addition to the ODG, it also outputs the distortion index (DI), which is not clipped at extremes and not fitted to score anchors. On the HA multiformat dataset:

  • The advanced model gives a correlation improvement of ~0.2 over basic;
  • DI is slightly better at predicting subjective scores than ODG, with a correlation improvement of ~0.03.

Comparison with subjective listening tests

HA comparisons between PEAQ and human raters remain inconclusive. PEAQ is considered useful for an approximation of human senses in codec development and research, but concrete results still need human participation.

Other implementations

PEAQ-Basic is simple enough to have many implementations.

  • peaqb is another implementation of PEAQ. Last updated 2003.
  • There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.

Other objective metrics

PEAQ is not the end. There are other metrics:[3]

  • ITU also has PESQ and POLQA, both designed for speech.
  • VISQOL is Google's open-source metric. It works for both speech and music, but the neural network (don't worry, it runs fast enough on a CPU) is trained for short clips only. Maybe someone can write a tool to one file into many clips and see individual segment scores.
  • CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.

There are also much more primitive methods that don't attempt anything perceptual, preferred by peddlers of Bluetooth codecs (they make more sense for analogue systems like DACs):

  • SNR
  • THD+N

External links

  1. ↑ https://www.opticom.de/download/CorrectionstoBS1387.pdf
  2. ↑ https://www.itu.int/en/ITU-R/study-groups/Pages/itu-r-patent-information.aspx
  3. ↑ https://github.com/jonnor/machinehearing/blob/09b5060bd03b8a49fc1d0afd8eedba4babca83ca/audio-quality/README.md
Retrieved from "https://wiki.hydrogenaud.io/index.php?title=BS.1387&oldid=37340"