Frequency analysis

Why Shouldn't We Use Frequency or Spectral Graphs to Compare Audio Codecs?

1. You can't know precisely what is audible and what is not audible just by looking at graphs. (Remember: It's music that we are talking about.)
2. FFT frequency response graphs are rough and can't tell practically anything about the temporal (short time) accuracy.
3. All psychoacoustic models are different. What can look good in graphs, doesn't necessarely sound too good and vice versa.
4. Graphical analysis is often useful, but only when used together with a listening test. The most useful view is the sound spectra view (spectral view in Adobe Audition).
5. Blind listening testing (ABX) is the only method that can really be justified when testing quality. Blind testing, when enough probability is reached, is not affected by anykind of bias or placebo effect.
6. Difference/substraction/inverse-mixing method is not conclusive either.

See the following links for more discussion

• Why subtracting one waveform from another is not a good analysis of encoding
• Why merely comparing waveforms is not really effective
• Why waveforms cannot be used to deduce quality, but may be useful to know what the encoder is doing