LossyWAV: Difference between revisions

From Hydrogenaudio Knowledgebase
No edit summary
Line 4: Line 4:
| caption =  
| caption =  
| maintainer = [http://www.hydrogenaudio.org/forums/index.php?showuser=42400 Nick.C]
| maintainer = [http://www.hydrogenaudio.org/forums/index.php?showuser=42400 Nick.C]
| stable_release = v0.9.2 RC3
| stable_release = 1.0.0
| preview_release = beta v0.9.8d
| preview_release =  
| operating_system = [[Wikipedia:Microsoft Windows|Windows]]
| operating_system = [[Wikipedia:Microsoft Windows|Windows]]
| use = [[Wikipedia:Digital signal processing|Digital signal processing]]
| use = [[Wikipedia:Digital signal processing|Digital signal processing]]
| license = [[Wikipedia:GNU Lesser General Public License|GNU LGPL]]
| license = [[Wikipedia:GNU General Public License|GNU GPL]]
| website = [http://www.hydrogenaudio.org/forums/index.php?showtopic=56129 Hydrogenaudio]
| website = [http://www.hydrogenaudio.org/forums/index.php?showtopic=56129 Hydrogenaudio]
}}
}}
Line 26: Line 26:
It must be stressed that lossyWAV is a pure [[variable bit rate]] pre-processor. Bits-to-remove from the audio data are calculated on a block-by-block basis (codec-block length = 512 samples, 11.6msec @ 44.1kHz) using overlapping [[fast Fourier transform]] (FFT) analyses of at least two lengths (default quality preset (-q 5) = 64 & 1024 [[Sampling %28signal processing%29|samples]]). After some manipulation, the results of each FFT analysis for a specific codec-block are then grouped and the minimum value used to determine bits-to-remove for the whole codec-block. Bit removal adds noise to the output, however the added noise has been pre-calculated and its level will be at or below the noise floor of the codec_block in question. Each sample in the codec-block is then rounded such that the first <bits-to-remove> lsb's are zero. In this way the wasted bits feature of [[FLAC]] et al is exploited.
It must be stressed that lossyWAV is a pure [[variable bit rate]] pre-processor. Bits-to-remove from the audio data are calculated on a block-by-block basis (codec-block length = 512 samples, 11.6msec @ 44.1kHz) using overlapping [[fast Fourier transform]] (FFT) analyses of at least two lengths (default quality preset (-q 5) = 64 & 1024 [[Sampling %28signal processing%29|samples]]). After some manipulation, the results of each FFT analysis for a specific codec-block are then grouped and the minimum value used to determine bits-to-remove for the whole codec-block. Bit removal adds noise to the output, however the added noise has been pre-calculated and its level will be at or below the noise floor of the codec_block in question. Each sample in the codec-block is then rounded such that the first <bits-to-remove> lsb's are zero. In this way the wasted bits feature of [[FLAC]] et al is exploited.


{| class="wikitable" style="text-align:center"
|-
!lossyWAV Test Set
!FLAC -8
!Version
!lossyWAV -1
!lossyWAV -2
!lossyWAV -3
!lossyWAV -4
!lossyWAV -5
!lossyWAV -6
!lossyWAV -7
|-
!10 Album Test Average
| 850kbps
| v0.9.2 RC3
| 479kbps
| 425kbps
| 375kbps
| 352kbps
| 334kbps
| 318kbps
| 306kbps
|-
!53 sample "problem" set               
| 784kbps
| v0.9.2 RC3
| 543kbps
| 494kbps
| 433kbps
| 408kbps
| 385kbps
| 365kbps
| 348kbps
|}
{| class="wikitable" style="text-align:center"
{| class="wikitable" style="text-align:center"
|-
|-
Line 79: Line 44:
|-
|-
!53 sample "problem" set
!53 sample "problem" set
| beta v0.9.8d
| beta 1.0.0
| 784kbps
| 784kbps
| 654kbps
| 654kbps
Line 97: Line 62:
lossyWAV-processed WAV files are named with a double filename extension, .lossy.wav, to make them instantly identifiable. e.g. ".lossy.flac" would indicate an audio file which was processed using lossyWAV, and subsequently encoded using FLAC.[http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=55522&view=findpost&p=498559]
lossyWAV-processed WAV files are named with a double filename extension, .lossy.wav, to make them instantly identifiable. e.g. ".lossy.flac" would indicate an audio file which was processed using lossyWAV, and subsequently encoded using FLAC.[http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=55522&view=findpost&p=498559]


From beta v0.6.1, the -correction parameter is used when processing to create a correction file which is named with the .lwcdf.wav double filename extension. When "added" to the corresponding .lossy.wav, using the -merge parameter, the original file will be reconstituted.
The --correction parameter is used when processing to create a correction file which is named with the .lwcdf.wav double filename extension. When "added" to the corresponding .lossy.wav, using the -merge parameter, the original file will be reconstituted.


Combinations of lossyWAV with each specific encoder are referred to as lossy'''X''', where '''X''' is an abbreviation of the lossless codec name. Combination names are listed in the "[[LossyWAV#Known supported codecs|known supported codecs]]" section below.
Combinations of lossyWAV with each specific encoder are referred to as lossy'''X''', where '''X''' is an abbreviation of the lossless codec name. Combination names are listed in the "[[LossyWAV#Known supported codecs|known supported codecs]]" section below.
Line 187: Line 152:
===Application settings===
===Application settings===
<pre>
<pre>
lossyWAV v0.9.2 RC3, Copyright (C) 2007,2008 Nick Currie.
lossyWAV 1.0.0, Copyright (C) 2007,2008 Nick Currie. Copyleft.
lossyWAV is issued with NO WARRANTY WHATSOEVER and is free software.
This program is free software: you can redistribute it and/or modify it under
 
the terms of the GNU General Public License as published by the Free Software
Usage  : lossyWAV <input wav file> <options>
Foundation, either version 3 of the License, or (at your option) any later
 
version.
Example : lossyWAV musicfile.wav
 
Quality Options:


-1/1a/1b      quality preset 1, circa 480kbps for 44.1khz, 2ch;
This program is distributed in the hope that it will be useful,but WITHOUT ANY
-2/2a/2b/2c    quality preset 2, circa 425kbps for 44.1khz, 2ch [default];
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-3/3a/3b/3c    quality preset 3, circa 375kbps for 44.1khz, 2ch;
PARTICULAR PURPOSE. See the GNU General Public License for more details.
-4/4a/4b/4c    quality preset 4, circa 350kbps for 44.1khz, 2ch;
-5/5a/5b/5c    quality preset 5, circa 335kbps for 44.1khz, 2ch;
-6/6a/6b/6c    quality preset 6, circa 320kbps for 44.1khz, 2ch;
-7/7a/7b/7c    quality preset 7, circa 305kbps for 44.1khz, 2ch;
              [a/b/c = 1/2/3 extra FFT analyses; default=3,2,2,2,2,2,2(max=5)]
 
Standard Options:
 
-o <folder>  destination folder for the output file(s).
-nts <n>      set noise_threshold_shift to n dB (-48.0dB<=n<=+36.0dB);
              (-ve values reduce bits to remove, +ve values increase).
-snr <n>      set minimum average signal to added noise ratio to n dB;
              (0.0dB<=n<=48.0dB) Increasing value reduces bits to remove.
-force        forcibly over-write output file if it exists; default=off.
-check        check if WAV file has already been processed; default=off.
              errorlevel=16 if already processed, 0 if not.
-noclips      set allowable number of clips per channel per codec block to 0;
              default= -1=0,-2=1,-3=2,-4=3,-5=4,-6=5,-7=6.
-correction  write correction file while processing WAV file; default=off.
-merge        merge existing lossy.wav and lwcdf.wav files.
-scale <n>    scaling factor from WaveGain / etc; default = 1.000000.
 
System Options:
 
-quiet        significantly reduce screen output.
-nowarn      suppress lossyWAV warnings.
-detail      enable detailed output mode
 
-below        set process priority to below normal.
-low          set process priority to low.
 
Special thanks:
 
David Robinson for the method itself and motivation to implement it.
Don Cross for the original Pascal source for the FFT algorithm used.
Horst Albrecht for valuable tuning input and feedback.</pre>
===Current beta version settings===
<pre>
lossyWAV beta v0.9.8d, Copyright (C) 2007,2008 Nick Currie.
lossyWAV is issued with NO WARRANTY WHATSOEVER and is free software.


You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.
Usage  : lossyWAV <input wav file> <options>
Usage  : lossyWAV <input wav file> <options>


Line 246: Line 170:
Quality Options:
Quality Options:


-q <n>       quality preset (10 = highest quality, 0 = lowest bitrate;
-q, --quality <n>   quality preset (10=highest quality, 0=lowest bitrate;
              -q 5 is generally accepted to be transparent) default = -q 5.
                    -q 5 is generally accepted to be transparent)
                    default=-q 5.


Standard Options:
Standard Options:


-check       check if WAV file has already been processed; default=off.
-c, --check         check if WAV file has already been processed; default=off.
              errorlevel=16 if already processed, 0 if not.
                    errorlevel=16 if already processed, 0 if not.
-correction   write correction file while processing WAV file; default=off.
-C, --correction   write correction file for processed WAV file; default=off.
-force       forcibly over-write output file if it exists; default=off.
-f, --force         forcibly over-write output file if it exists; default=off.
-help         display help.
-h, --help         display help.
-longhelp     display extended help.
-L, --longhelp     display extended help.
-merge       merge existing lossy.wav and lwcdf.wav files.
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-o <folder>   destination folder for the output file(s).
-N, --noclips      set allowable number of clips / channel / codec block to 0;
                    default=3,3,3,3,2,1,0,0,0,0,0 (-q 0 to -q 10)
-o, --outdir <dir> destination directory for the output file(s).
-v, --version      display the lossyWAV version number.


Special thanks:
Special thanks:


David Robinson for the method itself and motivation to implement it.
David Robinson for the method itself and motivation to implement it.
Don Cross for the original Pascal source for the FFT algorithm used.
Don Cross for the Complex-FFT algorithm used.
Horst Albrecht for valuable tuning input and feedback.</pre>
Horst Albrecht for valuable tuning input and feedback.</pre>
===Example Foobar2000 converter settings===
===Example Foobar2000 converter settings===
Line 271: Line 199:
<pre>
<pre>
@echo off
@echo off
z:\bin\lossyWAV %1 %3 %4 %5 %6 %7 %8 %9 -below -nowarn -quiet
z:\bin\lossyWAV %1 %3 %4 %5 %6 %7 %8 %9 -below -nowarnings -quiet
z:\bin\flac.exe -5 -f -b 512 "%~N1.lossy.wav" -o"%~N2.flac"  
z:\bin\flac.exe -5 -f -b 512 "%~N1.lossy.wav" -o"%~N2.flac"  
del "%~N1.lossy.wav"
del "%~N1.lossy.wav"
Line 297: Line 225:


==External links==
==External links==
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=55522&st=0 Original lossyFLAC thread] Where David Robinson (Replay Gain developer) introduces the method and a MATLAB implementation.
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=63225 lossyWAV 1.0.0 release thread]
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=56129 Current development thread] You will find the latest release candidate and latest beta version in post #1 of this thread.
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=56129 Current development thread] You will find the latest release candidate and latest beta version in post #1 of this thread.
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=55522&st=0 Original lossyFLAC thread] Where David Robinson (Replay Gain developer) introduces the method and a MATLAB implementation.


[[Category:Software]]
[[Category:Software]]
[[Category:Encoder/Decoder]]
[[Category:Encoder/Decoder]]
[[Category:Lossy]]
[[Category:Lossy]]

Revision as of 20:48, 12 May 2008

lossyWAV
{{{logo}}}

Developer(s) Nick.C
Release information
Initial release {{{released}}}
Stable release 1.0.0
Preview release
Compatibility
Operating system Windows
Additional information
Use Digital signal processing
License GNU GPL
Website Hydrogenaudio

lossyWAV is a new free lossy pre-processor for PCM audio contained in the WAV file format. It reduces bit depth of the input signal, which, when used in conjunction with certain lossless codecs, reduces the bitrate of the encoded file significantly compared to unpreprocessed compression. lossyWAV's primary goal is to maintain transparency with a high degree of confidence when processing any audio data.

History

lossyFLAC is an idea started by 2Bdecided at Hydrogenaudio, utilising the wasted bits feature of the FLAC lossless codec with the aim of transparently reducing audio bit depth (making some lower significant bits (LSB's) zero), consequently taking advantage of FLAC's detection of consistently-zeroed lower significant bits within each single frame and significantly increasing coding efficiency.[1] In this way the user can enjoy audio encoded using the same codec (which may be all important from a hardware compatibility perspective) at a reduced bitrate compared to the lossless version.

Nick.C ported the original MATLAB implementation to Delphi (Many thanks CodeGear for Turbo Explorer!!) with a liberal sprinkling of IA-32 and x87 Assembly Language for speed.

Subsequently, lossyFLAC proved itself to work with other lossless codecs, so the application name was changed to lossyWAV.

Since then, Nick.C has heavily developed and built upon lossyWAV, with valuable tuning performed by halb27 at Hydrogenaudio.

Indicative bitrate reduction

It must be stressed that lossyWAV is a pure variable bit rate pre-processor. Bits-to-remove from the audio data are calculated on a block-by-block basis (codec-block length = 512 samples, 11.6msec @ 44.1kHz) using overlapping fast Fourier transform (FFT) analyses of at least two lengths (default quality preset (-q 5) = 64 & 1024 samples). After some manipulation, the results of each FFT analysis for a specific codec-block are then grouped and the minimum value used to determine bits-to-remove for the whole codec-block. Bit removal adds noise to the output, however the added noise has been pre-calculated and its level will be at or below the noise floor of the codec_block in question. Each sample in the codec-block is then rounded such that the first <bits-to-remove> lsb's are zero. In this way the wasted bits feature of FLAC et al is exploited.

lossyWAV Test Set Version FLAC -8 -q 10 -q 9 -q 8 -q 7 -q 6 -q 5 -q 4 -q 3 -q 2 -q 1 -q 0
53 sample "problem" set beta 1.0.0 784kbps 654kbps 626kbps 596kbps 565kbps 534kbps 501kbps 470kbps 447kbps 408kbps 366kbps 329kbps

File identification

lossyWAV-processed WAV files are named with a double filename extension, .lossy.wav, to make them instantly identifiable. e.g. ".lossy.flac" would indicate an audio file which was processed using lossyWAV, and subsequently encoded using FLAC.[2]

The --correction parameter is used when processing to create a correction file which is named with the .lwcdf.wav double filename extension. When "added" to the corresponding .lossy.wav, using the -merge parameter, the original file will be reconstituted.

Combinations of lossyWAV with each specific encoder are referred to as lossyX, where X is an abbreviation of the lossless codec name. Combination names are listed in the "known supported codecs" section below.

From beta v0.5.9, lossyWAV inserts a variable-length FACT chunk into the WAV file immediately after the FMT chunk. This takes the form:

fact/<size>/lossyWAV beta vx.y.z @ dd/mm/yyyy hh:mm:ss, -q 5 -analyses 2 -clips 1

Where the version, date & time and user settings are copied. Additionally, if a lossyWAV FACT chunk is found in a file, the processing will be halted (exit code = 16) to prevent re-processing of an already processed file.

The -check parameter can be used to determine whether a file has previously been processed without trying to process it, exit code = 16 if already processed; exit code = 0 if not.

Quality presets

  • -q 10 to -q 8: Highest quality presets, disc space-saving alternative to lossless archiving for large audio collections.
  • -q 5: Default preset, generally accepted to be transparent.
  • -q 4 to -q 0: DAP quality presets of reducing bitrate with reducing quality preset number, for usage on a compatible DAP. [3]

All tuning has been performed on quality preset -q 5 with higher presets being more conservative. Quality preset -q 5 is generally accepted to be (and from testing so far is) transparent. If you find a track which -q 5 fails to achieve transparency after processing, please post a sample (no more than 30 seconds) in the development thread.

Supported input formats

  • WAV: 9-bit to 32-bit integer; sample rate ≥ 32kHz PCM. Very high sample rates (>48kHz) have not been extensively tested. Tunings have been focussed on 16-bit, 44.1kHz samples (i.e. CD PCM).

Codec compatibility

Codec Supported Encoder parameters Combination name
Apple Lossless No
FLAC Yes -5 -b 512 --keep-foreign-metadata lossyFLAC
LA No
LPAC Yes -b512 lossyLPAC
Monkey's Audio No
MPEG-4 ALS Yes -l -n512 lossyALS
OptimFROG No
TAK Yes -fsl512 lossyTAK
TTA No
WavPack Yes --blocksize=512 lossyWV
WMA Lossless Yes lossyWMALSL

There is also evidence — so-called "Bit Shifting" — to suggest that lossyWAV may work with MLP, but this remains untested due to prohibitive prices of encoders.

A comparison of portable media players is here, which shows FLAC and WMA Lossless compatibility among listed players. Any player supported by Rockbox can use FLAC or WavPack files after installing Rockbox.

Using lossyWAV

Application settings

lossyWAV 1.0.0, Copyright (C) 2007,2008 Nick Currie. Copyleft.
This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.
Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-q, --quality <n>   quality preset (10=highest quality, 0=lowest bitrate;
                    -q 5 is generally accepted to be transparent)
                    default=-q 5.

Standard Options:

-c, --check         check if WAV file has already been processed; default=off.
                    errorlevel=16 if already processed, 0 if not.
-C, --correction    write correction file for processed WAV file; default=off.
-f, --force         forcibly over-write output file if it exists; default=off.
-h, --help          display help.
-L, --longhelp      display extended help.
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-N, --noclips       set allowable number of clips / channel / codec block to 0;
                    default=3,3,3,3,2,1,0,0,0,0,0 (-q 0 to -q 10)
-o, --outdir <dir>  destination directory for the output file(s).
-v, --version       display the lossyWAV version number.

Special thanks:

David Robinson for the method itself and motivation to implement it.
Don Cross for the Complex-FFT algorithm used.
Horst Albrecht for valuable tuning input and feedback.

Example Foobar2000 converter settings

Example flossy3.bat file called from Foobar2000

@echo off
z:\bin\lossyWAV %1 %3 %4 %5 %6 %7 %8 %9 -below -nowarnings -quiet
z:\bin\flac.exe -5 -f -b 512 "%~N1.lossy.wav" -o"%~N2.flac" 
del "%~N1.lossy.wav"

Frequently asked questions

  • Question: Is it VBR?
  • Short answer: Yes.
  • Question: Is it transparent?
  • Short answer: Almost certainly.
  • Question: Is it lossless?
  • Short answer: No.
  • Question: Will it ever have a CBR mode?
  • Short answer: No.
  • Question: Why should I use this?
  • Answer:
  • high quality
  • extremely low chance of audible artefacts
  • reasonable bitrates
  • usable with unmodified, established lossless formats.

External links