Difference between revisions of "LossyWAV"

From Hydrogenaudio Knowledgebase
Jump to: navigation, search
m
Line 4: Line 4:
 
| caption =  
 
| caption =  
 
| maintainer = [http://www.hydrogenaudio.org/forums/index.php?showuser=42400 Nick.C]
 
| maintainer = [http://www.hydrogenaudio.org/forums/index.php?showuser=42400 Nick.C]
| stable_release = 1.1.0c
+
| stable_release = 1.2.0
| preview_release = 1.1.4n
+
| preview_release = <none>
 
| operating_system = [[Wikipedia:Microsoft Windows|Windows]]
 
| operating_system = [[Wikipedia:Microsoft Windows|Windows]]
 
| use = [[Wikipedia:Digital signal processing|Digital signal processing]]
 
| use = [[Wikipedia:Digital signal processing|Digital signal processing]]
Line 35: Line 35:
 
!--standard
 
!--standard
 
!--portable
 
!--portable
!-q 0
+
!--zero
 
|-
 
|-
 
!10 Album Test Set
 
!10 Album Test Set
 
| TAK
 
| TAK
 
| 820 kbit/s
 
| 820 kbit/s
| 615 kbit/s
+
| ??? kbit/s
| 532 kbit/s
+
| ??? kbit/s
| 447 kbit/s
+
| ??? kbit/s
| 359 kbit/s
+
| ??? kbit/s
| 266 kbit/s
+
| ??? kbit/s
 
|-
 
|-
 
!10 Album Test Set
 
!10 Album Test Set
 
| FLAC
 
| FLAC
 
| 854 kbit/s
 
| 854 kbit/s
| 632 kbit/s
+
| 627 kbit/s
| 548 kbit/s
+
| 544 kbit/s
| 463 kbit/s
+
| 460 kbit/s
 
| 376 kbit/s
 
| 376 kbit/s
| 285 kbit/s
+
| 288 kbit/s
 
|-
 
|-
 
!10 Album Test Set
 
!10 Album Test Set
 
| Wavpack
 
| Wavpack
 
| 852 kbit/s
 
| 852 kbit/s
| 641 kbit/s
+
| ??? kbit/s
| 563 kbit/s
+
| ??? kbit/s
| 481 kbit/s
+
| ??? kbit/s
| 390 kbit/s
+
| ??? kbit/s
| 296 kbit/s
+
| ??? kbit/s
 
|}
 
|}
  
Line 74: Line 74:
 
lossyWAV inserts a variable-length 'fact' chunk into the WAV file immediately after the 'fmt ' chunk. This takes the form:<pre>fact/<size>/lossyWAV x.y.z @ dd/mm/yyyy hh:mm:ss, -q 5</pre>Where the version, date & time and user settings are copied. Additionally, if a lossyWAV 'fact' chunk is found in a file, the processing will be halted (exit code = 16) to prevent re-processing of an already processed file.
 
lossyWAV inserts a variable-length 'fact' chunk into the WAV file immediately after the 'fmt ' chunk. This takes the form:<pre>fact/<size>/lossyWAV x.y.z @ dd/mm/yyyy hh:mm:ss, -q 5</pre>Where the version, date & time and user settings are copied. Additionally, if a lossyWAV 'fact' chunk is found in a file, the processing will be halted (exit code = 16) to prevent re-processing of an already processed file.
  
The -check parameter can be used to determine whether a file has previously been processed without trying to process it, exit code = 16 if already processed; exit code = 0 if not.
+
The --check parameter can be used to determine whether a file has previously been processed without trying to process it, exit code = 16 if already processed; exit code = 0 if not.
  
 
==Quality presets==
 
==Quality presets==
Line 80: Line 80:
 
*--extreme: (-q 7.5) High quality preset, disc space-saving alternative to lossless archiving for large audio collections, considered to be suitable for transcoding to other lossy codecs;
 
*--extreme: (-q 7.5) High quality preset, disc space-saving alternative to lossless archiving for large audio collections, considered to be suitable for transcoding to other lossy codecs;
 
*--standard: (-q 5) Default preset, generally accepted to be transparent;
 
*--standard: (-q 5) Default preset, generally accepted to be transparent;
*--portable: (-q 2.5) DAP quality preset for use on a compatible [[Wikipedia:Digital audio player|DAP]]. [http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=56129&view=findpost&p=531316]
+
*--portable: (-q 2.5) DAP quality preset for use on a compatible [[Wikipedia:Digital audio player|DAP]].[http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=56129&view=findpost&p=531316]
  
 
All tuning has been performed on quality preset --standard with higher presets being more conservative. Quality preset --standard is generally accepted to be (and from testing so far is) transparent. If you find a track which --standard fails to achieve transparency after processing, please post a sample (no more than 30 seconds) in the development thread.
 
All tuning has been performed on quality preset --standard with higher presets being more conservative. Quality preset --standard is generally accepted to be (and from testing so far is) transparent. If you find a track which --standard fails to achieve transparency after processing, please post a sample (no more than 30 seconds) in the development thread.
 +
 +
The --altpreset parameter was introduced at 1.2.0 which creates a second quality range using modified internal presets and extends the quality range from -4 to 10 (--quality -4 --altpreset is equivalent to --quality 0 --limit 15159 in the default quality quality range).
  
 
==Supported input formats==
 
==Supported input formats==
Line 166: Line 168:
 
===Application settings===
 
===Application settings===
 
<pre>
 
<pre>
lossyWAV 1.1.0c, Copyright (C) 2007,2008 Nick Currie. Copyleft.
+
lossyWAV 1.2.0, Copyright (C) 2007,2008,2009 Nick Currie. Copyleft.
  
 
This program is free software: you can redistribute it and/or modify it under
 
This program is free software: you can redistribute it and/or modify it under
Line 186: Line 188:
 
Quality Options:
 
Quality Options:
  
-I, --insane       highest quality output, suitable for transcoding;
+
-I, --insane         highest quality output, suitable for transcoding;
-E, --extreme       high quality output, also suitable for transcoding;
+
-E, --extreme       high quality output, also suitable for transcoding;
-S, --standard     default quality output, considered to be transparent;
+
-S, --standard       default quality output, considered to be transparent;
-P, --portable     good quality output for DAP use. Not considered to be fully
+
-P, --portable       good quality output for DAP use, not fully transparent.
                    transparent, but considered fit for its intended purpose.
+
-Z, --zero          lowest quality preset, probably contains artifacts.
  
 
Standard Options:
 
Standard Options:
  
-c, --check        check if WAV file has already been processed; default=off.
+
-C, --correction     write correction file for processed WAV file; default=off.
                    errorlevel=16 if already processed, 0 if not.
+
-f, --force         forcibly over-write output file if it exists; default=off.
-C, --correction   write correction file for processed WAV file; default=off.
+
-h, --help           display help.
-f, --force         forcibly over-write output file if it exists; default=off.
+
-L, --longhelp       display extended help.
-h, --help         display help.
+
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-L, --longhelp     display extended help.
+
-o, --outdir <t>     destination directory for the output file(s).
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
+
-v, --version       display the lossyWAV version number.
-o, --outdir <t>   destination directory for the output file(s).
+
-w, --writetolog    create (or add to) lossyWAV.log in the output directory.
-v, --version       display the lossyWAV version number.
+
  
Special thanks:
+
Special thanks go to:
  
David Robinson     for the publication of his lossyFLAC method, guidance, and
+
David Robinson       for the publication of his lossyFLAC method, guidance, and
                    the motivation to implement the method as lossyWAV.
+
                    the motivation to implement his method as lossyWAV.
Horst Albrecht     for ABX testing, valuable support in tuning the internal
+
 
                    presets, constructive criticism and all the feedback.
+
Horst Albrecht       for ABX testing, valuable support in tuning the internal
Sebastian Gesemann for the noise shaping coefficients and help in using them
+
                    presets, constructive criticism and all the feedback.
                    in the lossyWAV noise shaping implementation.
+
 
Don Cross           for the Complex-FFT algorithm used.</pre>
+
Sebastian Gesemann   for the noise shaping coefficients and help in using them
 +
                    in the lossyWAV noise shaping implementation.
 +
 
 +
Matteo Frigo and    for the excellent libfftw3-3.dll contained in the FFTW
 +
Steven G Johnson    distribution (v3.2.1 or v3.2.2).
 +
 
 +
Mark G Beckett      for the Delphi unit that provides an interface to the
 +
(Univ. of Edinburgh) relevant fftw routines in libfftw3-3.dll.
 +
 
 +
Don Cross           for the Complex-FFT algorithm originally used.</pre>
  
 
===Example drag 'n' drop batch file===
 
===Example drag 'n' drop batch file===
Line 223: Line 233:
 
goto repeat
 
goto repeat
 
:end</pre>
 
:end</pre>
 +
 +
===lossyWAV with WINE===
 +
The cause of lossyWAV's WINE incompatibility was found and removed during the development of 1.2.0 and retrospectively amended for 1.1.0b in a maintenance release (1.1.0c).
  
 
===Example [[foobar2000]] converter settings===
 
===Example [[foobar2000]] converter settings===
Line 283: Line 296:
 
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=55522 Original lossyFLAC thread] - Introduction of the concept by David Robinson (Replay Gain developer) and initial development
 
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=55522 Original lossyFLAC thread] - Introduction of the concept by David Robinson (Replay Gain developer) and initial development
 
----
 
----
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=65499 lossyWAV 1.2.0 Development Thread] - Latest release candidate and beta version in the first post
+
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=65499 lossyWAV 1.2.0 development thread]
 +
*[http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=77042 lossyWAV 1.2.0 release thread] - Release of version 1.2.0 on 16 December 2009
 
----
 
----
 
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=63254 lossyWAV 1.1.0 development thread]
 
*[http://www.hydrogenaudio.org/forums/index.php?showtopic=63254 lossyWAV 1.1.0 development thread]

Revision as of 09:30, 21 December 2009

lossyWAV
{{{logo}}}

Developer(s) Nick.C
Release information
Initial release {{{released}}}
Stable release 1.2.0
Preview release <none>
Compatibility
Operating system Windows
Additional information
Use Digital signal processing
License GNU GPL
Website Hydrogenaudio

lossyWAV is a free, lossy pre-processor for PCM audio contained in the WAV file format. Proposed by David Robinson, it reduces bit depth of the input signal, which, when used in conjunction with certain lossless codecs, reduces the bitrate of the encoded file significantly compared to unpreprocessed compression. lossyWAV's primary goal is to maintain transparency with a high degree of confidence when processing any audio data.

History

lossyWAV is based on the lossyFLAC idea proposed by David Robinson at Hydrogenaudio, which is a method of carefully reducing the bitdepth of samples, therefore utilising the wasted bits feature of the FLAC lossless codec. The aim is to transparently reduce audio bit depth (by making some lower significant bits (lsb's) zero), consequently taking advantage of FLAC's detection of consistently-zeroed lower significant bits within each single frame and significantly increasing coding efficiency.[1] In this way the user can enjoy audio encoded using the same codec (which may be all important from a hardware compatibility perspective) at a reduced bitrate compared to the lossless version.

Nick Currie ported the original MATLAB implementation to Delphi (Many thanks CodeGear for Turbo Explorer!!) with a liberal sprinkling of IA-32 and x87 Assembly Language for speed.

Subsequently, lossyFLAC proved itself to work with other lossless codecs, so the application name was changed to lossyWAV.

Since then, Nick has heavily developed and built upon lossyWAV, with valuable tuning performed by Horst Albrecht at Hydrogenaudio. Although the current lossyWAV implementation has built on David's original method, the method itself still very much belongs to its author.

Indicative bitrate reduction

It must be stressed that lossyWAV is a pure variable bit-depth pre-processor in that the overall sample size remains the same after processing but the number of significant bits used for the samples in a codec-block can change on a block-by-block basis. Bits-to-remove from the audio data are calculated on a block-by-block basis (codec-block length = 512 samples, 11.6msec @ 44.1kHz) using overlapping fast Fourier Transform (FFT) analyses of at least two lengths (default quality preset (-q 5) = 32, 64 & 1024 samples). After some manipulation, the results of each FFT analysis for a specific codec-block are then grouped and the minimum value used to determine bits-to-remove for the whole codec-block. Bit removal adds white noise to the output, however the level of the added noise associated with the removal of a number of bits has been pre-calculated and the number of bits to remove will depend on the level of the noise floor of the codec-block in question. Each sample in the codec-block is then rounded such that the first <bits-to-remove> lsb's are zero. In this way the wasted bits feature of FLAC et al. is exploited.

lossyWAV Test Set (16 bit / 44.1kHz) Codec lossless --insane --extreme --standard --portable --zero
10 Album Test Set TAK 820 kbit/s  ??? kbit/s  ??? kbit/s  ??? kbit/s  ??? kbit/s  ??? kbit/s
10 Album Test Set FLAC 854 kbit/s 627 kbit/s 544 kbit/s 460 kbit/s 376 kbit/s 288 kbit/s
10 Album Test Set Wavpack 852 kbit/s  ??? kbit/s  ??? kbit/s  ??? kbit/s  ??? kbit/s  ??? kbit/s

File identification

lossyWAV-processed WAV files are named with a double filename extension, .lossy.wav, to make them instantly identifiable. e.g. ".lossy.flac" would indicate an audio file which was processed using lossyWAV, and subsequently encoded using FLAC.[2]

The --correction parameter is used when processing to create a correction file which is named with the .lwcdf.wav double filename extension. When "added" to the corresponding .lossy.wav, using the --merge parameter, the original file will be reconstituted.

Combinations of lossyWAV with each specific encoder are referred to as lossyX, where X is an abbreviation of the lossless codec name. Combination names are listed in the "known supported codecs" section below.

lossyWAV inserts a variable-length 'fact' chunk into the WAV file immediately after the 'fmt ' chunk. This takes the form:
fact/<size>/lossyWAV x.y.z @ dd/mm/yyyy hh:mm:ss, -q 5
Where the version, date & time and user settings are copied. Additionally, if a lossyWAV 'fact' chunk is found in a file, the processing will be halted (exit code = 16) to prevent re-processing of an already processed file.

The --check parameter can be used to determine whether a file has previously been processed without trying to process it, exit code = 16 if already processed; exit code = 0 if not.

Quality presets

  • --insane: (-q 10) Highest quality preset, generally considered to be excessive;
  • --extreme: (-q 7.5) High quality preset, disc space-saving alternative to lossless archiving for large audio collections, considered to be suitable for transcoding to other lossy codecs;
  • --standard: (-q 5) Default preset, generally accepted to be transparent;
  • --portable: (-q 2.5) DAP quality preset for use on a compatible DAP.[3]

All tuning has been performed on quality preset --standard with higher presets being more conservative. Quality preset --standard is generally accepted to be (and from testing so far is) transparent. If you find a track which --standard fails to achieve transparency after processing, please post a sample (no more than 30 seconds) in the development thread.

The --altpreset parameter was introduced at 1.2.0 which creates a second quality range using modified internal presets and extends the quality range from -4 to 10 (--quality -4 --altpreset is equivalent to --quality 0 --limit 15159 in the default quality quality range).

Supported input formats

  • WAV: 9-bit to 32-bit integer; 1 to 8 channels; sample rate ≥ 32kHz PCM. Very high sample rates (>48kHz) have not been extensively tested. Tunings have been focussed on 16-bit, 44.1kHz samples (i.e. CD PCM).

Codec compatibility

Codec Supported Encoder parameters Combination name
FLAC Yes -5 -b 512 --keep-foreign-metadata lossyFLAC
LPAC Yes -b512 lossyLPAC
MPEG-4 ALS Yes -l -n512 lossyALS
TAK Yes -fsl512 lossyTAK
WavPack Yes --blocksize=512 lossyWV
WMA Lossless Yes lossyWMALSL
Apple Lossless No
LA No
Monkey's Audio No
OptimFROG No
TTA No
  • Combinations of lossyWAV with each specific encoder are referred to as lossyX, where X is an abbreviation of the lossless codec name.


There is also evidence — so-called "Bit Shifting" — to suggest that lossyWAV may work with MLP, but this remains untested due to prohibitive prices of encoders.

A comparison of portable media players is here, which shows FLAC and WMA Lossless compatibility among listed players. Any player supported by Rockbox can use FLAC or WavPack files after installing Rockbox.

Important note

NB: when encoding using a lossless codec, please ensure that the block size of the lossless codec matches that of lossyWAV (default = 512 samples). If this is not done then the lossless encoding of the processed WAV file will (almost certainly) be larger than it would otherwise have been. This is achieved by adding the "Encoder Parameters" in the table above to the command line of the lossless codec in question.

Bonus feature

Another, possibly not obvious, feature of lossyWAV is that the processed output can be "transcoded" from one lossless codec to another lossless codec with absolutely no loss of quality whatsoever. This is solely due to the fact that lossyWAV output is designed to be losslessly encoded - something that lossless codecs do very well indeed.

Using lossyWAV

Application settings

lossyWAV 1.2.0, Copyright (C) 2007,2008,2009 Nick Currie. Copyleft.

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-I, --insane         highest quality output, suitable for transcoding;
-E, --extreme        high quality output, also suitable for transcoding;
-S, --standard       default quality output, considered to be transparent;
-P, --portable       good quality output for DAP use, not fully transparent.
-Z, --zero           lowest quality preset, probably contains artifacts.

Standard Options:

-C, --correction     write correction file for processed WAV file; default=off.
-f, --force          forcibly over-write output file if it exists; default=off.
-h, --help           display help.
-L, --longhelp       display extended help.
-M, --merge          merge existing lossy.wav and lwcdf.wav files.
-o, --outdir <t>     destination directory for the output file(s).
-v, --version        display the lossyWAV version number.
-w, --writetolog     create (or add to) lossyWAV.log in the output directory.

Special thanks go to:

David Robinson       for the publication of his lossyFLAC method, guidance, and
                     the motivation to implement his method as lossyWAV.

Horst Albrecht       for ABX testing, valuable support in tuning the internal
                     presets, constructive criticism and all the feedback.

Sebastian Gesemann   for the noise shaping coefficients and help in using them
                     in the lossyWAV noise shaping implementation.

Matteo Frigo and     for the excellent libfftw3-3.dll contained in the FFTW
Steven G Johnson     distribution (v3.2.1 or v3.2.2).

Mark G Beckett       for the Delphi unit that provides an interface to the
(Univ. of Edinburgh) relevant fftw routines in libfftw3-3.dll.

Don Cross            for the Complex-FFT algorithm originally used.

Example drag 'n' drop batch file

Simply drag the FLAC files onto this batch file and it will process, recode in FLAC and copy ALL of the tags from the input FLAC file, placing the output lossyFLAC file in the same directory as the input FLAC file. Requires flac.exe and tag.exe to be somewhere on the path.

@echo off
:repeat
if %1.==. goto end
if exist %1 flac -d %1 --stdout --silent|lossywav - --stdout --standard --stdinname %1|flac - -b 512 -o "%~dpn1.lossy.flac" --silent && tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end

lossyWAV with WINE

The cause of lossyWAV's WINE incompatibility was found and removed during the development of 1.2.0 and retrospectively amended for 1.1.0b in a maintenance release (1.1.0c).

Example foobar2000 converter settings

lossyFLAC settings:
Encoder: C:\Windows\System32\cmd.exe
Extension : lossy.flac
Parameters: /d /c C:\"Program Files"\bin\lossywav - --standard --silent --stdout|
            C:\"Program Files"\bin\flac - -b 512 -5 -f -o%d
Format is : lossless or hybrid
Highest BPS mode supported: 24 
lossyTAK settings:
Encoder: C:\Windows\System32\cmd.exe
Extension  : lossy.tak
Parameters : /d /c C:\"Program Files"\bin\lossywav - --standard --silent --stdout|
             C:\"Program Files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24
lossyWV settings:
Encoder: C:\Windows\System32\cmd.exe
Extension : lossy.wv
Parameters: /d /c C:\"Program Files"\bin\lossywav - --standard --silent --stdout|
            C:\"Program Files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is : lossless or hybrid
Highest BPS mode supported: 24

There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. C:\"Program Files"\directory_where_executable_is\executable_name

Example EAC settings

See EAC and LossyWAV.

Frequently asked questions

  • Question: Why is the ".wav" file extension used?
  • Answer: The ".wav" file extension is used because lossyWAV is a digital signal processor and not a codec. No decoding is required for any program to play a WAV file which has been processed with lossyWAV as it remains compliant with the RIFF WAVE format.
  • Question: Why create a processor which means that I cannot be sure that a lossless file is truly lossless?
  • Answer: Unless one creates the lossless file personally, one can never be completely sure that the file is indeed lossless. e.g. If a WAV file is encoded to MP3 and then transcoded to a lossless codec, how can this pre-processing be easily determined?
  • Question: Is it VBR?
  • Short answer: Yes.
  • Question: Do I need to re-process to change lossless codecs?
  • Short answer: No.
  • Question: Is it transparent?
  • Short answer: At preset --standard, almost certainly.
  • Question: Is it lossless?
  • Short answer: No.
  • Question: Will it ever have a CBR mode?
  • Short answer: No.
  • Question: Why should I use this?
  • Answer:
  • high quality
  • extremely low chance of audible artifacts
  • reasonable bitrates
  • usable with unmodified, established lossless formats.

External links