lossyWAV 1.1.0 released., Added noise WAV bit reduction method. Options

[carpman]

linking --extreme & --insane to Q6 & Q7.5 just because you consider it 100% transparent is pure random ... personnaly I consider Q2.5 100% transparent until someone prove the contrary by providing an ABXable problem sample. So far Q2.5 is the "real life" transparency point of lossywav & Q5 is the theoric optimal setting for transparency of lossywav ... anything above Q5 is overkill, only maybe usefull for people willing to use a transform codec later.

Q10 --insane is a useless preset IMHO, but if Nick wants it, amen ... I will not ask him to remove it

you could randomly link --insane & --extreme to any setting above Q5 ... because it is overkill anyway ... using a 2.5 step is just a "little" less random ...

As I said: "Suggestion for later down the line, when LossyWAV has been more exposed and more thoroughly tested".

Everything you said is correct NOW, we'll have to wait and see if it's still correct a year from now. Until thousands of people are using it and testing it with all their myriad forms of music, we cannot be as certain as your post suggests you are.

C.

[EDIT: typo]

This post has been edited by carpman: Aug 19 2008, 16:12

[Nick.C]

All this talk of noise shaping, psy=modelling and spreading functions got me to thinking about something....

From the earliest stages of variable spreading lengths per frequency band, the "spread" result has been calculated as follows for different spreading lengths (spl) [fft_result = array of skewed magnitudes of the fft analysis]:

spl=1; result:=fft_result[a];
spl=2; result:=(fft_result[a]+fft_result[a+1])/2;
spl=3; result:=(fft_result[a]+fft_result[a+1]+fft_result[a+2])/3;
etc.

However for the even cases, maybe the following should apply:

spl=1; result:=fft_result[a];
spl=2; result:=(fft_result[a-1]/2+fft_result[a]+fft_result[a+1]/2)/2;
spl=3; result:=(fft_result[a-1]+fft_result[a]+fft_result[a+1])/3;
spl=4; result:=(fft_result[a-2]/2+fft_result[a-1]+fft_result[a]+fft_result[a+1]+fft_result[a+2]/2)/4;
etc.

This seems to have the advantage of taking the adjacent results into account for both odd and even spreading lengths rather than just for [edit] odd even [/edit] spreading lengths.

This post has been edited by Nick.C: Aug 20 2008, 07:51

[halb27]

... anything above Q5 is overkill, only maybe usefull for people willing to use a transform codec later ...

I think very high quality archiving as an alternative for a lossless archive makes the availability of --extreme (-q 7.5) and --insane (-q 10) desirable though I personally don't use --insane, and I use --extreme only for very precious tracks.
As carpman said we can't be absolutely sure about --standard's transparency so for archving quality the quality headroom of presumed overkill settings is welcome.
Everybody can use the quality settings according to his needs, and IMO for everybody there is a quality setting which makes him happy.

As for the correction file: I personally don't use it as a correction file and don't plan to use it this way. But I like the ability to be able to listen to the error signal. Once I did when using noise shaping improved my confidence in the usefulness of current noise shaping a lot.

This post has been edited by halb27: Aug 19 2008, 22:11

[carpman]

Precisely.

I'm using -q 6 because my lossyTAK files are a replacement for my lossless archive. I felt it sensible to have a little headroom over --standard as they'll be used for transcoding to MP3, and presently it's too early to tell whether lossyWAV is transparent from 2.5 upwards. Though it looks pretty good right now.

It just struck me that you probably would have to actually be insane to use --insane.

C.

[GeSomeone]

The opinions about the quality presets will be endless, but not important.
The scale to use is -q (and then there are some presets to give the general idea). As I understand it, the -q scale is meant cover all the bit rates lossyWav can be used for, without limiting to practical use only. So a gradual scale from almost lossless bit rates to the point where lossyWav becomes audibly non-transparent.
There is for everybody a choice that's right. You want minimum or maximum overkill/headroom? very safe or on the edge? It's there.

I think changing the quality scale causes more confusion than it's worth.

[Nick.C]

I think changing the quality scale causes more confusion than it's worth.

I agree - there are users with pretty much diametrically opposed views, e.g. some wanting lowest bitrate with acceptable quality and some wanting to see extremely high quality / bitrate (i.e. even closer to lossless).

As an aside, I am looking further into the spreading-function. Up to 1.1.0 it has always averaged an integer number of FFT_result bins to produce its output. Thinking through the modifications necessary to take partial results at the edges for even spreading lengths (i.e. still centred on a particular bin rather than centred between two bins) it has occurred to me that this could be extended to take any proportion of the outlying bins as one possible method. I am still working on this, but it will probably be included in the first beta 1.12.1.

This post has been edited by Nick.C: Aug 21 2008, 20:47

[sauvage78]

just for my own fun, & (to be honest) to convince myself that lossywav was the right way to go for my own use, I have made a .txt throwing ideas of why I should or shouldn't use lossywav as my main codec ... I just copy past it here so that anyone can improve/disagree with my opinion

Pro:
- Transparent at ~370Kbps (so far)
- ~50% space gain on default setting compared to wav (More if you push the codec to its limits)
- Editable (Split/Join Losslessly, most other lossy codecs split losslessly but cannot be rejoined losslessly) ==> Usefull for Lossy CDImage+cue
- 100% Native Gapless (No tagging tricks like mpeg codecs, no matter if the audio player can read the tag info, it IS gapless by nature)
- No Agressive Lowpass (Don't know if generous flat lowpass ... say 20Khz should really be considered evil as it would maybe save some Kbps, test (& new soundcard) needed)
- No Transform (No DCT, No Subband) ==> free of artefacts linked to transform codecs (Pre-echo ...)
- Future-safe: Usable with several of the best lossless codecs (Flac/Tak) (Lossywav improves as all supported lossless codecs improve so if any of these codec die, lossywav doesn't die)
- Best solution at overkill bitrate ~320Kbps
- Can (most likely) be trancoded with a transform codec once at --standard & above without audible loss.
(Lossywav artefacts being of a different nature than transform codec artefacts both (if not audible) can (most likely) be added while still achieving transparency)

Anti:
- Added noise compared to lossless (Immanent to any lossy codec)
- Young Codec: Transparency point (actually -- portable IMHO) is likely to change as more problem samples arise
[One could argue that this is a problem for all lossy codec Transparency point move at each new release, hopefully it will move downward but it could move upward until it becomes mature]
- Not optimal with Wavpack
- Can't compete & will (most likely) never compete at highbitrate ~256Kbps
- Can't compete & will never compete at midbitrate ~128Kbps
- Can't compete & will never compete at lowbitrate ~064Kbps

Note: Many of the advantages of lossywav (Editable/Gapless ...) come from the fact that it is in fact not a codec but a pre-filter for wav ==> NO decoder=more freedom

Edit1: Includes some halb27 comments

This post has been edited by sauvage78: Aug 24 2008, 17:06

[halb27]

Pro: ...
Anti ...

That's pretty perfect IMO.

I'd just add to the pro side that lossyWAV is future-safe in the sense that the lossless codec can be exchanged at any time whenever this is appropriate to the user. Exchanging the lossless codec is a lossless process (decode the lossless codec and reencode with the new lossless codec).

It's correct that it's a disadvantage that lossyWAV adds noise. But we should keep in mind that this is relevant only when comparing with lossless codecs. Adding noise is immanent to any lossy codec, and the essential question there is about the character of the added noise ('can it be kept kept inaudibel in a robust way?' on one end of the scale, or: 'can it be very annoying?' on the other end).
I personally don't think that it's likely that the transparency point will change in the future, though of course this can happen. In this problem area I'd rather focus on the fact that experience on transparency is restricted at the moment. I wouldn't care much about whether a problem will come up which will push the current transparency 'border' from -q 2.5 to -q 3 or -q 3.5. I'd rather see it this way: I expect from --portable great quality and wouldn't care about very subtle problems if they would come up. But I expect from --standard transparency. It would be a major issue if --standard should prove not to be transparent. And the restricted experience so far is a negative point. Though I don't consider it likely that --standard will be proven not to be transparent.

This post has been edited by halb27: Aug 24 2008, 16:34

[smok3]

the only questions (for me) is how it will behave in editing/mixing enviroment together with bunch of eq's and other DSPs compared to say high-bitrate mp3?

[jido]

One caveat is that some of the things you could do would be quite pointless because they would dramatically reduce the efficiency of the subsequent lossless coding.

Cheers,
David.

And that is what we expect from LossyWAV, efficient lossless compression for a minimal sacrifice in quality. If you introduce new algorithms they should not reduce subsequent lossless compression. In fact I believe new algorithms should specifically aim at producing sound waves that are known to compress well, so that you can keep constant quality for a better compression.

Keep up the good work!

[2Bdecided]

the only questions (for me) is how it will behave in editing/mixing enviroment together with bunch of eq's and other DSPs compared to say high-bitrate mp3?

I would guess the answer is "far better".

Cheers,
David.

[Dynamic]

the only questions (for me) is how it will behave in editing/mixing enviroment together with bunch of eq's and other DSPs compared to say high-bitrate mp3?

I would guess the answer is "far better".

I'd have to agree with David.

Traditional lossy encoders with a full psychoacoustic model try a number of tricks which can be problematic with various EQ, DSP or effects.

Temporal masking. A sudden loud sound can mask quieter sounds for a time period afterwards, with a masking level that decays for a time (post-masking) and masking sound for a much shorter period prior to the loud sound (pre-masking). This can be exploited by allowing added noise (e.g. coarser quantization) below the temporal masking threshold. Tempo adjustment to slow the tempo could unmask sounds and result in pre-echo or post-echo noise artifacts. LossyWAV doesn't exploit temporal masking, so this shouldn't be a problem. Neither should any temporal smearing occur with LossyWAV. Even high bitrate MP3 (e.g. LAME -V0 generally exploits these same effects, but uses the extra bits (compared to say -V2) to increase the margin of safety between the added noise and the masking threshold, and uses some extra bits to encode higher frequencies (increased lowpass filter) that are unlikely to be audible.

Tonal masking. A loud tone at one frequency can mask quieter simultaneous tones at other frequencies. This can be exploited by allowing added noise (e.g. coarser quantization) at adjacent frequencies according to the frequency-dependent masking threshold. While encoders usually make some allowances for frequency response of the playback equipment and room acoustics to vary from the ideal, the use of an extreme EQ setting, filter or tuning/pitch-shifting DSPs (or indeed a tweeter separated by some distance from the midrange speaker) could cause unmasking of certain distortions, which may sound very artificial and unmusical. LossyWAV does not exploit tonal masking at all, nor does it apply a lowpass filter to the audio. Again, high bitrate MP3 like LAME -V0 exploits the masking but sets a greater margin of safety between the calculated masking threshold and the permitted added noise.

The only frequency-dependency in lossyWAV is that the analysis by default ignores frequencies above 16 kHz when determining the noise floor (though this can be over-ridden). These frequencies are typically inaudible in music. For some high-fidelity music, content above 16 kHz is likely to have a low noise floor (e.g. if filtered or beyond the frequency response of equipment used) thereby reducing the bit-depth reduction that lossyWAV could achieve if it didn't ignore the >16 kHz region. There is potential for frequencies >16kHz to be brought into the audible range by pitch-shifting DSPs or to be enhanced dramatically by extreme EQ. The chances are that the noise floor determined with the original 16kHz analysis limit will be sufficient to achieve transparency, and if it's a high noise floor, the masking threshold into the >16kHz band is likely to be pretty high too so even if the noise floor is raised, it's still likely to remain transparent in most situations and to be far better than high bitrate conventional lossy encoding.

Lossy stereo. For some lower-bitrate approaches that aren't aiming for transparency, the stereo image is deliberately degraded to save bits in a less annoying way that other approaches. This generally doesn't apply to high-bitrate lossy, where lossless stereo is usually employed. LossyWAV makes no adjustment to the stereo image. However, lossyWAV now measures the noise floor in each channel separately, so it can independently adjust the bit-depth of the two channels rather than locking them together.

[Dynamic]

Another thing that occurs to me, that I don't recall any mention of is that if one uses lossyWAV as one's "lossless archive" from which to listen and transcode, any HDCDs will lose the subcarrier in the LSB that can be used for control of headroom extension (which I believe is audible) and one or two other things (which are possibly inaudible).

If archiving blindly (which is "safe" with pure lossless), it would be good if some software would detect HDCD and possibly pre-emphasis (info in CUEsheet) and optionally pause and issue a warning so that the user could process the PCM appropriately before running lossyWAV on the resulting WAV. E.g. Use SoX for de-emphasis into 24-bit WAV, for example, or using an HDCD decoder (again, ideally to 24-bit WAV) to achieve the "correct" sound.

I don't know where such functions would be best placed, though it may be possible to script something with REACT to use appropriate helper programs (I think there was an hdcd commandline prog in a thread some time ago).

This post has been edited by Dynamic: Aug 26 2008, 21:25

[cBc]

hello!

I recently came across a mention of lossyWAV , read thru the wiki page & here, and was able to set up EAC & ecode the files using it (50% smaller than vanilla FLACS - cool!)
However, I am now trying to use the batchfile mentioned on this post

http://www.hydrogenaudio.org/forums/index....st&p=581571

to drop flac files onto (I have latest TAG.exe, lossywav.exe, and flac.exe , along w/ the batchfile.bat in one folder, but dropping files onto the batchfile results in a quick blip of a dos window viewing then nothing.
I'm pretty DOS ignorant, can anyone advise what I need to do to get it working.
To create the batchfile I copied the text into a notepad page & renamed it as a .bat extension

Many thanks!

[Big_Berny]

There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. c:\"program files"\directory_where_executable_is\executable_name

IMHO it's simpler to enclose the whole path in quotation marks and it should work too. This way you can just use it always - also if there's no space in the path:
"<Path>"
"c:\program files\directory_where_executable_is\executable_name"

Just my two cents.

[cBc]

hello!

I recently came across a mention of lossyWAV , read thru the wiki page & here, and was able to set up EAC & ecode the files using it (50% smaller than vanilla FLACS - cool!)
However, I am now trying to use the batchfile mentioned on this post

http://www.hydrogenaudio.org/forums/index....st&p=581571

to drop flac files onto (I have latest TAG.exe, lossywav.exe, and flac.exe , along w/ the batchfile.bat in one folder, but dropping files onto the batchfile results in a quick blip of a dos window viewing then nothing.
I'm pretty DOS ignorant, can anyone advise what I need to do to get it working.
To create the batchfile I copied the text into a notepad page & renamed it as a .bat extension

Many thanks!

anyone?

[2Bdecided]

cBc,

Maybe no one knows the answer, or maybe it's in the more recent thread (this one is very old)...

http://www.hydrogenaudio.org/forums/index....5499&st=100

Sorry I can't help.

Cheers,
David.

[Dynamic]

cBc, you might be able to debug the Batch File by editing it (e.g. in Notepad) and inserting a PAUSE command near the end, which will ask you to press a key to continue, preventing Windows from clearing the window on completion.

[cBc]

cBc, you might be able to debug the Batch File by editing it (e.g. in Notepad) and inserting a PAUSE command near the end, which will ask you to press a key to continue, preventing Windows from clearing the window on completion.

Could it be that I don't have the required .exe files in the proper place? Currently they are all in one folder, along with the batchfile.

One of the posts earlier mention having them "in the path", and again my lack of DOS knowledge is hobbling me here.

Thanks

[Nick.C]

The "path" is a list of directories that DOS will look into, as well as the current directory, when searching for a COM/EXE/BAT file to execute before giving a "not found" error.

The batch file assumes that the executables are in a directory pointed to by the path variable list.

If your flac.exe, tag.exe and lossyWAV.exe and batch file were, for example, in C:\BIN then:

CODE

@echo off
:repeat
if %1.==. goto end
if exist %1 c:\bin\flac -d %1 --stdout --silent|c:\bin\lossywav - --stdout -P --stdinname %1|c:\bin\flac - -b 512 -o "%~dpn1.lossy.flac" --silent && c:\bin\tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end

should work....

This post has been edited by Nick.C: Jan 18 2009, 18:03

[cBc]

The "path" is a list of directories that DOS will look into, as well as the current directory, when searching for a COM/EXE/BAT file to execute before giving a "not found" error.

The batch file assumes that the executables are in a directory pointed to by the path variable list.

If your flac.exe, tag.exe and lossyWAV.exe and batch file were, for example, in C:\BIN then:

CODE

@echo off
:repeat
if %1.==. goto end
if exist %1 c:\bin\flac -d %1 --stdout --silent|c:\bin\lossywav - --stdout -P --stdinname %1|c:\bin\flac - -b 512 -o "%~dpn1.lossy.flac" --silent && c:\bin\tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end

should work....

Nick,

many thanks...worked a treat!! Completely missed the directory callouts in the string.

[Nick.C]

. Should anyone wish to comment / ask questions / etc. please do it at the end of this thread.

[Nick.C]

lossyWAV 1.1.0c attached to post #1 in this thread. This is a maintenance release as the cause of the WINE incompatibility has been determined.