lossyWAV 1.1.0 released., Added noise WAV bit reduction method. Options

[Nick.C]

Is that the reason why material with 96 kHz is currently reduced to the same filesize as material with 48 kHz? Are higher sample rates currently treated like if they are played back at a lower speed/rate, thus lowering the frequency spectrum accordingly, which results in a wrong calculation of how many bits can be removed?

Wrong might be a little strong - the sample rate is used correctly where it needs to be but the codec-block-size equates to less time and the granularity of the spreading function will be higher and possibly a bit too coarse.

Does that mean that if noise shaping is disabled less bits are removed? That would reassure me.

Exactly the same (except if noise-shaping causes clips) bits-to-remove values will be achieved for noise shaping 1 and 0.

lossyWAV 1.1.0b attached to post #1 in this thread:

• implementation of increasing fft length for increasing sample rate;
• improved logfile output and --detail output;
• reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.

[Hancoque]

Is it intended that the dithering noise is also shaped? A frequency analysis reveals that it's also mildly shaped. I always thought that rectangular or triangular dither corresponds to white noise.

I also wonder if dithering is really that useful. Look at the graph, it shows AC/DC - Fire Your Guns:



Wouldn't you rather use -q 3 without dithering than -q 2.5 with dithering when the bitrate is (nearly) identical?

[Nick.C]

Is it intended that the dithering noise is also shaped? A frequency analysis reveals that it's also mildly shaped. I always thought that rectangular or triangular dither corresponds to white noise.

Which version of lossyWAV are you using? I ask as the fix to ensure that added noise remains the same when dither is applied has only been introduced at 1.1.0b.

Also, with respect to the triangular dither, I have used a method (discussed on these forums previously) which uses 1 new random number per sample and recycles the previous random number, i.e. tpdf_random = new_random - old_random; (random values in the range +/-0.5). In my implementation and to achieve intermediate dither types I have used: tpdf_random = new_random * dither_type_multiplier - old_random; (dither_type_multiplier in the range 0..1). This may have had an effect on the shape of the dither.

[edit] One other thing: --dither 0 <> no dither. If the dither parameter is used then dither will be applied, somewhere between rectangular and triangular, depending on the parameter value after --dither, i.e. --dither 0 = rectangular dither; --dither 1 = triangular dither; --dither 0.5 = something in between rectangular and triangular dither. [/edit]

This post has been edited by Nick.C: Aug 4 2008, 08:12

[Hancoque]

I'm using the current version 1.1.0b.

I don't know if it's good to reuse old random values. Why not just use two independent random values? Is the performance impact too high? As it's not used by default I don't think that would be much of an issue.

I have redone the above graph and now differentiated correctly between dither off and dither 0. I also included ReplayGain values to illustrate the theoretically perceived loudness of the noise.

[Nick.C]

I'm using the current version 1.1.0b.

I don't know if it's good to reuse old random values. Why not just use two independent random values? Is the performance impact too high? As it's not used by default I don't think that would be much of an issue.

I have redone the above graph and now differentiated correctly between dither off and dither 0. I also included ReplayGain values to illustrate the theoretically perceived loudness of the noise.

I'll find my pest control gear and go bug-hunting - the -s 0 -D 1 curve looks suspiciously like it has been noise-shaped.

There is no real performance hit using two independent random values (especially as I have changed from a fibonacci based RNG to a multiply-with-carry RNG) - I'll have a look at that too.

[2Bdecided]

Of course the dither should/will be noise shaped. It should/will sit next to the quantisation stage within the noise shaping loop.

Current random number minus previous random number is a well accepted way of getting slightly high passed dither.

Hancoque, no one is suggesting that you should use dither. It's there for people who incorrectly think that they need it - not for those who know that they don't (nor for those that don't know or care!).

Cheers,
David.

This post has been edited by 2Bdecided: Aug 4 2008, 13:12

[Nick.C]

Of course the dither should/will be noise shaped. It should/will sit next to the quantisation stage within the noise shaping loop.

Current random number minus previous random number is a well accepted way of getting slightly high passed dither.

Hancoque, no one is suggesting that you should use dither. It's there for people who incorrectly think that they need it - not for those who know that they don't (nor for those that don't know or care!).

Cheers,
David.

1) Dither is added pre noise shaping;
2) Thanks - it will remain so;
3) I'll leave the option to dither up to the user.

Thanks David.

[Hancoque]

I don't plan to use dithering either. I was just curious.

Now that higher sample rates are officially supported, do I understand it right that they require a higher codec block size?

I will modify 1.1.0 to increase the FFT lengths at 69.08kHz, 138.15kHz and 276.3kHz, i.e. 64 to 128 to 256 and 512 samples respectively and correspondingly for the other lengths with a similar increase in codec-block length, 512 to 1024 to 2048 to 4096 samples.

So, does this equate to the following sample rate to codec block size analogy?

44.1/48 kHz -> 512
88.2/96 kHz -> 1024
176.4/192 kHz -> 2048

[Gow]

I started doing some personal listening and file size tests with 1.1.0b and I must say I am very impressed. The idea behind it all was quite something and now we have it implemented with TAK, FLAC and Wavpack. I use TAK on my computers because that is meant for foobar2000 playback and I use mp3 for my DAP because it is compatible and can be mp3gain'd.

So far, I have been using --extreme and I am in love with the results and it looks like I might switch the all lossless library to the lossyTAK library and if my calculation is correct I should expect the lossyTAK library to be 61% of the size of the current library at -p5. So roughly 305gb down to 186gb and have yet to notice anything that could be a problem sample. Of course, I still have the original image files of the CDs ripped using EAC safely tucked away on DVDs so if I do replace the entire library and a problem is discovered it is quite easy to recover from with the DVD, all done to increase disc longevity and to maintain perfect rips.

Good job, keep it up.

On a theoretical note,

This might move the lossless formats even more into the mainstream although the end file is not technically lossless but it so high quality of an output that the problem samples that exist in the lossy formats do not exist here.

I am all for increasing the overall usage of lossless over lossy and with the lossyWAV a new field of battle has been opened up that might sway those who think lossless is TOO big for use to the lossless side.

[2Bdecided]

1) Dither is added pre noise shaping;

But in-loop or out-of-loop? Traditionally it should be in-loop, like this...

http://www.hydrogenaudio.org/forums/index....st&p=580394

Cheers,
David.

[Nick.C]

But in-loop or out-of-loop? Traditionally it should be in-loop, like this...

http://www.hydrogenaudio.org/forums/index....st&p=580394

Cheers,
David.

Sorry, not clear enough - dither is added in the middle of the noise shaping process, just before rounding.

[smok3]

continued, about bat files and tagging, this oneliner seems to be working;

lossyflac.bat

CODE

flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~p1%~n1.lossy.flac --silent && tag --fromfile %1 %~p1%~n1.lossy.flac

requirements: flac, lossywav and tag somewhere on your %path%

example usage: lossyflac file.flac will return file.lossy.flac (with all the tags from file.flac)

p.s. the usual warning about spaces and batch files on win.

edit1: slight fix with paths, added %~p1
edit2: became oneliner

This post has been edited by smok3: Aug 7 2008, 15:10

[Nick.C]

continued, about bat files and tagging, this two liner seems to be working;

lossyflac.bat

CODE

flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~n1.lossy.flac --silent
tag --fromfile %1 %~n1.lossy.flac

...

Very, very elegant - now no need to use foobar2000 for the processing. You could put both lines on one line with a && in between .

[smok3]

You could put both lines on one line with a && in between

any benefits from that? or do we want oneliner?

This post has been edited by smok3: Aug 7 2008, 13:44

[Nick.C]

You could put both lines on one line with a && in between

any benefits from that? or do we want oneliner?

Minimalist, I am a fan of one-line code where it can be done.... The final batch file would be:

CODE

@flac -d %1 --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o %~p1%~n1.lossy.flac --silent && tag --fromfile %1 %~p1%~n1.lossy.flac

[Synthetic Soul]

One minor benefit: As you are using && (as opposed to &) the call to Tag will only be made if the previous command executed successfully.

[smok3]

fixed the post - 'downgraded' twoliner to oneliner http://www.hydrogenaudio.org/forums/index....st&p=581437

This post has been edited by smok3: Aug 7 2008, 15:06

[Hanky]

Nice batch file. But if the specified input file does not exist, this results in a crash of lossywav.exe
Windows XP SP2, lossywav 1.1.0b

CODE

F:\lossywavtest\>flac -d doesnotexist.flac --stdout --silent  | lossy
wav - --stdout -P  | flac - -b 512 -o doesnotexist.lossy.flac --silent

doesnotexist.flac: ERROR initializing decoder
                   init status = FLAC__STREAM_DECODER_INIT_STATUS_ERROR_OPENING_
FlossyWAV 1.1.0b, Copyright (C) 2007,2008 Nick Currie. Copyleft.
This is free software under the GNU GPLv3+ license; There is NO WARRANTY, to
the extent permitted by law. <http://www.gnu.org/licenses/> for details.
ILE

An error occurred opening the input file; it is likely that it does not exist
or is not readable.
ERROR: for encoding a raw file you must specify a value for --endian, --sign, --
channels, --bps, and --sample-rate
Type "flac" for a usage summary or "flac --help" for all options

F:\lossywavtest\>tag --fromfile doesnotexist.flac doesnotexist.lossy.
flac
Tag - Automatic Tag from filename      Copyright (c) 2002-2003 Case
Version 2.0.39b, Compiled 2003-04-11

File not found: 'doesnotexist.lossy.flac'
fromfile: Can't open file 'doesnotexist.flac'.

This post has been edited by Hanky: Aug 7 2008, 15:20

[Nick.C]

Nice batch file. But if the specified input file does not exist, this results in a crash of lossywav.exe

This could probably be dealt with

CODE

@if exist "%1" flac -d "%1" --stdout --silent|lossywav - --stdout -P|flac - -b 512 -o "%~n1.lossy.flac" --silent && tag --fromfile "%1" "%~n1.lossy.flac"

This post has been edited by Nick.C: Aug 7 2008, 15:48

[Hanky]

Thanks Nick.C, that fixed the error

One more small issue (not lossywav related) is the fact that TAG does not copy embedded cover art.

edit:
I found out I was using an old version of TAG (2.0.39b) and updated to 2.0.52, but that version did not copy embedded cover art either.

But this issue should not be discussed here, rather in Changes Made to TAG.EXE, Tags from text file & OptimFrog support

This post has been edited by Hanky: Aug 7 2008, 16:11

[Nick.C]

Thanks Nick.C, that fixed the error

This will process multiple files and place the result in the same directory as the original.

CODE

@echo off
:repeat
if %1.==. goto end
if exist %1 flac -d %1 --stdout --silent|lossywav - --stdout -P --stdinname %1|flac - -b 512 -o "%~dpn1.lossy.flac" --silent && tag --fromfile %1 "%~dpn1.lossy.flac"
shift
goto repeat
:end

[Hanky]

That one works great when you pass one or more filenames, but when I tried with *.flac I got a crash of LossyWav.exe again.
It seems that we need a construction with a FOR (%%I) IN %1 DO loop here to handle wild cards.

This command line does the job:

CODE

FOR %X in (*.flac) DO lossyflac.bat "%X"

edit: quotes added, thanks again Nick.C

This post has been edited by Hanky: Aug 7 2008, 20:48

[smok3]

my usual approach is to do one batch that handles one file (action)

and 2nd batch that loops over the first one, (loopy)

one way or the other, depending on the usage. That way you can keep your action bat short and dumb and leave all the file management logic to the loopy one.

[Nick.C]

That one works great when you pass one or more filenames, but when I tried with *.flac I got a crash of LossyWav.exe again.
It seems that we need a construction with a FOR (%%I) IN %1 DO loop here to handle wild cards.

This command line does the job:

CODE

FOR %X in (*.flac) DO lossyflac.bat %X

But this one skips filenames that include spaces.

If you enclose the second %X in double quotes it will handle the files properly.

This post has been edited by Nick.C: Aug 7 2008, 20:45

[sauvage78]

Quote: Nick.C
"My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind."

Do you have any clue of how long this will take ? 3 months, 6 months or a year ?
Is SebastianG giving you any accelerated private lessons ?
It's not that I want to hurry you or being rude in any way , but I care a lot for lossywav ... it's already my favorite lossy codec ... & I plan to convert tera of lossless to Lossy|Tak -P|-p2e ... (without lossless backup) so I care a lot for this new noise shaping method if it can make me save some kbps (& also for the new special Tak setting for lossywav )

without a 1.2.0 development thread I am asking myself everyday:
1: is it a TODO thing that is already actively worked on in the shadows.
2: is it a TODO thing that is just an idea.
3: are you in vacation with wife & kids.
so I'd rather simply ask ...

as I have been disapointed by vaporware feature from Christopher 'Monty' Montgomery in the past ... I am very suspicious about open source developers claims ... so excuse me if I sound rude I just want to test your determination to get lossywav to the max of its possible efficiency. (I don't want the "new noise shaping method" to become the "bitrate peeling" of lossywav)

I know you just released V1.1.0 a month ago & that I shouldn't already be longing for more ... but now that you are a developer you'll have to learn that end-users are relentless vampires !!! LOL

This post has been edited by sauvage78: Aug 17 2008, 07:47