lossyWAV 1.1.0 Development Thread., Added noise WAV bit reduction method. Options

[Nick.C]

Check the first post.

Change log 1.0.1q: 30/05/08

Maybe you should attach list of programs wich supporting lossywav?

p.S. GXTranscoder

The list of known supported codecs is in the wiki article linked from the first page in the thread.

[Nick.C]

lossyWAV beta 1.0.1s attached to post #1 in this thread.

[edit] The speed of transcoding in foobar2000 has been dramatically increased by the use of STDIN and STDOUT. I transcoded my Mike Oldfield collection (366 tracks, 33h29m) in 22m18s or approximately 90.09x overall {Intel E6600 C2D @ 3.0GHz, 4GB DDR2-800, Windows MCE2005 SP2, source on server, destination 160GB Maxtor SATA HDD}. This was definitely limited by the processor cores as the utilisation plot for the duration of the transcode was a virtual flat-line at 100%.

What this does mean though is that the 'fact' chunk will not be preserved - so be careful not to mix-up your archive FLAC files with your lossyFLAC files. [/edit]

This post has been edited by Nick.C: Jun 9 2008, 19:59

[GeSomeone]

I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size.

Thanks Nick,

Something has changed with STDIN but it seems still not quite right.

When I feed the intermediate file lossywav.lossy.wav to STDIN of FLAC (without --keep.. ) says
-: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
-: WARNING: unexpected EOF; expected 5869416 samples, got 5868544 samples
BTW according to foobar2000 the number of samples of the source is also 5869416 (and the lossywav and resulting lossy.flac are 5869056) I'm not sure foobar displays correct values.

When I let FLAC read the lossy file (with --keep.. )
lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

Is that FACT chunk becoming a nuisance now, or is it something else?

This post has been edited by GeSomeone: Jun 10 2008, 17:44

[Nick.C]

I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size.

Thanks Nick,

Something has changed with STDIN but it seems still not quite right.

When I feed the intermediate file lossywav.lossy.wav to STDIN of FLAC (without --keep.. ) says
-: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
-: WARNING: unexpected EOF; expected 5869416 samples, got 5868544 samples
BTW according to foobar2000 the number of samples of the source is also 5869416 (and the lossywav and resulting lossy.flac are 5869056) I'm not sure foobar displays correct values.

When I let FLAC read the lossy file (with --keep.. )
lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

Is that FACT chunk becoming a nuisance now, or is it something else?

Could you please post your converter settings so that I can duplicate the problem?

Many thanks.

[edit] Problem found here too - average of 249 samples dropped off the end of 85 album lossyFLAC files. It may be something to do with the last read (incomplete codec-block [probably]) - I'll get onto it and revert soon. Until such times, STDIN input is not recommended to be used (--stdout seems fine....) [/edit]

This post has been edited by Nick.C: Jun 10 2008, 20:44

[Nick.C]

[edit] Problem found here too - average of 249 samples dropped off the end of 85 album lossyFLAC files. It may be something to do with the last read (incomplete codec-block [probably]) - I'll get onto it and revert soon. Until such times, STDIN input is not recommended to be used (--stdout seems fine....) [/edit]

Many thanks for the heads up GeSomeone - it was as I surmised and it now seems to work properly in beta 1.0.1t.

lossyWAV beta 1.0.1t attached to post #1 in this thread.

This post has been edited by Nick.C: Jun 11 2008, 20:15

[halb27]

lossyWAV beta 1.0.1t attached to post #1 in this thread.

Thank you, Nick, for the new version.
I encoded my regular music test set with piping and with my bat file.
Speed is great when using piping (though it is good with the bat file).
I tested the musical content just to make sure, and with all the tracks content is identical no matter which encoding way.

Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).

This post has been edited by halb27: Jun 11 2008, 22:02

[Nick.C]

lossyWAV beta 1.0.1t attached to post #1 in this thread.

Thank you, Nick, for the new version.
I encoded my regular music test set with piping and with my bat file.
Speed is great when using piping (though it is good with the bat file).
I tested the musical content just to make sure, and with all the tracks content is identical no matter which encoding way.

Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).

I would guess that FLAC is inserting a larger number of seek points due to the fact that (in foobar2000) the assumed size of the piped file is maximum, i.e. 4,294,967,294 bytes.

Processing on my laptop, using two intermediate WAV files, used to be a real HDD thrashing exercise - 40 to 50 minutes for my 10 album test set (3.36GB, 9h22m57s total duration). Using piped input and output, the transcode now takes approximately 10 minutes - a really nice reduction in total processing time.

I've just finished a "stress test" of sorts: all of my FLAC'ed CD's on my server to lossyFLAC (-q 1.25). 12d4h24m45s (4064 tracks, 316 CD's, 108GB, 887kbit/s average) transcoded in 3h33m23s (40.5GB, 331 kbit/s average) - 82.22x overall and exactly the same number of samples out as went in . Bearing in mind that previously my 3686 track 100GB(ish) transcode took approximately 7h30m, that's an appreciable saving.

If there are no reported problems and if no-one has any further suggestions then now may be the time to move towards a release candidate for 1.0.2.

This post has been edited by Nick.C: Jun 11 2008, 22:33

[GeSomeone]

lossyWAV beta 1.0.1t attached to post #1 in this thread.

Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).

Yes thanks a lot, this time my little script works as intended

Except from the metadata it could be the larger seektable that FLAC allocates for a file of unknown length, as foobar2000's converter generates.

I'll show my script (batfile) that (with this fix)

• handles either - or %s as input.
• one intermediate file less than the "old" script
• the FLAC files are a few kB smaller because no large seektables and 2kB padding (more than enough for me) instead of 8kB.
• Foreign metadata is added to the FLAC file, though I doubt lossyWav will ever see it again when converting from STDIN (with foobar).

Cons

• can run only 1 converter thread at once
• takes less advantage of multi core/hyperthreading than piping directly from lossyWav into FLAC

foobar2000:
Encoder: C:\WINDOWS\system32\cmd.exe
Extension: lossy.flac
Parameters: /D /C C:\PROGRA~1\FLAC\lossyFlac.bat - %d -q 4.5 (or any lossyWav settings)

and my lossywav.bat

CODE

@echo off
set lossyWAV_path="C:\Progra~1\FLAC\lossyWAV.exe"
set FLAC_path="C:\Progra~1\FLAC\flac"
set Mylog="C:\Progra~1\FLAC\lossyFlac.log"

echo # lossyWAV.exe "%~N2.flac" %3 %4 %5 %6 %7 %8 --quiet >> %Mylog%
%lossyWAV_path% %1 %3 %4 %5 %6 %7 %8 --quiet --nowarnings --low --writetolog
type lossyWav.log >> %MyLog%
del lossyWav.log

set MyLossyFile="%~N1.lossy.wav"
if NOT %1 == - goto :FLAC
set MyLossyFile="lossywav.lossy.wav"

:FLAC
%FLAC_path% -5 -e -s -b 512 --force --padding=2048 --keep-foreign-metadata %MyLossyFile% -o%2 >>%Mylog%

del %MyLossyFile%

set MyLossyFile=
set Mylog=
set lossyWAV_path=
set flac_path=

Of course that logging stuff can be removed and on Vista it should be in another location.

[edit] some tidying up and removing a debug line from the script [/edit]

This post has been edited by GeSomeone: Jun 11 2008, 23:21

[halb27]

I would guess that FLAC is inserting a larger number of seek points ...

Yes, it's the seektable. When using --no-seektable with FLAC, I arrive at an expected file size.

Is the seektable necessary? I had no problems skipping part of the track (using foobar) with the --no-seektable version.

What kind of problems can I encounter when using --no-seektable?

This post has been edited by halb27: Jun 12 2008, 18:50

[Nick.C]

What kind of problems can I encounter when using --no-seektable?

Did Josh not explain this once - I think that it only takes a bit longer to seek when moving about the file.

[halb27]

I don't encounter any disadvantage using --no-seektable when playing on my DAP.

So I'll use --no-seektable together with GeSomeone's --padding=2048 proposal and enjoy piping.

...If there are no reported problems and if no-one has any further suggestions then now may be the time to move towards a release candidate for 1.0.2.

I second this.

[sidewalking]

I don't know if anyone uses REACT to do their EAC tasks, but I have a rough (but working) model of hacking the track-config file to use lossyWAV/lossFlac.

CODE

IF NOT @lossyFlac@==1 GOTO end_lossyFlac_tracks
    IF NOT EXIST %TrackDir_lossyFlac% MKDIR %TrackDir_lossyFlac%
    PUSHD %TrackDir_lossyFlac%
        IF @various@==1 SET VA_tag=-T "album artist=@VA@"
        IF %embed_cover%==1 SET Cover_tag=--picture="|image/jpeg|||@cover@"
        ECHO ON
    @tools@\lossywav.exe "@source@" @Opt_lossyWAV@
    RENAME *lossy.wav "temp.lossy.wav"
    DEL "@source@"
        @tools@\flac.exe @Opt_lossyFlac@ %Cover_tag% %VA_tag% %Disc_Flac% -T artist="@artist@" -T album="@album@" -T tracknumber="@track@" -T title="@title@" -T date="@year@" -T genre="@genre@" -T comment="@comment@" -T encoding="Flac @Ver_Flac@ @Opt_lossyFlac@" "temp.lossy.wav" -o "%TrackName%.lossy.flac"
    DEL "temp.lossy.wav"
        @ECHO OFF
        IF %have_cover%==1 IF NOT EXIST folder.jpg COPY "@cover@" folder.jpg
    POPD
:end_lossyFlac_tracks

(and these were in the REACT.ini:)

CODE

Opt_lossyFlac=-5 -b 512 --keep-foreign-metadata
Opt_lossyWAV=-q 5

Not groundbreaking, I guess, but a start. I don't know how to use the temp variables in batch files, so I use a static temp name and then delete it. But, it works fine. I get resulting lossyFlac files with proper tagging, album art embedded, and Replay Gain tagging.

Scott

[collector]

What kind of problems can I encounter when using --no-seektable?

Did Josh not explain this once - I think that it only takes a bit longer to seek when moving about the file.

At first, when ripping to images, I added seekpoints for every 90 seconds. Now I use -S- and noticed that the cue points from the embedded cue sheet are sufficient. And because I can use the cuesheet to play the disc image I can jump to any track or movement I like I like.

[sauvage78]

hi, today I played with lossywav again in order to have it working for good with F2K & tak, so I copy/pasted your command line from post N°1, & it worked like a charm with flac

/d /c c:\data_nic\bin\lossywav - --standard --silent --stdout|c:\data_nic\bin\flac - -b 512 -5 -f -o%d

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

But my knowledge being very limited I don't undertand the use of /d /c ...
I understand that I call cmd.exe with c:\windows\system32\cmd.exe then lossywav with c:\z\lossywav - --portable --silent --stdout| then tak with c:\z\Takc -e -p3 -fsl512 -ihs - %d, but I don't get the use of /d /c, but if I delete it it doesn't work anymore so could someone explain the use of /d /c ? What is it needed for ? Also is there any mistake in my own lossyTak command line ? Thks

This post has been edited by sauvage78: Jun 14 2008, 14:28

[Nick.C]

hi, today I played with lossywav again in order to have it working for good with F2K & tak, so I copy/pasted your command line from post N°1, & it worked like a charm with flac

/d /c c:\data_nic\bin\lossywav - --standard --silent --stdout|c:\data_nic\bin\flac - -b 512 -5 -f -o%d

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

But my knowledge being very limited I don't undertand the use of /d /c ...
I understand that I call cmd.exe with c:\windows\system32\cmd.exe then lossywav with c:\z\lossywav - --portable --silent --stdout| then tak with c:\z\Takc -e -p3 -fsl512 -ihs - %d, but I don't get the use of /d /c, but if I delete it it doesn't work anymore so could someone explain the use of /d /c ? What is it needed for ? Also is there any mistake in my own lossyTak command line ? Thks

CODE

C:\Documents and Settings\User>cmd /?
Starts a new instance of the Windows XP command interpreter....

/C      Carries out the command specified by string and then terminates
/D      Disable execution of AutoRun commands from registry (see below)

[sauvage78]

thks for cmd /? ... didn't knew about it

[Nick.C]

thks for cmd /? ... didn't knew about it

No problem - I'm glad that the command line is working properly for you. Thanks for the Tak command line - I'll include it in the first post.

From what collector and halb27 have said, maybe --no-seektable -P=4096 would be a sensible way of reducing the size of the seektable when using STDIN / STDOUT in foobar2000. I processed my 10 Album test set and I think this saved about 1MB (in 1.37GB processed). Still, 1MB is 1MB....

[TBeck]

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

Also is there any mistake in my own lossyTak command line ? Thks

Looks good

In my evaluations of LossyWav processed TAK test file sets there was no advantage of choosing a preset higher than -p2m. Some file sets even compressed about 0.05 percent worse with -p3 to -p5...

Thomas

[TBeck]

edit: Content of duplicate post deleted...

This post has been edited by TBeck: Jun 14 2008, 17:59

[sauvage78]

Nick.C:
... well I haven't yet read all the discussion of collector and halb27 on seektable ... I only come to read this thread once a week/month to see how things go on ... I know I will not use lossywav for production as long as it's not stable ... so I am not in a hurry to read it all

TBeck:
Thks for the hint ... I was planning to test the efficiency of flac -q6 vs tak -p3 vs wv -hm on lossywav processed files as I think I read somewhere not all lossless codecs where equal when encoding from lossywav ... I think I read wavpack wasn't optimal compared to tak & flac ... will add tak at different settings when I will test ... post lossywav V1.2 ... with some luck maybe by then I will have a frozen tak too

[Nick.C]

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

Also is there any mistake in my own lossyTak command line ? Thks

Looks good

In my evaluations of LossyWav processed TAK test file sets there was no advantage of choosing a preset higher than -p2m. Some file sets even compressed about 0.05 percent worse with -p3 to -p5...

Thomas

Thanks for the clarification, Thomas.

[sauvage78]

Just tested with wavpack 4.5 & it worked too, here is my 3 settings for Tak/Flac/Wv:

CODE

Encoder:    c:\windows\system32\cmd.exe
Extension:  lossy.tak
Parameters: /d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d
Format is:  lossless or hybrid
Highest BPS mode supported: 24

CODE

Encoder:    c:\windows\system32\cmd.exe
Extension:  lossy.flac
Parameters: /d /c c:\z\lossywav - --portable --silent --stdout|c:\z\flac - -6 -b 512 -f -o%d
Format is:  lossless or hybrid
Highest BPS mode supported: 24

CODE

Encoder:    c:\windows\system32\cmd.exe
Extension:  lossy.wv
Parameters: /d /c c:\z\lossywav - --portable --silent --stdout|c:\z\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is:  lossless or hybrid
Highest BPS mode supported: 24

wavpack works but even with --merge-blocks its efficiency seems reduced compared to real lossless source, for exemple on my single test album flac -6 compressed more than wavpack -h on this lossywav file while according to my experience wavpack -h usually beats flac -6 on real lossless music.

also when I tried to add --keep-foreign-metadata to the flac command line, it crashed.

This post has been edited by sauvage78: Jun 15 2008, 15:39

[GeSomeone]

also when I tried to add --keep-foreign-metadata to the flac command line, it crashed.

That option can only be used when reading from a file (not from STDIN).

[Nick.C]

lossyWAV 1.0.1u RC1 attached to post #1 in this thread.

[halb27]

lossyWAV 1.0.1u RC1 attached to post #1 in this thread.

Thanks for the new RC.
I'm looking forward to the next release.