(sorry - I missed that one).

[edit] ps. Thanks for spotting it.

Revised bitrates for my 55 problem sample set (FLAC -8 - 784 kbit/s) :

lossyWAV beta 1.0.1p attached to post #1 in this thread. [/edit]

Oh, I see how loss of code would capture the attention . Sorry about that; wasn't up-to-date with recent happenings.

On a different matter, I've been doing some tests with my classical CDs to see how lossyWAV does versus pure lossless:

In rare cases (e.g. solo piano) FLAC -5 was smaller than lossyWAV v1.0.1.14 at -q7.5.
So I thought perhaps it would be a good idea, where lossyWAV can't beat the lossless, it might skip the process and encode direct to FLAC or come up with some kind of notice. If I was encoding my whole collection I wouldn't stop to check versus FLAC each time, and later I'd be a little annoyed to find that the lossy was bigger than the lossless.

Glenn Gould 1980 Goldberg Variations FLAC -5 = 177 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 = 183 MB

Perhaps this could be done with a quick replay gain check, the above CD's recommended album gain is +1.13 dB and I imagine this has a lot to do with lossyWAV's performance.

Thoughts?

C.

I re-introduced the possibility to change the codec-block-size at beta 1.0.1o.

In this instance, increasing the length of the codec-block would probably improve the situation.

Use the -b <n> parameter with n=512,1024,2048 or 4096. The corresponding change to -b must be made in the flac encoding command line to take the benefit. I would probably use wavegain on the WAV file initially and set the codec-block-size depending on the result : higher positive = bigger codec-block-size. How to do it in batch I haven't yet had time to work out.

I've run upon this too.
Quiet music originating from a solo instrument (or music with very few instruments) is suspected in first place to yield these results.

When I had a lossless archive I produced the lossyFLAC results in the same folder as were the lossless tracks. So it's easy to compare.
The procedure is only necessary in folders with tracks that can be characterized the way I did it above.

BTW I found that in these cases FLAC often compresses in a poor way. TAK or wvPack often have a significantly better compression ratio. Sure I don't know whether this is specific to my music. But you can give it a try.

1.0.1p's average bitrate for my regular music test set:

--portable:  371 kbps
--standard: 463 kbps
--extreme:  544 kbps
--insane:    619 kbps

Thanks Nick & Halb27, interesting.

FYI:
Glenn Gould 1980 Goldberg Variations FLAC -5 = 177 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 512) = 183 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 1024) = 178 MB (- 5MB)
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 2048) = 175 MB (- 3MB)
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 4096) = 174 MB (- 1MB)

So it looks as though as the block size increases you get diminishing returns.

C.

With -b 4096 option, there's strange clicks in right channel...

Also Mardel's sample sounds better at -q 0 with -b 2048 option, though 100% ABXable. Bitrate is 314kbps (-q 0 -b 2048) vs. 310kbps (-q 0 -b 512).

That "sounds" like a bug - I'll get onto it.
[edit] Listening to it, it sounds like an overflow in one (or more) codec-blocks of the four that are stored in memory. It should only happen at -b 4096. I'll get a fix out later. [/edit]

When it's about changing block size (for the sake of a potentially increased efficiency of FLAC or whatever lossless codec is used) the essential thing is changing the block size of FLAC.
lossyWAV's blocksize can stay at 512.
The only thing to take care of is that FLAC block size is a multiple of 512 (or, if lossyWAV's block size is changed: a multiple of the lossyWAV blocksize).


Oh, I forgot to say: in these cases where lossless yields a smaller file size than lossyWAV + FLAC I just stick to lossless. I don't think it's worth while optimizing the lossyWAV + FLAC procedure: just use the lossless encoding in these cases. With emphasis when using wvPack or TAK instead of FLAC as usually file size goes down further, in many cases at a rate of ~10% which is significantly more than the usual ~2% difference between various lossless codecs.

lossyWAV beta 1.0.1q attached to post #1 in this thread.

Echoing what halb27 has said...

The lack of efficiency in these cases in mainly down to FLAC blocksize, rather than lossyWAV itself.

As long as you don't apply positive ReplayGain values, increased bitrates of lossy vs lossless are nothing to worry about, even if you deleted your lossless original. Just take the lossy.flac file and re-encode it with normal FLAC - there will be no further loss (FLAC is lossless!) but it defaults to a different block length, which probably reduces the bitrate for such tracks.

You can go back to the lossless original, and re-encode it with a different block size via lossyWAV and then FLAC, potentially giving a slightly lower bitrate still - but if this isn't possible, don't worry - the above approach will still bring the bitrate down (usually).


Do you think it would be worth adding a utility to check for this? e.g.

If the original isn't FLAC, it's not worth encoding everything to lossless FLAC just to check - but if it's already there, the speed hit should be tiny.

You could, of course, do the whole lot (lossyWAV and FLAC) with multiple block sizes and pick the smallest, but this would be overkill (though you could add an --overkill option Nick!) - whereas the above almost comes for free, and would solve the problem neatly if transcoding from FLAC.

Cheers,
David.

The same would apply if it was from another lossless codec (TAK, WavPack).

I would not call "re-encode lossyFLAC" "almost for free". I also think this is not a job for lossyWav, especially with piped input/output it couldn't tell. Maybe the script (that most use anyway at the moment) could be adapted for these cases.

I realise it can't go comfortably into lossyWAV - even when not "piped" it doesn't necessarily know what or where the source FLAC is, because the input is WAV. If it can be handled in the script though, that would be great.

By "almost for free" I meant the checking.


I guess it works for other codecs to, but not necessarily when converting from one lossless codec to another (via lossyWAV). In that case, an increase in filesize might be just because codec Y was less efficient than codec X, so changing the block size might make it even worse.

Cheers,
David.

So the default noise shaping is --shaping (q_value/10). But... longhelp still says

And yes, default=0; but this means that user should use --shaping 0.001 if he wants to disable noise shaping.

Longhelp is obviously wrong and will be corrected at the next revision;
Use --shaping 0 if you really want to disable noise shaping.

Great.

I tried but failed It looks like --shaping 0 has no effect.

Sorry - you're right - I'll re-write that bit of code so that if -s 0 or --shaping 0 is used then shaping will (definitely!!!) be disabled....

I've been working on a method of speeding up the code by using a combination of the fxtract assembly instruction and two lookup tables (2048 values for exponent, 4097 values for mantissa) to reduce the time spent calculating square roots. The final method reduces the processing time by about 10% at -q 0 (13.6 seconds down to 12.2) and is considered to be suitably accurate for its intended purpose.

I have also "fixed" the --scale parameter to accept values in the range 0<n<=8 as it was before I had my flash drive failure....

lossyWAV beta 1.0.1r attached to post #1 in this thread.

Thanks for that. I can't use --dither in versions 101q and 101r; in version 1.0.1k it worked well.
"Unknown error in unknown module" and the program bails out.

Maybe worth having a look here, maybe there's more speed to be had from the square root optimisations..
http://www.codemaestro.com/reviews/9

Also, if you're maybe going to get your hands dirty with a smidge of assembler, maybe there'll be gains to be had if you can work with the reciprocals instead..

Edit: Actually scratch that.. I've just run some tests and it wasn't pleasant

I had a look there - I don't know to convert that for a 64-bit float. I do use reciprocals and multiplication wherever possible - it is a lot quicker. Thanks for taking the time to make the suggestion!

[edit] One of the links points to a PDF with a 64-bit float version included - I'll see how it goes.... [/edit]

I ran into something little with version 1.0.1q.
At last I have a script for foobar2000 that takes either a sourcefile (%s) or stdin (-).
What I try to do is use stdin with lossyWav, then lossywav.lossy.wav as input file for FLAC. (Yes, trying to get the best of both worlds, one temp file less, keep the metadata and have normal seektables in FLAC.)

As I was trying to find out why it didn't work , I noticed FLAC --keep-foreign-metadata didn't seem to work when lossywav is fed from stdin, even though FLAC can read from a normal file.
Flac barfs: lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

And when I try FLAC on the same file without --keep-foreign-metadata

lossywav.lossy.wav: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
lossywav.lossy.wav: WARNING: unexpected EOF; expected 1073741812 samples, got 10825216 samples

now the file should have 10825668 samples according to foobar (I guess also the billion sample value comes from the foobar converter) ... and these last two are just warnings. The resulting file has the same length as the input.
So there is at least something strange with the FACT chunk when lossyWav uses stdin (or is that the result of the wrong sample value too?)

Maybe this can help to solve the stdin strangeness.


BTW am I being silly by still sticking to the FACT chunk idea?

Meh - it looks like I broke the stdin file output.... Sorry! I'll try to get beta 1.0.1s out tonight.

[edit] Actually, it is that foobar2000 tells an "encoder" using STDIN that the "WAV file" is Max size, i.e. 4,294,967,294 - 12 bytes long (12 bytes for 'RIFF'/0FFFFFFF2h/'WAVE').

I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size. [/edit]

Yes it is the foobar2000 converter. We (Halb27 and myself) tried to make you aware of this. But don't worry, this is what betas are for . I found an old posting where Peter replies about this.
If you could fix it (put the correct number of samples in the header) that would be nice.

Check the first post.


Maybe you should attach list of programs wich supporting lossywav?

p.S. GXTranscoder

The list of known supported codecs is in the wiki article linked from the first page in the thread.

lossyWAV beta 1.0.1s attached to post #1 in this thread.

[edit] The speed of transcoding in foobar2000 has been dramatically increased by the use of STDIN and STDOUT. I transcoded my Mike Oldfield collection (366 tracks, 33h29m) in 22m18s or approximately 90.09x overall {Intel E6600 C2D @ 3.0GHz, 4GB DDR2-800, Windows MCE2005 SP2, source on server, destination 160GB Maxtor SATA HDD}. This was definitely limited by the processor cores as the utilisation plot for the duration of the transcode was a virtual flat-line at 100%.

What this does mean though is that the 'fact' chunk will not be preserved - so be careful not to mix-up your archive FLAC files with your lossyFLAC files. [/edit]

Thanks Nick,

Something has changed with STDIN but it seems still not quite right.

When I feed the intermediate file lossywav.lossy.wav to STDIN of FLAC (without --keep.. ) says
-: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
-: WARNING: unexpected EOF; expected 5869416 samples, got 5868544 samples
BTW according to foobar2000 the number of samples of the source is also 5869416 (and the lossywav and resulting lossy.flac are 5869056) I'm not sure foobar displays correct values.

When I let FLAC read the lossy file (with --keep.. )
lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

Is that FACT chunk becoming a nuisance now, or is it something else?

Could you please post your converter settings so that I can duplicate the problem?

Many thanks.

[edit] Problem found here too - average of 249 samples dropped off the end of 85 album lossyFLAC files. It may be something to do with the last read (incomplete codec-block [probably]) - I'll get onto it and revert soon. Until such times, STDIN input is not recommended to be used (--stdout seems fine....) [/edit]

Many thanks for the heads up GeSomeone - it was as I surmised and it now seems to work properly in beta 1.0.1t.

lossyWAV beta 1.0.1t attached to post #1 in this thread.

Thank you, Nick, for the new version.
I encoded my regular music test set with piping and with my bat file.
Speed is great when using piping (though it is good with the bat file).
I tested the musical content just to make sure, and with all the tracks content is identical no matter which encoding way.

Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).

I would guess that FLAC is inserting a larger number of seek points due to the fact that (in foobar2000) the assumed size of the piped file is maximum, i.e. 4,294,967,294 bytes.

Processing on my laptop, using two intermediate WAV files, used to be a real HDD thrashing exercise - 40 to 50 minutes for my 10 album test set (3.36GB, 9h22m57s total duration). Using piped input and output, the transcode now takes approximately 10 minutes - a really nice reduction in total processing time.

I've just finished a "stress test" of sorts: all of my FLAC'ed CD's on my server to lossyFLAC (-q 1.25). 12d4h24m45s (4064 tracks, 316 CD's, 108GB, 887kbit/s average) transcoded in 3h33m23s (40.5GB, 331 kbit/s average) - 82.22x overall and exactly the same number of samples out as went in . Bearing in mind that previously my 3686 track 100GB(ish) transcode took approximately 7h30m, that's an appreciable saving.

If there are no reported problems and if no-one has any further suggestions then now may be the time to move towards a release candidate for 1.0.2.

Yes thanks a lot, this time my little script works as intended

Except from the metadata it could be the larger seektable that FLAC allocates for a file of unknown length, as foobar2000's converter generates.

I'll show my script (batfile) that (with this fix)

• handles either - or %s as input.
• one intermediate file less than the "old" script
• the FLAC files are a few kB smaller because no large seektables and 2kB padding (more than enough for me) instead of 8kB.
• Foreign metadata is added to the FLAC file, though I doubt lossyWav will ever see it again when converting from STDIN (with foobar).

Cons

• can run only 1 converter thread at once
• takes less advantage of multi core/hyperthreading than piping directly from lossyWav into FLAC

foobar2000:
Encoder: C:\WINDOWS\system32\cmd.exe
Extension: lossy.flac
Parameters: /D /C C:\PROGRA~1\FLAC\lossyFlac.bat - %d -q 4.5 (or any lossyWav settings)

and my lossywav.bat

Of course that logging stuff can be removed and on Vista it should be in another location.

[edit] some tidying up and removing a debug line from the script [/edit]

Yes, it's the seektable. When using --no-seektable with FLAC, I arrive at an expected file size.

Is the seektable necessary? I had no problems skipping part of the track (using foobar) with the --no-seektable version.

What kind of problems can I encounter when using --no-seektable?

Did Josh not explain this once - I think that it only takes a bit longer to seek when moving about the file.

I don't encounter any disadvantage using --no-seektable when playing on my DAP.

So I'll use --no-seektable together with GeSomeone's --padding=2048 proposal and enjoy piping.


I second this.

I don't know if anyone uses REACT to do their EAC tasks, but I have a rough (but working) model of hacking the track-config file to use lossyWAV/lossFlac.



(and these were in the REACT.ini:)



Not groundbreaking, I guess, but a start. I don't know how to use the temp variables in batch files, so I use a static temp name and then delete it. But, it works fine. I get resulting lossyFlac files with proper tagging, album art embedded, and Replay Gain tagging.

Scott

At first, when ripping to images, I added seekpoints for every 90 seconds. Now I use -S- and noticed that the cue points from the embedded cue sheet are sufficient. And because I can use the cuesheet to play the disc image I can jump to any track or movement I like I like.

hi, today I played with lossywav again in order to have it working for good with F2K & tak, so I copy/pasted your command line from post N°1, & it worked like a charm with flac

/d /c c:\data_nic\bin\lossywav - --standard --silent --stdout|c:\data_nic\bin\flac - -b 512 -5 -f -o%d

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

But my knowledge being very limited I don't undertand the use of /d /c ...
I understand that I call cmd.exe with c:\windows\system32\cmd.exe then lossywav with c:\z\lossywav - --portable --silent --stdout| then tak with c:\z\Takc -e -p3 -fsl512 -ihs - %d, but I don't get the use of /d /c, but if I delete it it doesn't work anymore so could someone explain the use of /d /c ? What is it needed for ? Also is there any mistake in my own lossyTak command line ? Thks

thks for cmd /? ... didn't knew about it

No problem - I'm glad that the command line is working properly for you. Thanks for the Tak command line - I'll include it in the first post.

From what collector and halb27 have said, maybe --no-seektable -P=4096 would be a sensible way of reducing the size of the seektable when using STDIN / STDOUT in foobar2000. I processed my 10 Album test set and I think this saved about 1MB (in 1.37GB processed). Still, 1MB is 1MB....

Looks good

In my evaluations of LossyWav processed TAK test file sets there was no advantage of choosing a preset higher than -p2m. Some file sets even compressed about 0.05 percent worse with -p3 to -p5...

Thomas

edit: Content of duplicate post deleted...

Nick.C:
... well I haven't yet read all the discussion of collector and halb27 on seektable ... I only come to read this thread once a week/month to see how things go on ... I know I will not use lossywav for production as long as it's not stable ... so I am not in a hurry to read it all

TBeck:
Thks for the hint ... I was planning to test the efficiency of flac -q6 vs tak -p3 vs wv -hm on lossywav processed files as I think I read somewhere not all lossless codecs where equal when encoding from lossywav ... I think I read wavpack wasn't optimal compared to tak & flac ... will add tak at different settings when I will test ... post lossywav V1.2 ... with some luck maybe by then I will have a frozen tak too

Thanks for the clarification, Thomas.

Just tested with wavpack 4.5 & it worked too, here is my 3 settings for Tak/Flac/Wv:







wavpack works but even with --merge-blocks its efficiency seems reduced compared to real lossless source, for exemple on my single test album flac -6 compressed more than wavpack -h on this lossywav file while according to my experience wavpack -h usually beats flac -6 on real lossless music.

also when I tried to add --keep-foreign-metadata to the flac command line, it crashed.

That option can only be used when reading from a file (not from STDIN).

lossyWAV 1.0.1u RC1 attached to post #1 in this thread.

Thanks for the new RC.
I'm looking forward to the next release.