Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: lossyWAV 1.1.0 Development Thread. (Read 184337 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

lossyWAV 1.1.0 Development Thread.

Reply #200 –
Quote from: halb27 on
I wondered why bitrate came down a bit with 1.0.1o as compared to 1.0.1k and played around with --shaping. The default shaping (--shaping q-value/10) has gone. With the corresponding --shaping bitrate goes up.
  (sorry - I missed that one).

[edit] ps. Thanks for spotting it.

Revised bitrates for my 55 problem sample set (FLAC -8 - 784 kbit/s) :
Code: [Select]
|-------------|------------|------------|------------|------------|------------|
| Version     | --insane   | --extreme  | --standard | --portable |    -q 0    |
|-------------|------------|------------|------------|------------|------------|
| 1.0.1n      | 645 kbit/s | 571 kbit/s | 491 kbit/s | 410 kbit/s | 312 kbit/s |
|-------------|------------|------------|------------|------------|------------|
| 1.0.1p      | 656 kbit/s | 585 kbit/s | 509 kbit/s | 425 kbit/s | 321 kbit/s |
|-------------|------------|------------|------------|------------|------------|
| 1.0.1p --LC | 672 kbit/s | 605 kbit/s | 531 kbit/s | 445 kbit/s | 336 kbit/s |
|-------------|------------|------------|------------|------------|------------|


lossyWAV beta 1.0.1p attached to post #1 in this thread. [/edit]
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #201 –
Quote from: Nick.C on
Thanks for that - I saw it at the time, but my loss of code kind of focussed my attention elsewhere.... I'll amend the wiki.

Oh, I see how loss of code would capture the attention  . Sorry about that; wasn't up-to-date with recent happenings.

On a different matter, I've been doing some tests with my classical CDs to see how lossyWAV does versus pure lossless:

In rare cases (e.g. solo piano) FLAC -5 was smaller than lossyWAV v1.0.1.14 at -q7.5.
So I thought perhaps it would be a good idea, where lossyWAV can't beat the lossless, it might skip the process and encode direct to FLAC or come up with some kind of notice. If I was encoding my whole collection I wouldn't stop to check versus FLAC each time, and later I'd be a little annoyed to find that the lossy was bigger than the lossless.

Glenn Gould 1980 Goldberg Variations FLAC -5 = 177 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 = 183 MB

Perhaps this could be done with a quick replay gain check, the above CD's recommended album gain is  +1.13 dB and I imagine this has a lot to do with lossyWAV's performance.

Thoughts?

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

lossyWAV 1.1.0 Development Thread.

Reply #202 –
Quote from: carpman on
Oh, I see how loss of code would capture the attention  . Sorry about that; wasn't up-to-date with recent happenings.

On a different matter, I've been doing some tests with my classical CDs to see how lossyWAV does versus pure lossless:

In rare cases (e.g. solo piano) FLAC -5 was smaller than lossyWAV v1.0.1.14 at -q7.5.
So I thought perhaps it would be a good idea, where lossyWAV can't beat the lossless, it might skip the process and encode direct to FLAC or come up with some kind of notice. If I was encoding my whole collection I wouldn't stop to check versus FLAC each time, and later I'd be a little annoyed to find that the lossy was bigger than the lossless.

Glenn Gould 1980 Goldberg Variations FLAC -5 = 177 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 = 183 MB

Perhaps this could be done with a quick replay gain check, the above CD's recommended album gain is  +1.13 dB and I imagine this has a lot to do with lossyWAV's performance.

Thoughts?

C.
I re-introduced the possibility to change the codec-block-size at beta 1.0.1o.

In this instance, increasing the length of the codec-block would probably improve the situation.

Use the -b <n> parameter with n=512,1024,2048 or 4096. The corresponding change to -b must be made in the flac encoding command line to take the benefit. I would probably use wavegain on the WAV file initially and set the codec-block-size depending on the result : higher positive = bigger codec-block-size. How to do it in batch I haven't yet had time to work out.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #203 –
Quote from: carpman on
... In rare cases (e.g. solo piano) FLAC -5 was smaller than lossyWAV v1.0.1.14 at -q7.5. ...


I've run upon this too.
Quiet music originating from a solo instrument (or music with very few instruments) is suspected in first place to yield these results.

When I had a lossless archive I produced the lossyFLAC results in the same folder as were the lossless tracks. So it's easy to compare.
The procedure is only necessary in folders with tracks that can be characterized the way I did it above.

BTW I found that in these cases FLAC often compresses in a poor way. TAK or wvPack often have a significantly better compression ratio. Sure I don't know whether this is specific to my music. But you can give it a try.
lame3995o -Q1.7 --lowpass 17

lossyWAV 1.1.0 Development Thread.

Reply #204 –
1.0.1p's average bitrate for my regular music test set:

--portable:  371 kbps
--standard: 463 kbps
--extreme:  544 kbps
--insane:    619 kbps
lame3995o -Q1.7 --lowpass 17

lossyWAV 1.1.0 Development Thread.

Reply #205 –
Quote from: Nick.C on
In this instance, increasing the length of the codec-block would probably improve the situation.

Use the -b <n> parameter with n=512,1024,2048 or 4096. The corresponding change to -b must be made in the flac encoding command line to take the benefit.

Thanks Nick & Halb27, interesting.

FYI:
Glenn Gould 1980 Goldberg Variations FLAC -5 = 177 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 512) = 183 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 1024) = 178 MB (- 5MB)
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 2048) = 175 MB (- 3MB)
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 4096) = 174 MB (- 1MB)

So it looks as though as the block size increases you get diminishing returns.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

lossyWAV 1.1.0 Development Thread.

Reply #206 –
With -b 4096 option, there's strange clicks in right channel...

Also Mardel's sample sounds better at -q 0 with -b 2048 option, though 100% ABXable. Bitrate is 314kbps (-q 0 -b 2048) vs. 310kbps (-q 0 -b 512).

lossyWAV 1.1.0 Development Thread.

Reply #207 –
Quote from: lvqcl on
With -b 4096 option, there's strange clicks in right channel...
That "sounds" like a bug - I'll get onto it.
[edit] Listening to it, it sounds like an overflow in one (or more) codec-blocks of the four that are stored in memory. It should only happen at -b 4096. I'll get a fix out later. [/edit]
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #208 –
When it's about changing block size (for the sake of a potentially increased efficiency of FLAC or whatever lossless codec is used) the essential thing is changing the block size of FLAC.
lossyWAV's blocksize can stay at 512.
The only thing to take care of is that FLAC block size is a multiple of 512 (or, if lossyWAV's block size is changed: a multiple of the lossyWAV blocksize).

Quote from: carpman on
Quote from: Nick.C on

In this instance, increasing the length of the codec-block would probably improve the situation.

Use the -b <n> parameter with n=512,1024,2048 or 4096. The corresponding change to -b must be made in the flac encoding command line to take the benefit.

Thanks Nick & Halb27, interesting.

FYI:
Glenn Gould 1980 Goldberg Variations FLAC -5 = 177 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 512) = 183 MB
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 1024) = 178 MB (- 5MB)
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 2048) = 175 MB (- 3MB)
Glenn Gould 1980 Goldberg Variations Lossy.FLAC -q7.5 (-b 4096) = 174 MB (- 1MB)

So it looks as though as the block size increases you get diminishing returns.

C.

Oh, I forgot to say: in these cases where lossless yields a smaller file size than lossyWAV + FLAC I just stick to lossless. I don't think it's worth while optimizing the lossyWAV + FLAC procedure: just use the lossless encoding in these cases. With emphasis when using wvPack or TAK instead of FLAC as usually file size goes down further, in many cases at a rate of ~10% which is significantly more than the usual ~2% difference between various lossless codecs.
lame3995o -Q1.7 --lowpass 17

lossyWAV 1.1.0 Development Thread.

Reply #209 –
lossyWAV beta 1.0.1q attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #210 –
Echoing what halb27 has said...

The lack of efficiency in these cases in mainly down to FLAC blocksize, rather than lossyWAV itself.

As long as you don't apply positive ReplayGain values, increased bitrates of lossy vs lossless are nothing to worry about, even if you deleted your lossless original. Just take the lossy.flac file and re-encode it with normal FLAC - there will be no further loss (FLAC is lossless!) but it defaults to a different block length, which probably reduces the bitrate for such tracks.

You can go back to the lossless original, and re-encode it with a different block size via lossyWAV and then FLAC, potentially giving a slightly lower bitrate still - but if this isn't possible, don't worry - the above approach will still bring the bitrate down (usually).


Do you think it would be worth adding a utility to check for this? e.g.
Code: [Select]
If original = FLAC, then
  If filesize lossyFLAC > filesize lossless FLAC, then
    re-encode lossyFLAC with same parameters (blocksize, compression) as lossless FLAC to generate new lossy FLAC.
    If lossless FLAC still smaller, replace lossy with lossless; endif
  endif
endif

If the original isn't FLAC, it's not worth encoding everything to lossless FLAC just to check - but if it's already there, the speed hit should be tiny.

You could, of course, do the whole lot (lossyWAV and FLAC) with multiple block sizes and pick the smallest, but this would be overkill (though you could add an --overkill option Nick!) - whereas the above almost comes for free, and would solve the problem neatly if transcoding from FLAC.

Cheers,
David.

lossyWAV 1.1.0 Development Thread.

Reply #211 –
Quote from: 2Bdecided on
.. the above almost comes for free, and would solve the problem neatly if transcoding from FLAC.

The same would apply if it was from another lossless codec (TAK, WavPack).

I would not call "re-encode lossyFLAC" "almost for free".  I also think this is not a job for lossyWav, especially with piped input/output it couldn't tell. Maybe the script (that most use anyway at the moment) could be adapted for these cases.
In theory, there is no difference between theory and practice. In practice there is.

lossyWAV 1.1.0 Development Thread.

Reply #212 –
I realise it can't go comfortably into lossyWAV - even when not "piped" it doesn't necessarily know what or where the source FLAC is, because the input is WAV. If it can be handled in the script though, that would be great.

By "almost for free" I meant the checking.


I guess it works for other codecs to, but not necessarily when converting from one lossless codec to another (via lossyWAV). In that case, an increase in filesize might be just because codec Y was less efficient than codec X, so changing the block size might make it even worse.

Cheers,
David.

lossyWAV 1.1.0 Development Thread.

Reply #213 –
So the default noise shaping is --shaping (q_value/10). But... longhelp still says
Quote
-s, --shaping <n>  enable fixed noise shaping; (0.00<=n<=1.00); default=0;
                    0.00 = 0%, 1.00=100% effectiveness, 0.5=50%, etc.

And yes, default=0; but this means that user should use --shaping 0.001 if he wants to disable noise shaping. 

lossyWAV 1.1.0 Development Thread.

Reply #214 –
Quote from: lvqcl on
So the default noise shaping is --shaping (q_value/10). But... longhelp still says
Quote
-s, --shaping <n>  enable fixed noise shaping; (0.00<=n<=1.00); default=0;
                    0.00 = 0%, 1.00=100% effectiveness, 0.5=50%, etc.
And yes, default=0; but this means that user should use --shaping 0.001 if he wants to disable noise shaping. 
Longhelp is obviously wrong and will be corrected at the next revision;
Use --shaping 0 if you really want to disable noise shaping.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #216 –
Quote from: lvqcl on
Quote from: Nick.C on
Longhelp is obviously wrong and will be corrected at the next revision;
  Great.
Quote from: Nick.C on
Use --shaping 0 if you really want to disable noise shaping.
I  tried but failed    It looks like --shaping 0 has no effect.
Sorry - you're right - I'll re-write that bit of code so that if -s 0 or --shaping 0 is used then shaping will (definitely!!!) be disabled....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #217 –
I've been working on a method of speeding up the code by using a combination of the fxtract assembly instruction and two lookup tables (2048 values for exponent, 4097 values for mantissa) to reduce the time spent calculating square roots. The final method reduces the processing time by about 10% at -q 0 (13.6 seconds down to 12.2) and is considered to be suitably accurate for its intended purpose.

I have also "fixed" the --scale parameter to accept values in the range 0<n<=8 as it was before I had my flash drive failure....

lossyWAV beta 1.0.1r attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #218 –
Quote from: Nick.C on
I have also "fixed" the --scale parameter to accept values in the range 0<n<=8 as it was before I had my flash drive failure....

Thanks for that. I can't use --dither in versions 101q and 101r; in version 1.0.1k it worked well.
"Unknown error in unknown module" and the program bails out.

lossyWAV 1.1.0 Development Thread.

Reply #219 –
Quote from: Nick.C on
I've been working on a method of speeding up the code by using a combination of the fxtract assembly instruction and two lookup tables (2048 values for exponent, 4097 values for mantissa) to reduce the time spent calculating square roots. The final method reduces the processing time by about 10% at -q 0 (13.6 seconds down to 12.2) and is considered to be suitably accurate for its intended purpose.
Maybe worth having a look here, maybe there's more speed to be had from the square root optimisations..
http://www.codemaestro.com/reviews/9

Also, if you're maybe going to get your hands dirty with a smidge of assembler, maybe there'll be gains to be had if you can work with the reciprocals instead..

Edit: Actually scratch that.. I've just run some tests and it wasn't pleasant

lossyWAV 1.1.0 Development Thread.

Reply #220 –
Quote from: m00 on
Maybe worth having a look here, maybe there's more speed to be had from the square root optimisations..
http://www.codemaestro.com/reviews/9

Also, if you're maybe going to get your hands dirty with a smidge of assembler, maybe there'll be gains to be had if you can work with the reciprocals instead..

Edit: Actually scratch that.. I've just run some tests and it wasn't pleasant
I had a look there - I don't know to convert that for a 64-bit float. I do use reciprocals and multiplication wherever possible - it is a lot quicker. Thanks for taking the time to make the suggestion!

[edit] One of the links points to a PDF with a 64-bit float version included - I'll see how it goes.... [/edit]
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #221 –
I ran into something little with version 1.0.1q.
At last I have a script for foobar2000 that takes either a sourcefile (%s) or stdin (-).
What I try to do is use stdin with lossyWav, then lossywav.lossy.wav as input file for FLAC. (Yes, trying to get the best of both worlds, one temp file less, keep the metadata and have normal seektables in FLAC.)

As I was trying to find out why it didn't work , I noticed FLAC --keep-foreign-metadata didn't seem to work when lossywav is fed from stdin, even though FLAC can read from a normal file.
Flac barfs: lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

And when I try FLAC on the same file without --keep-foreign-metadata

lossywav.lossy.wav: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
lossywav.lossy.wav: WARNING: unexpected EOF; expected 1073741812 samples, got 10825216 samples

now the file should have 10825668 samples according to foobar (I guess also the billion sample value comes from the foobar converter) ... and these last two are just warnings. The resulting file has the same length as the input.
So there is at least something strange with the FACT chunk when lossyWav uses stdin (or is that the result of the wrong sample value too?)

Maybe this can help to solve the stdin strangeness.


BTW am I being silly by still sticking to the FACT chunk idea? 
In theory, there is no difference between theory and practice. In practice there is.
 

lossyWAV 1.1.0 Development Thread.

Reply #222 –
Quote from: GeSomeone on
I ran into something little with version 1.0.1q.
At last I have a script for foobar2000 that takes either a sourcefile (%s) or stdin (-).
What I try to do is use stdin with lossyWav, then lossywav.lossy.wav as input file for FLAC. (Yes, trying to get the best of both worlds, one temp file less, keep the metadata and have normal seektables in FLAC.)

As I was trying to find out why it didn't work , I noticed FLAC --keep-foreign-metadata didn't seem to work when lossywav is fed from stdin, even though FLAC can read from a normal file.
Flac barfs: lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

And when I try FLAC on the same file without --keep-foreign-metadata

lossywav.lossy.wav: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
lossywav.lossy.wav: WARNING: unexpected EOF; expected 1073741812 samples, got 10825216 samples

now the file should have 10825668 samples according to foobar (I guess also the billion sample value comes from the foobar converter) ... and these last two are just warnings. The resulting file has the same length as the input.
So there is at least something strange with the FACT chunk when lossyWav uses stdin (or is that the result of the wrong sample value too?)

Maybe this can help to solve the stdin strangeness.


BTW am I being silly by still sticking to the FACT chunk idea? 
Meh - it looks like I broke the stdin file output.... Sorry! I'll try to get beta 1.0.1s out tonight.

[edit] Actually, it is that foobar2000 tells an "encoder" using STDIN that the "WAV file" is Max size, i.e. 4,294,967,294 - 12 bytes long (12 bytes for 'RIFF'/0FFFFFFF2h/'WAVE').

I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size. [/edit]
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

lossyWAV 1.1.0 Development Thread.

Reply #223 –
Quote from: Nick.C on
Quote from: GeSomeone on
[..](I guess also the billion sample value comes from the foobar converter) ...
[edit] Actually, it is that foobar2000 tells an "encoder" using STDIN that the "WAV file" is Max size, i.e. 4,294,967,294 - 12 bytes long (12 bytes for 'RIFF'/0FFFFFFF2h/'WAVE').

I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size. [/edit]

Yes it is the foobar2000 converter. We (Halb27 and myself) tried to make you aware of this. But don't worry, this is what betas are for . I found an old posting where Peter replies about this.
If you could fix it (put the correct number of samples in the header) that would be nice.
In theory, there is no difference between theory and practice. In practice there is.

lossyWAV 1.1.0 Development Thread.

Reply #224 –
Check the first post.
Quote
Change log 1.0.1q: 30/05/08


Maybe you should attach list of programs wich supporting lossywav?

p.S. GXTranscoder