New FLAC encoder Options

[HbG]

CODE

Pearl Jam - Ten (new european version) / Track 11: Release / Length: 543.426 seconds
System: Athlon XP 3200 - 1GB DDR400 - NForce2

Flac version:  libFLAC CVS 20060425
Flake version: flake-06

encode   | ratio | bitrate | speed  | dec. speed
---------|-------|---------|--------|-------------
flac   8 | 0.534 | 754.?   | 11.93x | 192.30x
flake  8 | 0.533 | 752.1   | 23.31x | 187.11x
flake 12 | 0.531 | 749.8   |  4.27x | 162.23x

Commandlines for encoding:
timethis "start /realtime /b /wait flac -8 test.wav -o NUL: -f"
timethis "start /realtime /b /wait flake -8 test.wav -o NUL:"
timethis "start /realtime /b /wait flake -12 test.wav -o NUL:"

Encodes were done three times, fastest run counts. Spread between runs was small, no more than 0.1s. Speeds are rounded.

Decoding speed was measured with Foobar2000 0.9.1, 10 passes, no dsp, high priority, buffer in memory.
Flake -8 file decodes about 44% faster with flacdec.

A 95.5% speedup. Impressive. Flake -8 is my new god.

I still miss some options, most important to me is -V as i like to overclock my hardware. Veryfying doesn't slow encoding down much anyway. Also reading from stdin doesn't seem to work on windows, it will say "invalid wav file: -" and then proceeds create a 0 byte flac output file, another feature it could do without
For the rest it's nothing metaflac can't handle.

[Justin Ruggles]

CODE

Pearl Jam - Ten (new european version) / Track 11: Release / Length: 543.426 seconds
System: Athlon XP 3200 - 1GB DDR400 - NForce2

Flac version:  libFLAC CVS 20060425
Flake version: flake-06

encode   | ratio | bitrate | speed  | dec. speed
---------|-------|---------|--------|-------------
flac   8 | 0.534 | 754.?   | 11.93x | 192.30x
flake  8 | 0.533 | 752.1   | 23.31x | 187.11x
flake 12 | 0.531 | 749.8   |  4.27x | 162.23x

Wonderful. This is typical of the results I get as well. I'm glad to see the relative encoding times are comparable on a fast, modern system.

A 95.5% speedup. Impressive. Flake -8 is my new god.

I'm glad you like it. Keep in mind that it is experimental at this point, but there will eventually be another release which will use FFmpeg's wav decoder & FLAC encoder. So far it seems it might end up being faster as well.

I still miss some options, most important to me is -V as i like to overclock my hardware. Veryfying doesn't slow encoding down much anyway.

Oddly enough, I was just thinking about this today. I noticed many people on Hydrogenaudio use this option. I will definitely add something similar when I start back working on Flake. I might have to use a modified version of FFmpeg's FLAC decoder instead of doing it directly through libavcodec. But you're right that this is an important feature of the reference encoder.

Also reading from stdin doesn't seem to work on windows, it will say "invalid wav file: -" and then proceeds create a 0 byte flac output file, another feature it could do without

Hmm. Maybe that has to do with having to set stdin to binary mode in windows...maybe. This is good to know though. Thanks for the feedback.

For the rest it's nothing metaflac can't handle.

I might even add metadata/cuesheet/seektable support someday. Personally, I'm not a fan of the vorbis/flac comment specification, but it works.

All of this is probably at least a month or two (or three) away. I have a lot going on in the next couple months.

Thanks,
-Justin

[surf]

hello.
does this new encoder supports unicode?
i can't connect to the page...404

[sundance]

Justin,

I'd also like to join in saying "thank-you" for your efforts to improve FLAC encoding. Highly appreciated!
I also integrated it into fb2k; the only drawback (= speed penalty) here is that reading from stdin doesn't work here either so temp. WAVs have to be uses with the converter...

Keep it up!

-sundance-


Btw, why didn't you use another vendor string than "[put libavcodec/libavformat versions here]"?

[Justin Ruggles]

Hello,

Btw, why didn't you use another vendor string than "[put libavcodec/libavformat versions here]"?

Because one of my main purposes for writing the encoder was to integrate it with FFmpeg. I just needed a frontend to test it with before integrating it, so I made Flake. The next release will have the lavc version as the vendor string.

Thanks,
-Justin

[HbG]

I just needed a frontend to test it with before integrating it, so I made Flake. The next release will have the lavc version as the vendor string.

Thanks,
-Justin

I can't speak for the rest of us, but i really like the standalone, dropin flac replacement that flake is. I'm not very familiar with the ffmpeg project, but i fear it won't be as easy to use from the commandline or as encoder for eac or foobar when integrated into ffmpeg.

[sundance]

...but i really like the standalone, dropin flac replacement that flake is.

Second that!

[Justin Ruggles]

I just needed a frontend to test it with before integrating it, so I made Flake. The next release will have the lavc version as the vendor string.

Thanks,
-Justin

I can't speak for the rest of us, but i really like the standalone, dropin flac replacement that flake is. I'm not very familiar with the ffmpeg project, but i fear it won't be as easy to use from the commandline or as encoder for eac or foobar when integrated into ffmpeg.

Not to worry. I will still re-release Flake as a standalone encoder once the code is part of FFmpeg. I'll just use the relevant code from FFmpeg and might have to restructure it a little bit for use with Flake. FFmpeg has/is a commandline frontend to its libavcodec and libavformat libraries. When using FFmpeg, you'll most likely only be able to specify the compression level, not each individual option. Flake will use the FFmpeg code, but be specifically only for FLAC encoding and will provide more detailed options along the lines of what it has currently.

As far as familiarity, you can find out more about FFmpeg at
http://ffmpeg.mplayerhq.hu/
It is a wonderful project with very good developers. If you don't mind the 50 or so emails a day, subscribe to ffmpeg-devel and watch for a while...or just browse through the archives.

-Justin

[wisodev]

Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/. All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)

[Justin Ruggles]

Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/. All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)

Thank you! Especially for the win32 build environment stuff! I am currently working on a simple configure/build system for unix environments. I'll try to use some of what you have done as a basis to improve Windows compatibility in the next release.

The current development version of Flake has about a 15% speed improvement over pre-release 06 (on x86 linux gcc 3.4.4). Compression has generally improved only slightly so far, except at compression level 0, which has a significant improvement. FFmpeg integration of the full encoder is only about 1/2 done right now, but is going very well I think. Right now it has levels 0 to 5, and you can change specific options on the commandline. If you build FFmpeg from current SVN, here are some of the FLAC encoding options.

Encode at level 5:
ffmpeg -i test.wav test.flac

Encode at level 0:
ffmpeg -i test.wav -compression_level 0 test.flac

Encode using individual options:
ffmpeg -i test.wav -frame_size 2048 -use_lpc 1 -min_prediction_order 6 -max_prediction_order 12 -min_partition_order 0 -max_partition_order 6 -lpc_coeff_precision 15 test.flac

Quickly convert ogg-flac to raw flac:
ffmpeg -i test.ogg -acodec copy test.flac

Also, the most updated Flake webpage is at another location. I will be able to post some test samples of my own instead of linking to other sites since I now have more storage space. I am limited to only 10MB on my Earthlink account.
http://jbr.homelinux.org/flake/

-Justin

[wisodev]

Thank you! Especially for the win32 build environment stuff! I am currently working on a simple configure/build system for unix environments.

Quick note about the win32 project files (VC++ 7.1 and ICL 8.0 compatible): the FLAKE code compiles only with ICL not CL because of C99 option in intel compiler settings (without this option code does not compile). So the build environment works only with visual studio and intel compiler integrated. But it works very well without any nasty errors and produce great binarys ;-). I will in near future update the build environment to Intel 9.1 (actually I have done this locally and only need to update online stuff, but before that some tests will be committed to check if new version works good as prevoius).

[HbG]

Well I am propobly too late but I have builded some binarys for win32 using intel compiler. If someone want to try them here is link http://thefrontend.sourceforge.net/flake/. All versions compiled be me where faster then kurtnoise builds on my machine (Athlon XP 2000+). Archives include full sources, binarys and project files. In file TESTS-0.4-win32.zip are some speed tests results just to prove me claims ;-)

Nice, for version 0.6 at -8 it's 26% faster here, with 29.40x vs 23.31x.

That makes it 146% faster than flac!

Oh, one more thing that just comes to mind, i miss a -q option for flake, but that's a minor issue. It does boost performance a tiny bit.

[Justin Ruggles]

Oh, one more thing that just comes to mind, i miss a -q option for flake, but that's a minor issue. It does boost performance a tiny bit.

That just never crossed my mind. I'll add it to the next release.

Thanks,
Justin

[jcoalson]

another thing I would suggest is a big fat warning if the compression setting used is not in the FLAC subset. not sure, looks like currently all settings are in the subset. but I'm considering restricting the subset further (to lpc order<=12 and blocksize <=4608) for the next release, at least for <=48kHz.

Josh

[Justin Ruggles]

another thing I would suggest is a big fat warning if the compression setting used is not in the FLAC subset. not sure, looks like currently all settings are in the subset. but I'm considering restricting the subset further (to lpc order<=12 and blocksize <=4608) for the next release, at least for <=48kHz.

Thanks Josh. The 0-8 presets in Flake are within the subset, but the block size can be specified on the commandline to be any valid size within the spec, not just limited to the predefined ones required for subset.

I agree with your intentions to further restrict the subset requirements. Block size over 4608 is rarely beneficial anyway unless it is a quite stationary signal, but the affect on decoding speed is substantial. Maybe this would also make it easier to standardize hardware support. There are products which say that they support FLAC levels 0 to 8. This doesn't really say much about what else is truly supported outside the predefined levels. If subset limits were restricted to the settings of level 8 it would be simpler and more definitive to just say that a product supports the FLAC Subset format or the full FLAC format (or FLAC 2 ).

-Justin

[Justin Ruggles]

Hello,

I'm posting again here because the FFmpeg FLAC encoder has some significant changes. Nearly all of Flake-06 has been integrated, along with some pretty great improvements by Michael Niedermayer (lead FFmpeg developer). It is quite a bit faster than Flake (level 8 is about 60% faster than the reference encoder). Some of the key improvements are:

• encoding speed at lower levels is now on-par with the reference encoder
• better stereo decorrelation
• new, faster log search for finding optimal LPC order
• optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion

I have done a few benchmark comparisons.

The FFmpeg homepage has instructions for downloading/compiling the current SVN version.

One major thing left to do is to work out how to encode the MD5 sum in the global header. Since FFmpeg supports many container formats, it is proving to be a little tricky.

-Justin

[krmathis]

Great work!
Hope you plan to "back-port" these improvements to the standalone flake encoder as well?

[Garf]

• optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion

What's the advantage of the Cholesky method? I'm not familiar with it.

[Justin Ruggles]

• optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion

What's the advantage of the Cholesky method? I'm not familiar with it.

Michael Niedermayer explains it better than I ever could here. It took me a while to understand it even partially.

You probably have a better math background than me, so these might help.
Choleski decomposition
Gaussian elimination
Toeplitz matrix
Autocorrelation
Levinson recursion

Great work!
Hope you plan to "back-port" these improvements to the standalone flake encoder as well?

Yes I do. Some have been already, but I don't want to release anything new until the FFmpeg encoder gets to a good stable state. (i.e. until the MD5/extradata/other containers issue has been solved).

-Justin

This post has been edited by Justin Ruggles: Jul 17 2006, 10:36

[Garf]

Okay, I understand the what and why

What's the speed impact?

[Justin Ruggles]

Okay, I understand the what and why

What's the speed impact?

Very big. 1-pass is only a little slower, but it does not give as good results as Levinson-Durbin. The two seem to be equivilant in compression at around 3-4 pass. So the real benefit is if you have a fast computer and/or lots of time to spare to wait for it to squeeze out a few more tenths of a percent. Decoding speed is not affected.

example:
Levinson-Durbin level 12 - 80sec
Cholesky 10-pass level 12 - 1500sec
compression difference 0.3%

A good thing to add in the future that might give similar compression results would be to adaptively select or brute-force search for an optimal window function to go along with the Levinson-Durbin recursion. I think Josh might be adding something like that to the reference encoder, but I'm not sure. I know it currently is user-selectable, but it would be great if the best window was chosen by the encoder for each frame. FFmpeg currently uses only a Welch window function. I just chose that because it is a simple calculation and seems to work well most of the time.

-Justin

[SebastianG]

• optional alternative LPC calculation using Cholesky factorization instead of Levinson-Durbin recursion

Heh! This is actually what I was suggesting during the "best window search for FLAC" period.
Too bad it's that slow.

Sebastian

[jcoalson]

I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.

Josh

(edit: lower bound on the least-squares solution I mean; I have some other experiments solving the system another way but that's too new.)

This post has been edited by jcoalson: Jul 24 2006, 07:06

[SebastianG]

I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.

I suppose the pseudo-inverse approach (through SVD) is even slower (but more accurate) than the Cholesky approach.

[TBeck]

I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.

Can you please tell which compression improvement can be achieved?