New FLAC encoder Options

[guruboolez]

Thank you very much Wombat

May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?

[Josef Pohm]

Exactly how you predicted. -10 still is playable. From -11 on it stutters.

To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.

Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).
Flake -11 and -12 uses 32, so it requires much more decoding power.

I hope that's correct.

[jcoalson]

I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.

most of the compression advantage of flake over FLAC 1.1.2 has already been incorporated into FLAC 1.1.3. the extra compression from 32-order LPC is minimal.

Josh

[wisodev]

Flake (Win32) from SVN revision 104 available:

Download from my Website

or use direct links:

Download binarys:
flake-svn-r104-win32-bin.zip

Download sources:
flake-svn-r104-win32-src.zip

NOTE: I have added my small patch to code (quality or speed is not affected), can be found here, the sources are already patched (this patch comes from my local subversion repository and is not related with sf.net subversion).

wisodev

[fairway]

Exactly how you predicted. -10 still is playable. From -11 on it stutters.

To my little knoledge decoding speed of lossless codecs mostly depends on the maximum number of predictor order used.

Flake -5 uses 8 max predictor order, like Flac -5 (and Garf version of Flac).
Flake -8,-9 and -10 uses 12 max predictor order like Flac -8 (and Garf version of Flac).

Though seeking with a flake encoded file is still significantly slower than with a FLAC (reference encoder) encoded file (try it in winamp with the default flac plugin).

[Justin Ruggles]

Thank you very much Wombat

May I suggest to Justin that the first official release of flake should come with a small warning (inside the documentation and/or inside flake.exe with -h switch) about possible hardware players' issues with -11 & -12 setting?

Okay, that sounds like a good idea.

I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.

Cool. Based on this, and the hardware decoding issues, I'll go ahead and add a separate warning for any lpc order over 12 for any sample rate up to 48kHz. Once the official subset standard changes, I'll change Flake's warnings accordingly.

-Justin

[JWolf]

Ah, oh. I see a problem. Does -12 take that much processing power or isn´t it 100% compatible to flac 1.1.2 decoding anymore?
My Squeezebox can´t play flake -12 on all titles. It stutters around. Flake -5 for example is fine.

Edit:
The compile Garf offered a while back works flawlessly on the Squeezebox and has higher compression already as standard 1.1.2.

Using the current version of FLAKE, I was able to play -12 compressed files on my Rio Karma with no hiccups.

Jon

[jcoalson]

I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.

it is done.

the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12. all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).

http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset

Josh

[wisodev]

I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build) to 173.906 seconds (test r104 build) using -12 command-line option.

Download test binary:
flake_r104_test.zip

wisodev

[MedO]

I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build) to 173.906 seconds (test r104 build) using -12 command-line option.

Download test binary:
flake_r104_test.zip

wisodev

Your test compile is a bit faster than mine on my machine (Celeron M 1,4Ghz). I can't give any good values, though, because I already got several seconds difference when running my own build twice, so the measurement is not very reliable.

[Synthetic Soul]

I have builded a faster FLAKE binary from rev. 104 but it needs some testing.

On one of my test files encoding time has droped from 193.468 seconds (previous r104 build) to 173.906 seconds (test r104 build) using -12 command-line option.

Here's some results for my "FLAC" corpus:

CODE

Setting       R104    R104 Test   Diff
======================================
-10       286.726s     274.561s    96%
-11       375.687s     347.035s    92%
-12      1002.105s     914.239s    91%

Edit: Here's the figures for my MinGW compile:

CODE

Setting  MinGW R104   R104 Test   Diff
======================================
-10        394.015s    274.561s    70%
-11        486.764s    347.035s    71%
-12       1322.178s    914.239s    69%

I don't think I'll bother anymore! Excellent work wisodev.

NB: Timings were recorded using TIMER, and CPU-only times are reported.

This post has been edited by Synthetic Soul: Oct 7 2006, 20:32

[Justin Ruggles]

I think I am going to officially limit the subset to 12-order LPC for 44.1kHz to head off problems like this. with FLAC at that sample rate, using high orders like that is usually a poor tradeoff and letting users do this without a warning is just going to create confusion. this goes for flac too which currently allows -l 32 without a warning.

it is done.

the new limits: if the sample rate is <=48kHz, max blocksize = 4608 and max LPC order = 12. all the tests I did showed that 32-order LPC got <0.1% more compression (usually a lot less).

http://flac.cvs.sourceforge.net/*checkout*...mat.html#subset

Josh

Splendid. I have modified the Flake warning message output and the return code of the flake_validate_params function to reflect the new standard.

[iGold]

I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:

Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)

If I try to compile this test code:

CODE

int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;

I get two "integer overflow" warnings.

I think int is too small here (in flake.c at line 447).

[Justin Ruggles]

I checkout rev.107 from SVN and compile it myself under x86_64 Ubuntu. While encoding disc image I see:

Signed 16-bit 44100 Hz stereo
samples: 116712120 (0m16.961s)

If I try to compile this test code:

CODE

int th, tm, ts, tms;
tms = 116712120 * 1000 / 44100;

I get two "integer overflow" warnings.

I think int is too small here (in flake.c at line 447).

Thanks. Fixed.
Also, anyone can submit bug reports on Flake's Sourceforge project page. Getting bug reports via HA makes me feel a little guilty for taking up space with small fixes. They are appreciated either way though.

[wisodev]

Flake (Win32) from SVN revision 108 available:

Download from my Website

or use direct links:

Download binarys:
flake-svn-r108-win32-bin.zip

Download sources:
flake-svn-r108-win32-src.zip

wisodev

[Justin Ruggles]

although i ran into a couple of problems compiling.

Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?

optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.

in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];

thanks,
will

Both MSVC issues have now been fixed.

Also, I am almost done with a new website for Flake. The only thing left is API documentation and benchmarks. I am putting in a link to Synthetic Soul's lossless benchmarks, but it would also be nice to have more extensive tests to better illustrate the differences between the compression levels. I'm not so worried anymore about detailed comparisons to the reference encoder because Synthetic Soul's tests give a good enough overview. Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels. Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful. Corpus totals would be better than file-by-file results.

Thanks!
Justin

[Will Fisher]

Both MSVC issues have now been fixed.

Excellent, thanks

I have included my enc_flac plugin (sources are attached somewhere in this thread =) built with wisodevs libFlake r108 to the beta branch of Winamp.

If it gets a good response from our beta team and QA department (and I can't see why it wouldn't) maybe we'll put it in a subsequent version of winamp.

[Synthetic Soul]

Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels. Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful. Corpus totals would be better than file-by-file results.

Has anyone provided this yet?

I don't mind running the tests if no-one else will. My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.

[wisodev]

Since my computer is quite slow, it would really help me out if someone could benchmark the current SVN version of Flake using the rw corpus at all compression levels. Also, adding in tests for "-8 -v 1", "-12 -v 1", and "-99" would be useful. Corpus totals would be better than file-by-file results.

Has anyone provided this yet?

I don't mind running the tests if no-one else will. My machine isn't hugely fast (Athlon XP 2400+, 512MB RAM, Win2K) but it's OK, and I will need a compile from wisodev as I have proved already that my compiles are far inferior to his.

I will provide fresh compile in few hours including new experimental build using P3 optimizations (using only compiler options - not assembly optimized!!!) witch seems to be even faster then the last one.

This post has been edited by wisodev: Oct 17 2006, 17:48

[wisodev]

Flake (Win32) from SVN revision 110 available:

Download from my Website

or use direct links:

Download binarys:
flake-svn-r110-win32-bin.zip

Download sources:
flake-svn-r110-win32-src.zip

wisodev

This post has been edited by wisodev: Nov 29 2006, 17:41

[Synthetic Soul]

Thanks wisodev.

I have an AMD XP CPU. Should I be looking at /org/flake.exe, /p3/flake.exe or /flake.exe?

[MedO]

Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...

Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.

This post has been edited by MedO: Oct 17 2006, 22:16

[PrakashP]

@wisodev

I would be interested what compiler flags you use (and which compiler). Have you added asm to the sources?

[MedO]

Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)

CODE

Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)

I thought this was solved?

[Justin Ruggles]

Also, I still get this time bug when encoding with wisodev's latest build (all binaries show this)

CODE

Signed 16-bit 44100 Hz stereo
samples: 13265868 (0m8.638s)

I thought this was solved?

I did too...sorry. I just committed a 2nd attempt, which seems to work for me. I just tested it on a 11 million sample wav and it correctly reported around 4 minutes.
-Justin

Something interesting here, I noticed that -5 often gives better results than -6, in fact, the size is often similar to that of -7 and -8. Fiddling with the settings a bit, I noticed that this is because of the setting "order method: estimate" which seems to produce better results than "2-Level" and sometimes better than "4-Level" as well.

I didn't test on many files, so this might be a coincidence, but on the few files I tested the commandline "flake -8 -m 1" was always faster and always produced bitrates similar or better than "flake -8". Might be worth investigating...

Edit: I just did a short test with the album "Supertramp - Some Things Never Change", comparing the commandlines "-8" and "-8 -m 1".
"-8 -m 1" was of course always way faster, and compressed 8 of the 12 files better than "-8". The differences in bitrate were quite small though, around 0.1-0.2kbps average.

This is really interesting. My tests resulted in not quite as good compression. It could be coincedence. More testing would be great. The order estimate method is a pretty crude 1st attempt at an idea I got from a paper which dealt with speech compression. It suggested a method using the highest order reflection coefficient above a certain threshold. I've found that the method varies widely depending on the type of audio...I imagine it works well for speech, but would need some tweaking for other types of signals. I had once tried to do some testing using gnuplot output, but my statistical analysis skills aren't really advanced enough to know what to look for...so I have left it at a simple threshold for now.

edit: I also want to note that I could almost always pick out the best order or close to it using visual output from gnuplot, but I could never translate that into a mathematical analysis that worked well. There is definitely potential there though.

-Justin

This post has been edited by Justin Ruggles: Oct 18 2006, 02:53