New FLAC encoder Options

[Justin Ruggles]

Well, I've gotten a lot done today. I just applied code to SVN for variable block size. You can test it out using "-v 1" on the commandline. It has much less of an adverse effect on speed than I had anticipated, which is great. Hopefully in the future I'll be able to enhance the algorithm for the block splitting decision. This works pretty well for a first attempt though.

One down-side is that the resulting files are not within the FLAC Subset format. They do, however, decode just fine with the libFLAC and FFmpeg decoders. Hardware support may be sporatic though.

I'm not going to add anything big for a few days and just focus on bug fixes and small changes. Then I'll release a new version.

-Justin

[Synthetic Soul]

Compiles using current SVN (rev 42): cygwin version | mingw version

Sorry if i should be asking for too much. You allready have done so much to help me!

Hey, no worries Thomas. I am doing this for my own interest, although of course I'm hoping it will benefit you (and Justin and Josh perhaps).

A test of V0.12 isn't urgent for me, but if it would be easier for you to perform both tests simultaneously, i could work a bit faster.

No, that's fine Thomas, I wouldn't want to rush you. It doesn't make too much difference to me if I perform the tests at different times. Thanks anyway.

Well, I've gotten a lot done today. I just applied code to SVN for variable block size. You can test it out using "-v 1" on the commandline. It has much less of an adverse effect on speed than I had anticipated, which is great.
...
I'm not going to add anything big for a few days and just focus on bug fixes and small changes. Then I'll release a new version.

Cool, sounds interesting. Once I have wisodev's build I'll set some scripts running.

[bukem]

@Justin:
I'll check new build on my iAudio U3 this evening. I'm just curious if it's able to handle variable block sized flac files.

Edit: Huston we have a problem! I've just downloaded Synth's compile and -v switch IS NOT SUPPORTED

CODE

D:\documents\music\.work>flake -p 256 -12 -v 1 "01. autechre - kalpol introl.wav" "01. autechre - kalpol introl.flac"

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

invalid option: -v
usage: flake [options] <input.wav> <output.flac>
type 'flake -h' for more details.

Edit2: I've checked the SVN (v42) and found -v switch implemented in flake.c - seems that there's something wrong with Synth's compile...

This post has been edited by bukem: Sep 22 2006, 15:05

[bukem]

OK, I've managed to compile flake v42 by myself and to do a quick test with variable block sized flac on my iAudio U3 - and I'm simply astonished because it just WORKS! There is small glitch though, the player doesn't display remaining play time of the track anymore but that's not a problem for me.

FYI: I've converted Thunderball's album "Scorpio Rising" with flake -p 256 -12 -v 1. I just love my iAudio U3

This post has been edited by bukem: Sep 22 2006, 14:21

[Synthetic Soul]

Edit2: I've checked the SVN (v42) and found -v switch implemented in flake.c - seems that there's something wrong Synth's compile...

How bloody annoying! Only time they may have been of use and they weren't right! I have recompiled, checked the -v switch, and uploaded, just in case. I have no idea what happend this morning, I'm sure some files had updated...

Thanks for highlighting the error.

Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions. Am I missing something?

E.g.:

CODE

C:\C\Flake\SVN\flake>flake-svn-cygwin -5 "C:\Documents and Settings\Neil\My Docu
ments\Audio\Flac Corpus\source\20.wav" 5_no.flac

Flake: FLAC audio encoder
© 2006 Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 2743608

block size: 4608
variable: no
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096

progress: 100% | ratio: 0.413 | bitrate: 583.2 kbps | bytes: 4525937


C:\C\Flake\SVN\flake>flake-svn-cygwin -5 -v 1 "C:\Documents and Settings\Neil\My
Documents\Audio\Flac Corpus\source\20.wav" 5_yes.flac

Flake: FLAC audio encoder
© 2006 Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 2743608

** WARNING! FLAC file will not be Subset compliant **

block size: 4608
variable: yes
prediction type: levinson-durbin
prediction order: 1,8
partition order: 0,6
order method: estimate
stereo method: mid-side
header padding: 4096

progress: 100% | ratio: 0.414 | bitrate: 583.5 kbps | bytes: 4528149

This post has been edited by Synthetic Soul: Sep 22 2006, 14:49

[Justin Ruggles]

Edit: OK, I've just done some (after the horse has bolted) testing with -v 1 at -0, -5, -8 and -12 and in all cases the -v versions are bigger than the fixed block size versions. Am I missing something?

Hmmm. It worked for my samples. I'll do more testing tonight. I may temporarily add a variable frame-split-threshold until I can figure things out.

Thanks for the feedback,
-Justin

[bukem]

In my case I saved 1 Mbyte of 319 Mbytes flac album when converted with -v 1 switch.

[Synthetic Soul]

Justin, my post was a bit premature. I was just a little surprised at the (minor) increase in size over various compression levels, so posted without really thinking.

What I totaly forgot was that this was just one file.

To rectify I have just tested -5 on my "FLAC corpus" (28 files) and the results are a lot better.

Out of the 28 files 12 files suffer with the -v switch, but on the whole the compression improves from 59.843% to 59.754%, or 1,221,384 bytes in 1,371,283,088 (a 0.001% improvement).

As you can see from the figures, those that suffer from variable blocks (-v 1) are the files at the outer edges - generally those that compress the best (e.g.: Charlotte Church, Andrea Bocelli), but also two Motorhead tracks (-274 and -5,698). This variation reminds me of the window testing I was involved in for FLAC 1.1.2_CVS (1.1.3), with the two bounds acting differently than the core files... not sure if that's relevant.

When wisodev is kind enough to provide a compile of Rev. 42 I'll soon provide some more figures, using my Yalac corpus, including timings and using various "key" compression levels.

CODE

Wav            Off                    On
               Filesize     Comp %    Filesize     Comp %       Diff
====================================================================
  44205884     30037774    67.950%    29996667    67.857%     41,107
  40501484     18854486    46.553%    18860045    46.566%     -5,559
  41860940     32194824    76.909%    32179874    76.873%     14,950
  36997004     11961940    32.332%    11965877    32.343%     -3,937
  41849180     22570842    53.934%    22499530    53.763%     71,312
  43095740     31742309    73.655%    31646473    73.433%     95,836
  98031404     52089827    53.136%    52103883    53.150%    -14,056
  29964524     22412071    74.795%    22412345    74.796%       -274
  44963228     18196142    40.469%    18207104    40.493%    -10,962
  37462700     25086008    66.963%    24958122    66.621%    127,886
  35209484     24646149    69.999%    24626419    69.943%     19,730
  39280796     26879523    68.429%    26836293    68.319%     43,230
  35898620     24982661    69.592%    24917519    69.411%     65,142
  44295260     30913592    69.790%    30808634    69.553%    104,958
.134433308     84608716    62.937%    84103316    62.561%    505,400
  46214492     33193196    71.824%    33150151    71.731%     43,045
  70752908     31462354    44.468%    31470864    44.480%     -8,510
  37989548     22986231    60.507%    22940337    60.386%     45,894
  56989004     29822007    52.329%    29827193    52.339%     -5,186
  47973788     28413887    59.228%    28367289    59.131%     46,598
  10974476      4525937    41.241%     4528149    41.261%     -2,212
  45400700     17135222    37.742%    17141711    37.756%     -6,489
  79469420     56785940    71.456%    56754994    71.417%     30,946
  35719868     26702556    74.755%    26675013    74.678%     27,543
  32309468     13942750    43.154%    13947729    43.169%     -4,979
  73810508     30283837    41.029%    30312140    41.068%    -28,303
  55389644     44922899    81.103%    44928597    81.114%     -5,698
  30239708     23258281    76.913%    23224309    76.801%     33,972
====================================================================
1371283088    820611961    59.843%   819390577    59.754%    1221384

This post has been edited by Synthetic Soul: Sep 22 2006, 17:41

[wisodev]

FLAKE SVN REVISION 42

Download binarys:
flake-svn-r42-win32-bin.zip

Download sources:
flake-svn-r42-win32-src.zip

wiso

[jcoalson]

I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications. there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change). this is not a problem when in other containers e.g. Ogg FLAC.

definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.

Josh

[TBeck]

I tested wisodev's build on my two primary file sets as before flake V0.10.

Participiants

Yalac V0.11a slightly improved over V0.11.
FLAC 1.1.2
Flake 0.10 file flake-0.10-win32-i686.zip from sourceforge.
Flake sv42 flake-svn-r42-win32-bin.zip from wisodev.

Results

CODE

          Yalac V0.11a |  FLAC  |  Flake 0.10   |  Flake sv42   |
          Turbo  Fast  |  -8    |  -8     -12   |  -8     -12   |
-----------------------+--------+---------------+---------------+
rw                     |        |               |               |
Ratio:    58.03  57.14 |  59.54 |  59.06  58.74 |  58.88  58.51 |
-----------------------+--------+---------------+---------------+
songs                  |        |               |               |
Ratio:    49.09  48.37 |  51.35 |  50.27  49.59 |  50.27  49.58 |
-----------------------+--------+---------------+---------------+

Ratio is the compression ratio in percent. No speed data this time, because i am not sure, if both flake versions have been bulit with the same compiler optimizations. Subjectively Flake sv42 with -v 1 seems not to be significantly slower than V0.10.

For the test sets:

"rw" contains 46 files from http://www.rarewares.org/test_samples/. Should be the whole dir. I call this my primary test set and prefer to use it for my tuning.

"songs" contains CD-Ripps of the beginning of some songs. This set contains many files that are benefiting from higher predictor orders. It may be less representative than set rw, but i keep it because it has been my primary set for many years.

Some remarks

Up to 0.23 percent better compression for sample set "rw", only 0.01 better compression for "songs".

Not too surprising for me: "rw" contains many samples with very fast changes of the signal characteristics, where variable block sizes should help most.

Possibly Justin could try to use some of the "rw" files for further optimizations of the block length switching. I found them very useful for this.

"songs" contains files, which can quite well be encoded with constant frame sizes of up to 200 ms.

Keep on the great work!

Thomas

This post has been edited by TBeck: Sep 22 2006, 19:48

[TBeck]

Here the results of set "rw" for each file:

CODE

                     |  FLAC  |  Flake 0.10   | Flake sv42 -v1|
                     |  -8    |  -8     -12   |  -8     -12   |
---------------------+--------+---------------+---------------+
41_30sec.flac          65.56     65.77  65.66    65.33  65.20
ATrain.flac            51.58     50.08  49.66    49.87  49.34
Bachpsichord.flac      64.67     64.20  63.33    63.99  63.08
Bartok_strings2.flac   53.29     51.85  51.16    51.94  51.14
BeautySlept.flac       63.37     63.35  62.81    63.25  62.68
BigYellow.flac         75.40     75.12  74.94    74.81  74.61
Blackwater.flac        60.64     59.43  59.30    59.38  59.21
bodyheat.flac          59.79     59.32  58.86    59.51  58.95
chanchan.flac          64.35     64.08  64.01    63.77  63.69
DaFunk.flac            66.87     66.87  66.79    66.13  66.00
death2.flac            40.31     41.52  41.03    38.94  38.58
Debussy.flac           28.54     27.95  27.83    28.01  27.84
EnolaGay.flac          65.48     65.17  63.54    65.40  63.80
experiencia.flac       71.82     71.65  71.47    71.66  71.46
female_speech.flac     34.15     34.47  34.08    34.06  33.44
FloorEssence.flac      64.68     64.23  64.13    64.12  63.99
getiton.flac           53.79     53.15  53.02    52.70  52.56
gone.flac              65.67     65.12  65.06    65.09  65.01
Hongroise.flac         40.84     36.65  36.45    36.53  36.31
Illinois.flac          59.58     59.19  58.86    59.23  58.84
ItCouldBeSweet.flac    56.86     56.65  56.46    56.95  56.65
kraftwerk.flac         46.50     46.82  46.34    45.84  45.40
Layla.flac             65.93     65.59  65.21    65.18  64.80
Leahy.flac             73.84     73.82  73.75    73.76  73.67
LifeShatters.flac      70.58     69.95  69.78    70.06  69.83
macabre.flac           63.27     62.80  62.02    62.89  62.04
Mahler.flac            49.86     48.91  48.86    48.85  48.80
male_speech.flac       36.28     36.36  36.08    35.94  35.57
Mama.flac              66.85     66.47  66.43    66.47  66.41
MidnightVoyage.flac    62.40     61.92  61.23    61.91  61.13
mybloodrusts.flac      70.14     70.01  69.94    70.08  69.97
NewYorkCity.flac       78.30     78.33  78.14    78.36  78.18
OrdinaryWorld.flac     67.66     67.22  66.59    66.77  66.16
Polonaise.flac         36.50     33.94  33.46    34.05  33.46
Quizas.flac            60.78     60.66  60.55    59.91  59.77
riteofspring.flac      51.95     53.39  53.33    53.32  53.23
rosemary.flac          55.77     55.31  55.16    55.19  55.01
Scars.flac             59.81     59.16  58.62    59.06  58.53
SinceAlways.flac       71.10     70.69  70.64    70.28  70.21
thear1.flac            74.01     73.64  73.61    73.64  73.62
TheSource.flac         57.28     57.05  56.88    56.61  56.43
TomsDiner.flac         53.73     53.81  52.96    53.85  52.82
trust.flac             61.49     60.71  60.64    60.70  60.59
Twelve.flac            72.16     71.85  71.55    71.96  71.53
velvet.flac            68.12     68.12  67.70    67.70  67.48
Waiting.flac           67.67     67.18  66.55    67.19  66.46

[guruboolez]

CODE

                     |  FLAC  |  Flake 0.10   | Flake sv42 -v1|
                     |  -8    |  -8     -12   |  -8     -12   |
---------------------+--------+---------------+---------------+
...
riteofspring.flac      51.95     53.39  53.33    53.32  53.23
...

I guess that it can't be a typo, but could you confirm these values?

[TBeck]

CODE

                     |  FLAC  |  Flake 0.10   | Flake sv42 -v1|
                     |  -8    |  -8     -12   |  -8     -12   |
---------------------+--------+---------------+---------------+
...
riteofspring.flac      51.95     53.39  53.33    53.32  53.23
...

I guess that it can't be a typo, but could you confirm these values?

Yes, they are right. This file is one of the very rare cases, where a 16 bit file (or a part of it) contains a constant least significant bit, which can be removed by Flac's "Look for wasted bits" option. Probably in Flake this option is not implemented.

[guruboolez]

Confirmation... and explanation! Thanks

N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

[TBeck]

N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.

"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.

"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:

CODE

  decimal  binary
    9        1001
   18       10010

This post has been edited by TBeck: Sep 22 2006, 22:23

[Will Fisher]

new winamp encoder plugin as per wizos drop: http://stashbox.org/uploads/1158969212/enc_flac.dll

although i ran into a couple of problems compiling.

Would it be too much of a problem to put my changes into the code, to keep it portable across compilers?

optimize.c line 219 replaced with: uint32_t bits[32]; (where 32 used to be levels. MSVC doesn't like non-constant sizes for these things.) I am sure that 32 is as large as levels gets.

in encode.c flake_set_defaults, msvc doen't like this syntax:
foo = ((int[]){1,2,3})[2];
so i defined all these "nameless" arrays as ones with names, so it becomes:
int sel[] = {1,2,3};
foo = sel[2];

thanks,
will

[Justin Ruggles]

I would caution against encoding variable-blocksize files in native FLAC unless familiar with the implications. there is a design flaw in the native FLAC container that can make variable blocksize files difficult to handle in some ways (I may fix this later but it's a format change). this is not a problem when in other containers e.g. Ogg FLAC.

definitely I would say that a fraction of a percent size decrease is not worth the useability tradeoff.

Josh

I agree. The variable block size headers are a bit tricky. I don't have any issue with using it for my personal files because I know the players I use support it. If I was sharing the files with other people I would definitely not use that option though. Perhaps I should change the warning to something less technical like "Warning! The chosen options are not part of the FLAC Subset format and therefore may not be compatible with some FLAC players."

N.B. What does that exactly mean? That the 16th bit is always the same through the whole file? In other words 15 bit recording pad to 16 bit?

I have not performed a deeper analysis of this file. Possibly one channel has a constant least significant bit (LSB), because compression can be improved by about 3 percent by the wasted bits option. If both channels were affected, it would be about 6 percent.

"That the 16th bit is always the same through the whole file?" Through the whole file or through a whole frame (my notation) or block (Flac's notation). Usually the analysis is beeing performed on a frame basis (implementaion dependend). And theoretically more than 1 bit can be constant.

"In other words 15 bit recording pad to 16 bit?" If someone multiplies each sample with 2 (or shifts it by one place), the least significant bit will become zero:

CODE

  decimal  binary
    9        1001
   18       10010

I think I may have removed that feature because I could not find any files which had any wasted bits...I can't really remember. I at least know I had implemented it at one point, but it's definitely no longer there. It would be easy enough to add.

As far as the compression gain for variable block size, I have a feeling I can do better. This was a first attempt and pretty much a shot-in-the-dark. What I think I'll do next is a really slow brute-force approach so that I will have something to compare results against.

Thanks for all the great results. It would be interesting to see some speed comparisons. I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).

-Justin

[TBeck]

Thanks for all the great results. It would be interesting to see some speed comparisons. I know that the current SVN version is quite a bit faster on my machine than 0.10, but my machine is certainly not typical (Linux, AMD K6-2).

Same as before with encoding times.

Participiants

FLAC 1.1.2
Flake 0.10 file flake-0.10-win32-i686.zip from sourceforge.
Flake sv42 flake-svn-r42-win32-bin.zip from wisodev.

Test system

P3-800 with a (slow) 40 GB HD.

CODE

            FLAC  |  Flake 0.10   | Flake sv42 -v1|
            -8    |  -8     -12   |  -8     -12   |
------------------+---------------+---------------+
rw                |               |               |
Ratio:      59.54 |  59.06  58.74 |  58.88  58.51 |
EncoTime:  280.47 | 133.57 871.84 |  79.03 362.25 |
------------------+---------------+---------------+
songs             |               |               |
Ratio:      51.35 |  50.27  49.59 |  50.27  49.58 |
EncoTime:  185.51 |  86.14 591.69 |  50.76 241.82 |
------------------+---------------+---------------+

Can -12 in combination with -v1 be so much faster now?

[Justin Ruggles]

Can -12 in combination with -v1 be so much faster now?

Yes indeed. Level 12 makes more use of 2 functions which were optimized since version 0.10. The different binaries might be making some of that difference as well.

BTW...where might I find the "rw" corpus?

-Justin

[Justin Ruggles]

Hi,
Flake r46 has a new variable block size algorithm, which is selected by using "-v 2". It gives a lower-bound for even 3-level splitting (splitting frames exactly in half up to 3 times). Encoding takes about 4x as long. Hopefully I will be able to use this to help improve -v 1 by comparing the resulting files. Future developments might also include uneven splitting, which could better isolate changes within a frame with less overhead.
-Justin

Edit: In r47 I modified -v 1 to give better compression.

This post has been edited by Justin Ruggles: Sep 23 2006, 08:13

[guruboolez]

Justin Ruggles> BTW...where might I find the "rw" corpus?

here.

[Synthetic Soul]

Compiles using current SVN (rev 48): cygwin version | mingw version


I have added Flake SVN Rev.42 (wisodev build) to my Yalac comparison. Please see this post on the Yalac thread for info.

Here's the FLAC-related results in compression order:

CODE

Encoder Setting    Compression    Encode    Decode
==================================================
Flake -12 -v 1         65.259%     7.29x    70.41x
Flake -12              65.368%     7.38x    69.31x
Flake -11              65.376%    19.10x    69.19x
FLAC 1.1.2_CVS -9      65.528%     5.73x    71.34x
Flake -8 -v 1          65.530%    37.94x    74.49x
Flake -10              65.555%    24.82x    73.63x
Flake -9               65.559%    35.48x    75.07x
FLAC 1.1.2_CVS -8      65.611%     9.25x    70.96x
Flake -8               65.658%    41.21x    74.93x
Flake -5 -v 1          65.668%    64.70x    74.89x
Flake -5               65.798%    69.44x    75.44x
FLAC 1.1.2_CVS -5      65.916%    36.75x    71.46x
FLAC -8                66.028%     9.43x    71.42x
FLAC -5                66.279%    38.44x    72.05x
FLAC 1.1.2_CVS -0      70.734%    69.34x    72.35x
FLAC -0                70.734%    68.97x    75.84x
Flake -0               71.679%    82.18x    71.13x
Flake -0 -v 1          71.728%    76.70x    72.80x

[wisodev]

FLAKE SVN REVISION 48

Download binarys:
flake-svn-r48-win32-bin.zip

Download sources:
flake-svn-r48-win32-src.zip

wiso

[krmathis]

Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:

$ ./configure
source path /Users/krmathis/Downloads/flake-enc
C compiler cc
make make
CPU powerpc (generic)
big-endian yes
inttypes.h yes
AltiVec enabled yes
debug symbols yes
strip symbols yes
optimize yes
Creating config.mak and config.h...
$ make
make -C libflake all

** Snip **

cc -O3 -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2

I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")

Any idea whats wrong?

B.U.M.P.
Revision 48 still fails with the same error...