flac-1.1.4-alpha1 Options

[jcoalson]

I've uploaded an alpha of flac-1.1.4 here. normally I don't do alphas but the bitbuffer is new so I'd like to have people put it through the ringer. also, this is a debug build with a lot of extra assertions and verify mode is hardcoded on, so it will not be as fast as the real thing. I promise to have a realistic-speed beta out as soon as the alpha holds up to your scrutiny.

flac-1.1.4 has compression improvements and both decoder and encoder speedups, again with no change to the format. with this alpha you can probably expect 0.3-0.7% improvement depending on material. because of the speedups I may dial up the compression even a little more for -3..-8 in the beta.

the speedups are more dramatic even though they're not evident from the debug build. flac -8 encoding is 2x faster, -5 probably 20% faster. decoding is also 15-20% faster on x86, but probably more on non-x86 because the decoding speedup comes from a completely rewritten bitbuffer. the old bitbuffer operated on bytes (which is still relatively fast on x86), but the new one uses the native machine word size, which should be much faster on chips whose ISAs have slow byte ops.

I humbly request you all to try out on what ever corpus you have and let me know in this thread if you run into any bugs. I'd also be interested in the compression ratio difference with 1.1.3, and if you're really feeling generous, try the same test but with a blocksize of 4096, i.e. add "-b 4096" to the end of your command-line options and report the ratio for that. I have a feeling that it might be a better default than 4608.

thanks in advance!

P.S. some other bugs are also fixed, like the -A locale bug, flac-to-flac with --picture, and other flac-to-flac transcoding bugs.

Josh

[tempnegro]

lets start to get up to TAK's standards and give it a run for its money

[WaldoMonster]

Nice work Josh,

One small problem: msvcrt.dll missing.
Can be downloaded from: http://www.dll-files.com/dllindex/dll-files.shtml?msvcrt
I hope this dll is not needed in the next alpha/beta version.

[jorsol]

Actually it ask me for msvcrtd.dll, the debug version of msvcrt.dll I gess...

I have that dll because I copy it from one program that use it.... right now testing...

[jcoalson]

ugh, sorry about that. I reuploaded another version that should not need that debug dll: http://www.hydrogenaudio.org/forums/index....ost&id=2839

Josh

[Martin H]

Thank you so much Josh for this new release It's very much appreciated

I have tested the new v1.1.4a1 and compared the compression ratio to that of v1.1.3. I have tested with 7 FLAC images with embedded cuesheets and EAC logs(consisting of 107 tracks in all) and all files completed encoding succesfully without any verification errors

All files was encoded with the default compression mode(-5).

Results per album :

319.159.124 Evanescence - Fallen.flac : flac.exe v1.1.3 -A "tukey(0,5)"
318.368.604 Evanescence - Fallen.flac : flac.exe v1.1.4a1
318.297.383 Evanescence - Fallen.flac : flac.exe v1.1.4a1 -b 4096

496.037.106 Korn - Follow The Leader.flac : flac.exe v1.1.3 -A "tukey(0,5)"
495.438.737 Korn - Follow The Leader.flac : flac.exe v1.1.4a1
495.129.913 Korn - Follow The Leader.flac : flac.exe v1.1.4a1 -b 4096

380.228.515 Korn - Issues.flac : flac.exe v1.1.3 -A "tukey(0,5)"
379.534.482 Korn - Issues.flac : flac.exe v1.1.4a1
379.396.875 Korn - Issues.flac : flac.exe v1.1.4a1 -b 4096

443.456.905 Korn - Korn.flac : flac.exe v1.1.3 -A "tukey(0,5)"
442.814.446 Korn - Korn.flac : flac.exe v1.1.4a1
442.425.336 Korn - Korn.flac : flac.exe v1.1.4a1 -b 4096

339.413.021 Korn - Life Is Peachy.flac : flac.exe v1.1.3 -A "tukey(0,5)"
339.197.838 Korn - Life Is Peachy.flac : flac.exe v1.1.4a1
338.972.245 Korn - Life Is Peachy.flac : flac.exe v1.1.4a1 -b 4096

451.636.419 Korn - Untouchables.flac : flac.exe v1.1.3 -A "tukey(0,5)"
450.177.434 Korn - Untouchables.flac : flac.exe v1.1.4a1
449.876.917 Korn - Untouchables.flac : flac.exe v1.1.4a1 -b 4096

342.658.608 Nirvana - Unplugged in New York.flac : flac.exe v1.1.3 -A "tukey(0,5)"
341.967.540 Nirvana - Unplugged in New York.flac : flac.exe v1.1.4a1
341.731.122 Nirvana - Unplugged in New York.flac : flac.exe v1.1.4a1 -b 4096

Total results for all 7 FLAC images :

2.772.589.698 : flac.exe v1.1.3 -A "tukey(0,5)"
2.767.499.081 : flac.exe v1.1.4a1
2.765.829.791 : flac.exe v1.1.4a1 -b 4096

Again, many thank's Josh for all your efforts

This post has been edited by Martin H: Feb 5 2007, 18:14

[...Just Elliott]

do you ever sleep?

[DrGreen]

Thanks for the release (specially the flac-to-flac bug with --picture)

Could you indicate the cvs revision of this alpha, so I can compile it in linux?

[gib]

Despite the hour here, I was too curious to not run some tests. I tested 6 CD images that were encoded with FLAC 1.1.3 with the -6 setting. For consistency, I also used the -6 setting for 1.1.4a.

The overall numbers:

CODE

FLAC 1.1.3 -6:           2,115,187,542
FLAC 1.1.4a -6:          2,108,233,277
FLAC 1.1.4a -6 -b 4096:  2,106,823,342

The per image results are very similar to Martin H's; using -b 4096 improved the compression in every case, so I didn't bother to post them all separately. That being said, there was one outlier in the tests. The album "Wintersong" by Sarah McLachlan saw a compression improvement of about 1.3%. It is quieter than the other albums I tested, so that probably explains it. Still, it was nice to see so I thought I'd mention it.

[jorsol]

These are my results... five wav images... without cuesheet or tags...

CODE

Original  -----------------------> 2,826,435 KB --- 2,760 MB
FLAC 1.1.3 -8 -------------------> 1,741,130 KB --- 1,700 MB
FLAC 1.1.3 -8 -b 4096 -----------> 1,739,723 KB --- 1,698 MB
FLAC 1.1.4 -8 -------------------> 1,730,959 KB --- 1,690 MB
FLAC 1.1.4 -8 -b 4096 -----------> 1,729,470 KB --- 1,688 MB

These are nice results... that means that FLAC 1.1.4 compress better...

Thanks Josh you are great...

[DrGreen]

ok, I got the cvs head and tested with a few image files

The order is
flac 1.1.3 -V -5
flac 1.1.4a1 -V -5
flac 1.1.4a1 -V -5 -b 4096

CODE

BobbyTimmons-TheSoulMan
256143018
255662303 (-0.19%)
255549289 (-0.23%)

DexterGordon-Generation
265700793
265332929 (-0.14%)
265112694 (-0.22%)

JackieMcLean-Action
244021606
243062676 (-0.39%)
242966380 (-0.43%)

KeithJarrett-ViennaConcert
230301326
223857095 (-2.8%)
223878287 (-2.79%)

LeeMorgan-Expoobident
340513941
340217464 (-0.09%)
340154456 (-0.11%)

KeithJarrett-ViennaConcert is piano only for the whole album, maybe thats why it compressed much better than the others, although it was the only one that got bigger with -b 4096

[jcoalson]

thanks for the feedback so far everyone, keep it comin'!

the CVS tag you can use if you want to build yourself is FLAC_RELEASE_1_1_4_ALPHA1__2007_02_04 (currently the same as CVS HEAD).

Josh

[aharden]

CODE

Porcupine Tree - Signify
406,889,575 bytes: FLAC 1.1.3 -8
405,970,072 bytes: FLAC 1.1.4a -8
405,708,893 bytes: FLAC 1.1.4a -8 -b 4096

Thanks, Josh.

[krabapple]

Cue1.wav (a live Genesis track I digitized from tape): 80,876 KB

Flac Frontend/Flac1.1.3 -8 -V: 40,605
EAC/Flac1.1.3 -8 -A tukey(0.25) -A gauss(0.1875) -b 4096 -V ("insane"): 40,391
Flac Frontend/Flac1.1.4a1 -8 -V: 39,178
EAC/Flac1.1.4a1 "insane" : 38,755 (and notably faster than 113 insane)

nice work.

This post has been edited by krabapple: Feb 5 2007, 19:55

[gaekwad2]

Steely Dan - Gaucho (album image)

CODE

1.1.3 -0 241.736.229
1.1.4a1 -0 241.736.236
1.1.3 -1 237.947.852
1.1.4a1 -1 237.935.135
1.1.3 -2 235.722.554
1.1.4a1 -2 235.718.085
1.1.3 -3 235.803.215
1.1.4a1 -3 234.016.453
1.1.3 -4 231.128.113
1.1.4a1 -4 229.242.692
1.1.3 -5 229.703.049
1.1.4a1 -5 227.810.948
1.1.3 -6 229.368.051
1.1.4a1 -6 227.471.096
1.1.3 -7 228.635.523
1.1.4a1 -7 226.875.449
1.1.3 -8 228.032.231
1.1.4a1 -8 225.811.793
original 404.477.996

-7 and -8 gave the following message at 90% (but continued encoding): FLAC__lpc_quantize_coefficients: negative shift=-1 order=8 cmax=2809,049316

This post has been edited by gaekwad2: Feb 5 2007, 20:40

[Justin Ruggles]

flac-1.1.4 has compression improvements and both decoder and encoder speedups, again with no change to the format. with this alpha you can probably expect 0.3-0.7% improvement depending on material. because of the speedups I may dial up the compression even a little more for -3..-8 in the beta.

This is great. Are the same improvements are already in FLAC SVN?
edit: sorry, you answered that already...

Also, I'm curious to know what settings you were able to tweak to get the compression gain? I could never figure out why Flake sometimes has better compression...I just coded each part of the spec based on what seemed intuitive and on a few of my own tests, which were fairly non-scientific and a bit hackish really.

If more improvements such as these keep coming, I'll probably slow down the Flake development. I made it mostly for fun and as a learning experience, but released it publicly when it turned out to be so fast. I much prefer a faster libFLAC.

Keep up the great work as always.

-Justin

This post has been edited by Justin Ruggles: Feb 5 2007, 23:32

[TBeck]

Participiants:

FLAC 1.1.3
FLAC 1.1.4 Alpha 1 - debug

CODE

            FLAC    FLAC    FLAC
            1.1.3   1.1.4   1.1.4
Preset      -8      -8      -8
Framesize   4608    4608    4096
----------------------------------
rw
Ratio:      59.15   58.97   58.91
----------------------------------
songs
Ratio:      50.64   50.28   50.23
----------------------------------
sh_2444
Ratio:      59.62   59.60   59.56
----------------------------------
sh_2496
Ratio:      56.23   56.16   56.13
----------------------------------

Ratio is the compression ratio in percent.

For the test sets:

"rw" contains 46 files from http://www.rarewares.org/test_samples/.
"songs" contains CD-Ripps of the beginning of the some songs.
"sh_2444" contains 5 files with 24 bit and 44 or 48 Khz.
"sh_2496" contains 6 files with 24 bit and 96 Khz.

Some remarks

- Nice improvements for the first two sets.

- A frame size of 4096 is preferable. That's also my experience from my earlier tests, when TAK wasl using fixed frame sizes.

- Several reports of negative shifts in coeff quantization. None of the 24/96 files compressed without 3 to 4 of this messages. While 1.1.4 already seems to be much faster than 1.1.3, files affected by negative shifts take very long to encode.

This looks very promising!

Thomas

[Zoom]

I ran a few tests also:

CODE

Size/Bytes Genre Ratio
______________________________________
ORIGINAL WAV FILES
______________________________________
497,822,012 Pop 100%
753,874,844 Country 100%
736,223,084 Classical 100%
443,222,684 Pop/Rock 100%
799,480,124 Metal 100%
______________________________________
FLAC 1.1.3 -8
______________________________________
323,240,382 Pop 64.93%
479,650,971 Country 63.62%
290,479,412 Classical 39.46%
296,422,386 Pop/Rock 66.88%
595,061,318 Metal 74.31%
______________________________________
FLAC 1.1.3 -8 -b 4096
______________________________________
323,032,660 Pop 64.89%
479,138,462 Country 63.56%
290,569,776 Classical 39.47%
296,176,736 Pop/Rock 66.82%
594,947,534 Metal 74.42%
______________________________________
FLAC 1.1.4 alpha1_debug -8
______________________________________
322,260,848 Pop 64.73%
478,567,645 Country 63.48%
287,905,316 Classical 39.11%
295,581,240 Pop/Rock 66.69%
594,101,619 Metal 74.31%
______________________________________
FLAC 1.1.4 alpha1_debug -8 -b 4096
______________________________________
322,054,360 Pop 64.69%
478,027,837 Country 63.41%
288,014,996 Classical 39.12%
295,330,557 Pop/Rock 66.63%
593,977,839 Metal 74.30%

Interesting to me that the only one that wasn't positively affected by the blocksize of 4096 was the Classical music. Anyway, results look promising. Glad that I haven't undertaken the 1.1.2 -> 1.1.3 yet, now I can wait and just use 1.1.4.

[jcoalson]

Also, I'm curious to know what settings you were able to tweak to get the compression gain? I could never figure out why Flake sometimes has better compression...I just coded each part of the spec based on what seemed intuitive and on a few of my own tests, which were fairly non-scientific and a bit hackish really.

pretty good for hackish! actually flac and flake are now pretty close compression-wise when comparing options apples-to-apples. the only real differences in flake now are 1) better lpc order searching; 2) faster estimation of stereo decorrelation method; 3) variable blocksize. because these are not in libFLAC yet, flake is still significantly faster.

I have a couple more compression ideas to try too but they're novel and I have to bone up on some math first, which I don't enough have time for right now.

some other comments about the results so far:

I've noticed too that sometimes -b 4096 is very slightly worse for classical but it seems overall to be a better default based on the average use case.

also, I forgot to mention to please report those debug outputs. Thomas, the negative shift messages are not a problem but I added them to try and gauge how frequently they occur in real data, so I can figure out how much effort to expend in properly handling them. as you have already figured out, FLAC does need some tweaks to do 24bit really well but I didn't want to hold 1.1.4 up for that.

Josh

[jorsol]

If I encode one file with long filename, flac show the percentage in a loop.. it don't fit in the line of the console and thats why it show the loop... the same thing with 1.1.3 BTW...

How long do you ask?:

Luis y Carlos Mejia Godoy - Lo Mejor De Luis y Carlos Mejia Godoy.wav

[A_Man_Eating_Duc...]

also, I forgot to mention to please report those debug outputs.

Is there any chance of getting a switch that will log these debug messages to a file, i tried to use the old > flacinfo.txt trick but it doesn't catch anything. i have a fair few albums i can run through for you.

@jorsol
I suggested the percent and ratio parts of the output to be put on a new line in the 1.1.3 flac thread but i was told to just increase the width of the CMD box.

EDIT: THIS is the thread i was talking about

This post has been edited by A_Man_Eating_Duck: Feb 6 2007, 07:20

[Synthetic Soul]

CODE

Setting |      FLAC 1.1.3         |      FLAC 1.1.4a1        |  FLAC 1.1.4a1 -b 4096
========+=========================+==========================+========================
        |   Comp %    Enc    Dec  |   Comp %    Enc     Dec  |   Comp %    Enc     Dec
========+=========================+==========================+========================
0       |  70.744%    92x   103x  |  70.744%    80x    130x  |  70.744%    63x    123x
5       |  65.926%    39x    94x  |  65.803%    36x    115x  |  65.759%    36x    114x
6       |  65.881%    35x    94x  |  65.760%    35x    116x  |  65.716%    35x    114x
7       |  65.790%    12x    94x  |  65.707%    20x    116x  |  65.668%    20x    114x
8       |  65.621%     9x    92x  |  65.498%    16x    112x  |  65.463%    16x    111x
8 -Ax2  |  65.538%     6x    92x  |  65.417%    11x    113x  |  65.393%    11x    111x

NB: "8 -Ax2" uses -A tukey(0.5) -A flattop.

I generally leave my scripts running unattended; however I did see some of the -b 4096 test running and saw numerous occurrences of the "negative shifts in coeff quantization" message while the 8 -Ax2 encode was running - on average twice per file (sometimes none, sometimes 4).

[TBeck]

also, I forgot to mention to please report those debug outputs. Thomas, the negative shift messages are not a problem but I added them to try and gauge how frequently they occur in real data, so I can figure out how much effort to expend in properly handling them. as you have already figured out, FLAC does need some

Fine. I repeated the test with -8 - b 4096 and tried to write down the occurences of the messages. I couldn't see the output of the last file, because my script is clearing the screen too fast.

Set rw is 46 files, 16-bit, each about 30 seconds long:

3 files with 1 message
1 file with 2 messages
1 file with 5 messages

I may have missed one, because encoding was too fast with 30 second files.

Set songs is 8 files, 16-bit, each about 120 seconds long:

2 files with 1 message
1 file with 4 messages

I often got some message like "Output file already exists". This seemed not to depend on the occurence of negative shift values and nevertheless a output file was created.

much effort to expend in properly handling them. as you have already figured out, FLAC does need some tweaks to do 24bit really well but I didn't want to hold 1.1.4 up for that.

Ah, ok. I simply used my standard test script, including 24 bit.

Thomas

[jcoalson]

ok, I would expect that a few 'negative shift' messages could occur for each file, especially at -8, but if there are thousands I would be interested in seeing the file to find out why. apart from that the exact number is not critical.

as for logging, the messages are going to stderr; not sure how to redirect that with the dos shell.

thanks everyone! the alpha seems to not have any obvious problems. I think with some more feedback here it will be possible to skip the beta since not much else in flac has changed.

Josh

[Synthetic Soul]

CODE

flac114alpha1_debug.exe myfile.wav 2>>stderr.txt