New FLAC encoder Options

[jcoalson]

I have actually been doing experiments solving the full prediction linear system with SVD; this should give a lower bound on the compression achievable by the FLAC filter. so far it works but it will not be ready for the next release.

I suppose the pseudo-inverse approach (through SVD) is even slower (but more accurate) than the Cholesky approach.

yep, you're right. I'm mainly interested in establishing the lower bound first to see if other solvers would even be practical.

as for compression improvement, I don't have good numbers yet but it's not dramatic. the autocorrelation formulation is surpisingly good with the proper window.

Josh

[HbG]

Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.

I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.

[Justin Ruggles]

Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.

I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.

Sorry about that. The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder. I'll run valgrind on Flake and see if I can fix that leak.
-Justin

[HbG]

Thanks in advance!

[Justin Ruggles]

Only noticed this now, it's leaking memory like mad. It's about the size of the uncompressed audio that is being leaked. Both flake_06.zip and flake-0.6-win32.rar from Justin's site builds do it.

I like to encode whole cd's as a single file, normally only one at the time from the console, but i'm doing a batch job now and notice my machine is very unresponsive because it's swapping everything out to disk for flake.

Sorry about that. The Flake development is in a lull right now since I'm working feverishly on the Aften AC-3 encoder. I'll run valgrind on Flake and see if I can fix that leak.
-Justin

Here is a new bug-fix release to correct the memory leaks. This download also includes some pre-compiled binaries for Linux and Windows.

http://jbr.homelinux.org/flake/flake-07.tar.gz

-Justin

[HbG]

That fixed it, thanks.

But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.

Only tested the windows binary, the output is correct.

[wisodev]

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Download:
http://www.thefrontend.net/flake/flake-0.7-win32.rar
Tests:
http://www.thefrontend.net/flake/TESTS-0.7-win32.zip
Old builds:
http://www.thefrontend.net/flake/

wiso

[Justin Ruggles]

That fixed it, thanks.

But the speed's now dropped from 29.40x to about 18.5, still much faster than flac (11.9x) but not nearly as impressive.

Only tested the windows binary, the output is correct.

Okay, well that has convinced me to take a leap and release a more up-to-date version of Flake. I'm not sure if the windows binaries will still be slower than before since I was not the one who created the binary for version 06. All I can say is that it's faster for me on Linux, and I hope it is on Windows as well.

http://jbr.homelinux.org/flake/

It still doesn't have all the features in the FFmpeg encoder, but I'll get to that eventually.

Enjoy.
-Justin

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Sorry to do two releases back-to-back. I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it. I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.

Also, my versioning has changed. I was using "pre-release 01,02,...07". Now I've switched to more normal versioning. The latest is version 0.08.

Thanks!
-Justin

[wisodev]

I have build the binary for Win32 (version 07 of flake) using my build enviroment and its lot faster then Win32 build from Justin Ruggles ;-) (I have tested and compared speed for this binary on my prevoius wav files and builds).

Sorry to do two releases back-to-back. I've had the 0.08 version sitting around and had wanted to tweak it some more before release, but I realized that might be a while, so I went ahead and released it. I'll probably create a Sourceforge page so that I can do SVN updates and not have to release so often.

Also, my versioning has changed. I was using "pre-release 01,02,...07". Now I've switched to more normal versioning. The latest is version 0.08.

Thanks!
-Justin

This is no problem for me. The progress is there and this is more important.

wiso

[HbG]

Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.

Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:

CODE

D:\MP3\Pearl Jam\Pearl Jam - Ten>type test.wav |flake -8 - test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 23965116

blocksize: 4608
max order: 12
order method: 4-level
stereo method: search

| bytes: 4448

The process tried to write to a nonexistent pipe.

0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:

CODE

D:\MP3\Pearl Jam\Pearl Jam - Ten>flake-win32-i586.exe test.wav test.flac

Flake: FLAC audio encoder
(c) 2006  Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 23965116

D:\MP3\Pearl Jam\Pearl Jam - Ten>

All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?

Like wisodev says, there is improvement and that is most important. Flake has a lot of potential, keep up the good work!

[Justin Ruggles]

Oh yeah, wisodev's .07 build clocks in at about 29.1x for me with no memory leaks. Slightly slower than the leaky .06'es, but a major improvement nevertheless. Compression is the same. I should mention that i made some slight changes to my system and a reinstall, so there may be inaccuracies with my old benchmark results, but these should be fairly small, probably no more than .1x.

Pipe support still seems disfunctional, i get the following with both wisodev's .07 and justin's original fixed .06:

Hmm...I'll try to fix that after I get the Sourceforge page up and running. I'm hoping for approval tomorrow.

0.08's windows binaries do not work for me. It reads the files but never starts encoding, just throws me back to the commandline, like this:
...
All three builds do it. It does parse options and respond to -h, it creates a 0 byte output file.
Also, is there any chance of a changelog?

Now that is strange. If I have a chance I'll install Wine so I can do some debugging. I'm guessing it's related to configure & Makefile.

Ok, I'll create a changelog. Having the source in an svn repository will help document development as well since there will be commit messages.

[wisodev]

He, I have found too that 0.08 windows binarys are not working (all of them) even my builds are not working.

When it starts it throws stack overflow error on my machine (Windows XP)

But I have found the source of this problem.

in file encode.c in function

CODE

flake_encode_init(FlakeContext *s)

there was changed since version 07 way of handling context (FlacEncodeContext ctx_local;) memory allocation
and this change to local variable is causing under windows stack overflow

CODE

    FlacEncodeContext ctx_local;
    FlacEncodeContext *ctx;
    int i, header_len;

    if(s == NULL) {
        return -1;
    }

    ctx = &ctx_local;

I have changed to method used in 07 (allocating memory at the beginning of the function) and it works OK and do not throws any errors.

wiso

[sundance]

Justin,

would you consider adding wisodev's "patch" to flake.c that he did in v06?
Or is there any catch I can't see?

CODE

    if(!strncmp(opts.infile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
        _setmode(_fileno(stdin), _O_BINARY);
#endif
        ifp = stdin;
    } else {
        ifp = fopen(opts.infile, "rb");
        if(!ifp) {
            fprintf(stderr, "error opening input file: %s\n", opts.infile);
            return 1;
        }
    }
    if(!strncmp(opts.outfile, "-", 2)) {
#ifdef _WIN32 // win32 pipes patch by wiso
        _setmode(_fileno(stdout), _O_BINARY);
#endif
         ofp = stdout;
    } else {
        ofp = fopen(opts.outfile, "wb");
        if(!ofp) {
            fprintf(stderr, "error opening output file: %s\n", opts.outfile);
            return 1;
        }
    }

It would help a lot to have piping work in WIN32...

Thanks a lot!

.sundance.

This post has been edited by sundance: Sep 11 2006, 22:01

[Justin Ruggles]

would you consider adding wiso's "patch" to flake.c:

Yeah, I'll apply both of wiso's fixes and put them into SVN as soon as Sourceforge approves the project. I'll also do a bit of code cleanup and documentation before doing the initial import and then jump up to version 0.10 as the 1st Sourceforge release.

-Justin

[sundance]

Justin,

that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake

.sundance.

[Justin Ruggles]

Justin,

that's really good news.
Too bad that there's already a project called FLAKE registered at SF: http://sourceforge.net/projects/flake

.sundance.

I know! I guess I should have looked around first... I had no idea until I tried to register. Mine doesn't use all caps though. The "unix name" will be flake-enc. Still, I don't know if that will be grounds for rejection? I may have to rename it, which would really suck!

-Justin

[Justin Ruggles]

Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/

I'll work on getting everything uploaded and active this afternoon and evening.

[jcoalson]

"flache" (High-Efficiency? painful?) is not taken...

[Justin Ruggles]

Whew...it was accepted.
http://sourceforge.net/projects/flake-enc/

I'll work on getting everything uploaded and active this afternoon and evening.

The SVN repository is up.
Version 0.10 has been released.

For now I have built binaries for Linux, optimized for i586, i686, or k8. Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686. If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.

Thanks,
Justin

[wisodev]

For now I have built binaries for Linux, optimized for i586, i686, or k8. Also I included a couple (probably still slow) cross-compiled binaries for Windows, optimized for i586 or i686. If wiso decides to be very kind again and builds a faster Windows binary, I would be happy to put it on Sourceforge as well.

Thanks,
Justin

I have downloaded the sources from sourceforge and I will upload binarys as soon as I come back home from work.

[sundance]

What do you think about adding some version information to the console output like

CODE

Flake: FLAC audio encoder version 0.10
(c) 2006  Justin Ruggles

to be able to identify which version of flake you're testing more easier.
Great you already included that version information as a vendor string to the flac header!

.sundance.

This post has been edited by sundance: Sep 12 2006, 14:04

[krmathis]

Justin. I pulled revision 2 from SVN, and tried to build this on Mac OS 10.47 PowerPC/ GCC 4.0.1 (Xcode Tools 2.4). But it errors out like this:

$ ./configure
source path /Users/krmathis/Downloads/flake-enc
C compiler cc
make make
CPU powerpc (generic)
big-endian yes
inttypes.h yes
AltiVec enabled yes
debug symbols yes
strip symbols yes
optimize yes
Creating config.mak and config.h...
$ make
make -C libflake all

** Snip **

cc -O3 -no-cpp-precomp -pipe -force_cpusubtype_ALL -Wno-sign-compare -fomit-frame-pointer -faltivec -g -Wdeclaration-after-statement -Wall -Wno-switch -mdynamic-no-pic -DHAVE_CONFIG_H -I. -I.. -I/Users/krmathis/Downloads/flake-enc/libflake -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -c -o wav.o wav.c
cc -L../libflake -Wl,-dynamic,-search_paths_first -g -o flake_g flake.o wav.o -lflake
/usr/bin/ld: Undefined symbols:
_bswap_16
collect2: ld returned 1 exit status
make[1]: *** [flake_g] Error 1
make: *** [progs] Error 2

I successfully compiled flake 0.7 two days ago. But the build system have changed since then (from "make" to "./configure && make")

Any idea whats wrong?

[wisodev]

FLAKE 0.10 Win32 Build and Sources

Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.rar
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.rar

Alternative ZIP download:

Download binarys:
http://www.thefrontend.net/flake/flake-0.10-win32-bin.zip
Download sources:
http://www.thefrontend.net/flake/flake-0.10-win32-src.zip

I have tested my build and it looks like encoding speed has increased , this is great news!

Test results (using my build, sf.net build i686 and flac 1.1.2 build):

CODE

TEST FILE: 44100Hz, 16bit, Stereo, 431 916 620 bytes (13 Tracks)

ENCODE_FLAKE_INTEL [-8]

Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10

Flake: FLAC audio encoder
© 2006 Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 107972676

blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side

progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517


Kernel Time = 3.328 = 00:00:03.328 = 3%
User Time = 93.218 = 00:01:33.218 = 89%
Process Time = 96.546 = 00:01:36.546 = 92%
Global Time = 104.172 = 00:01:44.172 = 100%
ENCODE_FLAKE_i686 [-8]

Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10

Flake: FLAC audio encoder
© 2006 Justin Ruggles

Signed 16-bit 44100 Hz stereo
samples: 107972676

blocksize: 4608
max order: 12
order method: 4-level
stereo method: mid-side

progress: 100% | ratio: 0.741 | bitrate: 1045.2 kbps | bytes: 319851517


Kernel Time = 3.078 = 00:00:03.078 = 2%
User Time = 132.953 = 00:02:12.953 = 94%
Process Time = 136.031 = 00:02:16.031 = 96%
Global Time = 140.671 = 00:02:20.671 = 100%
ENCODE_FLAC_1_1_2 [-8]

Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

options: -P 4096 -b 4608 -m -l 12 -e -q 0 -r 0,6
Catamenia - [1999] - Morning Crimson (HQ).wav: wrote 321356528 bytes, ratio=0,74
4

Kernel Time = 3.828 = 00:00:03.828 = 1%
User Time = 206.640 = 00:03:26.640 = 96%
Process Time = 210.468 = 00:03:30.468 = 97%
Global Time = 214.891 = 00:03:34.891 = 100%
DECODE_FROM_FLAKE_INTEL [-8]

Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

t_INTEL.flac: done

Kernel Time = 3.328 = 00:00:03.328 = 7%
User Time = 22.703 = 00:00:22.703 = 53%
Process Time = 26.031 = 00:00:26.031 = 61%
Global Time = 42.188 = 00:00:42.188 = 100%
DECODE_FROM_i686 [-8]

Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

t_i686.flac: done

Kernel Time = 3.140 = 00:00:03.140 = 7%
User Time = 23.078 = 00:00:23.078 = 56%
Process Time = 26.218 = 00:00:26.218 = 63%
Global Time = 41.094 = 00:00:41.094 = 100%
DECODE_FROM_FLAC_1_1_2 [-8]

Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10

flac 1.1.2, Copyright © 2000,2001,2002,2003,2004,2005 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

t_FLAC.flac: done

Kernel Time = 3.015 = 00:00:03.015 = 7%
User Time = 22.953 = 00:00:22.953 = 56%
Process Time = 25.968 = 00:00:25.968 = 63%
Global Time = 40.765 = 00:00:40.765 = 100%

---

FILE SIZE

t_FLAC.flac 321 356 528
t_i686.flac 319 851 517
t_INTEL.flac 319 851 517

[lexor]

CODE

FILE            SIZE

t_FLAC.flac        321 356 528
t_i686.flac        319 851 517
t_INTEL.flac        319 851 517

wisodev, what's up with the second column?

This post has been edited by lexor: Sep 12 2006, 20:21

[HbG]

My run went at 37.00x That's more than three times faster than flac at -8 and 7.5x up from 0.7. Memory usage dropped a little, and no leaks.

Pipe support works, i can use it as external encoder in foobar2000 without problems.
What's even better, it seems foobar can automatically emebed a .cue file in the tag, only it doesn't preserve indexes, and it doesn't embed it in the actual flac way. Can't win 'em all.

Still, this is an awesome achievement. I can't believe the reference encoder had so much room for improvement!

This needs to get on rarewares.

This post has been edited by HbG: Sep 13 2006, 01:21