New FLAC encoder Options

[Northpole]

@Mangix

Thanks for the tip. Worked good.

[JWolf]

Can someone please post a Win32 compile of FLAKE 112 please? Thanks!

[Synthetic Soul]

As per post #319:

http://win32builds.sourceforge.net/flake/index.html

[JWolf]

As per post #319:

http://win32builds.sourceforge.net/flake/index.html

Nice. Now I can bookmark it. Thank you!

[wisodev]

I have uploaded new experimental Win64 builds of FLAKE using latest sources. Please try this builds and submit results.

Binarys for 64 bit windows (should work on EM64T and AMD64 compatible processors):

flake-svn-r112-win64-bin.zip

Other related downloads are placed here.

This post has been edited by wisodev: Dec 17 2006, 17:49

[miros]

The variable block size mode has a small bug that makes seeking impossible. In frame header is stored frame number instead of sample number. A quick hack:

CODE

Index: libflake/encode.c
===================================================================
--- libflake/encode.c    (revision 112)
+++ libflake/encode.c    (working copy)
@@ -973,7 +973,10 @@
         assert(!ctx->bw->eof);
     }
     if(frame_buffer != NULL) {
-        ctx->frame_count++;
+        if(ctx->params.variable_block_size)
+            ctx->frame_count += s->params.block_size;
+        else
+            ctx->frame_count++;
     }
     return bitwriter_count(ctx->bw);
}

[jcoalson]

yes. actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.

Josh

[Justin Ruggles]

ok, turns out the trouble with this particular file is actually in the wave chunk handling; I think it is including non-'data' chunks that follow the 'data' chunk and treating them as audio. when I have more time after the next release I will look into the rest of the suite.
[attachment=2624:attachment]

You're right about the bug. It is now fixed. The wav reader was only checking for end-of-file not end-of-data-chunk. Thanks for the bug repot...sorry for the slow reply. I'm back doing development after some unexpected downtime.

yes. actually, storing the frame number in a variable-blocksize file is illegal according to the FLAC spec.

This issue has been fixed as well. Thanks miros for the info.

-Justin

[wisodev]

My Flake SVN builds at revision 114 are available for download.

http://win32builds.sourceforge.net/flake/index.html

Download binaries: flake-svn-r114-bin.zip

Download sources: flake-svn-r114-src.zip

wisodev

This post has been edited by wisodev: Jan 9 2007, 21:34

[NuSkooler]

Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:

Fatal Error : This program was not built to run on the processor in your system.

I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?

[jcoalson]

I've reworked the flac test suite to use flake. it actually took a bit of work since the options are so different but I think the coverage is pretty good for the flac test streams at least. it could use more tests that exercise specific features of flake.

but, even then the tests crash on a lot of inputs. also there are places where it doesn't handle odd resolutions correctly; but except for those, I haven't run across any places where it is silently non-lossless yet (the worst kind of bug). the tests are still running though.

Justin, I'll PM you about the scripts so if you want you can try to reproduce them.

Josh

[jcoalson]

PS uploaded here: http://www.hydrogenaudio.org/forums/index....showtopic=51745

[wisodev]

Using the file flake.exe in ~/flake-svn-r114-bin/exe_x64 I get the following error:

Fatal Error : This program was not built to run on the processor in your system.

I am using a AMD 64 X2 5000+ on Windows XP 64. Am I wrong in thinking this should be the proper executable or is this a bug?

I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.

Edit:
From compiler documantation (I used /QxP):

/QxP
Intel® Core™ Duo processors and Intel® Core™ Solo processors, Intel Pentium 4 processors with Streaming SIMD Extensions 3, and compatible Intel processors with Streaming SIMD Extensions 3 (SSE3)

If a program compiled with -x (Linux) or /Qx (Windows) is executed on a non-compatible processor, it might fail with an illegal instruction exception or display other unexpected behavior. Executing programs compiled with -xN, -xB or -xP (Linux) or /QxN, /QxB or /QxP (Windows) on unsupported processors will display a run-time error similar to the following:

Fatal Error : This program was not built to run on the processor in your system.

This post has been edited by wisodev: Jan 10 2007, 13:19

[alvaro84]

AFAIK Intel's compiler generates such a code that explicitly checks for the 'GenuineIntel' string in the CPUID, and if it started on an AMD processor, it simply quits even though an X2 should run the program without problems. Someone should try to 'crack out' this check from the startup code...

This post has been edited by alvaro84: Jan 10 2007, 17:48

[PrakashP]

It might not worth the trouble, because usually gcc generates better code for Athlon CPUs - if optimization for specific Athlon CPUs is chosen.

[NuSkooler]

I was using as target system based on a processor with Intel® EM64T running Windows Server 2003 x64 Edition or Windows XP Professional x64 Edition as described in documentation. I was thinking that this should work on Intel and AMD processor running on 64 bit CPU, but now I am not sure. I need to investigate this or maybe someone can put more light here? Sorry for any inconvenience from my side.

I'm not sure if this is the problem, but Intel does have the Itanium 64 chipset that uses a different instruction base than the "x86" 64's (AMD 64's, etc.). If compiled for said chips, the code would not function on non-Itanium processors. AFAIK, this is only for special builds of Windows XP / 2003 and M$ has started (Vista+) supporting the more standard "x86" instruction set that is used by AMD.

[wisodev]

I have added new flake builds to my win32builds project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.

Download:

Visual Studio 2005 builds of flake svn revision 114.

Visual Studio 2005 and Intel C++ Compiler 9.1 builds of flake svn revision 114.

Thanks,
wisodev

[NuSkooler]

I have added new flake builds to my win32builds project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.

I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!

[wisodev]

I have added new flake builds to my win32builds project. This time there are two experimental packages. Both are built under Visual Studio 2005 and one of them using Intel C++ Compiler. There are two output binaries for each package. One for Win32 and other for x64 platform.

I just tested the release binaries of both the M$ and Intel compilers. Good news: They both work great!

Great news!

Anyway both releases have some potential for speed up. I didn't used PGO optimizations (VC2005 compiler has also this feature).

[Caroliano]

I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed, the generated file is 1211905 samples long, while the original wav file is 1211894.

With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.

BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.

I can reproduce the error here, and you can find the source file here, to check if it is realy an error. The file for download was encoded by an old version of Flac, not by Flake.

This post has been edited by Caroliano: Jan 25 2007, 03:29

[Justin Ruggles]

I have an MD5 mismach with the "flake-0.10-win32-wisodev-bin" that I downloaded from Flake SourceForge project page. If I compare with foobar, I get: "ERROR (foo_bitcompare) : Files have different length". Indeed, the generated file is 1211905 samples long, while the original wav file is 1211894.

With the "flake-icl91-r114", also compiled by wisodev, I have an perfect output. So, whatever it was, is already corrected.

BUT not in the Flake SourceForge project page, that is where people expect to find an good binary. We need a new official release, if this error is real.

I can reproduce the error here, and you can find the source file here, to check if it is realy an error. It was encoded by an old version of Flac, not by Flake.

I could not reproduce, but I believe you and agree with your point. There have been some critical bug fixes since version 0.10 was released. I am on vacation next week, so I hope to be able to package up a new release and put it up on Sourceforge. I also hope to get Flake to pass the FLAC test suite before then.

[jcoalson]

Justin, I found another bug that may be the culprit. I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's. then I noticed a problem in bitwriter_writebits():

CODE

static inline void
bitwriter_writebits(BitWriter *bw, int bits, uint32_t val)
{
    assert(bits == 32 || val < (1U << bits));

    if(bits == 0) return;
    if(bw->eof || (bw->buf_ptr+3) >= bw->buf_end) {
        bw->eof = 1;
    } else {
        if(bits < bw->bit_left) {
            bw->bit_buf = (bw->bit_buf << bits) | val;
            bw->bit_left -= bits;
        } else {
            bw->bit_buf <<= bw->bit_left;    ///////////// <-------- HERE
            bw->bit_buf |= val >> (bits - bw->bit_left);
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
            bw->bit_left += (32 - bits);
            bw->bit_buf = val;
        }
    }
}

if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits. according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.

I think it's just lucky that it is rarely (never?) triggered. most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion. but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.

in my new writer I am doing something like this (trying to translate back into flake-style, may be buggy):

CODE

        } else if(bw->bit_left<32) {
            bw->bit_buf <<= bw->bit_left;    ///////////// <-------- OK now
            bw->bit_buf |= val >> (bits - bw->bit_left);
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
            bw->bit_left += (32 - bits);
            bw->bit_buf = val;
        } else {
            assert(bits==32);
            bw->bit_buf = val;
            if(bw->buffer != NULL) {
                *(uint32_t *)bw->buf_ptr = be2me_32(bw->bit_buf);
            }
            bw->buf_ptr += 4;
        }

it can actually be optimized more but that's the idea.

Josh

[Justin Ruggles]

Justin, I found another bug that may be the culprit. I've gone back to finish the speedups of the libFLAC bitbuffer I started a long time ago and realized it is similar to flake's. then I noticed a problem in bitwriter_writebits():
...
if bits==32 and bw->bit_left==32, then it will get to that line and try to shift left by 32 bits. according to C semantics that is a NOP, so bit_buf will have junk instead of being 0 for the next instruction.

I think it's just lucky that it is rarely (never?) triggered. most writes are <32bit quantities and you do do 31 bits at a time of the rice unary portion. but some metadata writes are doing 32 bits and if they are word-aligned there could be trouble.
...

Thanks Josh! I actually did not know that a left shift by 32 was a NOP. The code which this bit writer was based upon only allowed for doing 31 bits at once, and I did not catch that outside case when trying to adapt it to handle 32 bits. I applied the fix as you suggested.

-Justin

[Madman2003]

What's the state of flake in the sense of trustworthyness?

Also, any info if/when metadata support becomes available?

This post has been edited by Madman2003: Feb 16 2007, 16:10

[Justin Ruggles]

What's the state of flake in the sense of trustworthyness?

Also, any info if/when metadata support becomes available?

I'll let others answer as to general trustworthyness. I will say that Flake does generate MD5 checksums, but it does not have the verify-during-encoding option like flac does.

Metadata is far down the TODO list. For now, the only option is to apply metadata to flake-generated files using metaflac.