What is the compression percentage of your FLAC library? Options

[JJZolx]

I've always been curious about the average compression rate across my whole FLAC library.

My library is just shy of 36,000 files, with a makeup of about 40% rock, 40% jazz, and the remainder a mixture of bluegrass, country, classical, folk, rap, electronic, etc. It's all encoded at the default -5 level, and the tagging consists of basic text fields, with no embedded images. I compared the total file sizes in bytes to the calculated expanded music size.

It worked out to 41.9% compression on the entire library.

Total FLAC files size: 925.092 GB (993,310,377,824 bytes)
Total uncompressed size: 1580.926 GB (1,708,243,555,030 bytes)
Ratio: 0.581
Compression: 41.9%

[tuffy]

Not counting the metadata blocks, it works out to:

Compressed: 688.5GB (739,319,931,688 bytes)
Uncompressed: 1243.7GB (1,335,454,659,871 bytes)
Ratio: 0.5536

The metadata blocks take up an additional 6.2GB, which is mostly embedded cover art.

[godrick]

Not counting the metadata blocks, it works out to:

Compressed: 688.5GB (739,319,931,688 bytes)
Uncompressed: 1243.7GB (1,335,454,659,871 bytes)
Ratio: 0.5536

The metadata blocks take up an additional 6.2GB, which is mostly embedded cover art.

Tuffy, how did you determine the amount of space taken by your metadata blocks? Thanks!

[tuffy]

Tuffy, how did you determine the amount of space taken by your metadata blocks? Thanks!

I wrote a little Python script to do it. FLAC's metadata blocks are pretty easy to walk through so it didn't take long.

[godrick]

Tuffy, how did you determine the amount of space taken by your metadata blocks? Thanks!

I wrote a little Python script to do it. FLAC's metadata blocks are pretty easy to walk through so it didn't take long.

To channel a kid/young adult in a cheesy commercial for a product I can't recall who repeatedly whines "....and?"...

It would be great if you could post your admittedly clever python script thanks!

[tuffy]

To channel a kid/young adult in a cheesy commercial for a product I can't recall who repeatedly whines "....and?"...

It would be great if you could post your admittedly clever python script thanks!

Here goes. It's called with a single directory as the first argument which it searches recursively. It assumes your FLAC files end in ".flac" and match the spec (without any ID3 junk at the beginning, an initial STREAMINFO block and so on). There's no progress indicator either, but you're welcome to add some if you like.

CODE

#!/usr/bin/python

import sys
import os.path
import struct

def read_block_header(f):
    """takes FLAC file stream
    returns (is last, block ID, block length)"""

    i = struct.unpack(">I", f.read(4))[0]
    return (i >> 31,
            (i >> 24) & 7,
            i & 0xFFFFFF)

def parse_streaminfo(s):
    """takes 34 byte STREAMINFO block as string
    returns (sample rate, channel count, bits per sample, PCM frames)"""

    i = struct.unpack(">10xQ16x", s)[0]

    return (i >> 44,
            ((i >> 41) & 7) + 1,
            ((i >> 36) & 0x1F) + 1,
            i & 0xFFFFFFFFF)

def FLAC_sizes(filename):
    """returns (metadata length, frames length, uncompressed size)
    or raises IOError or ValueError if some error occurs"""

    metadata_length = 0

    f = open(filename, "rb")
    if (f.read(4) == "fLaC"):
        metadata_length += 4
    else:
        raise ValueError("not a FLAC file")

    (is_last, block_ID, block_length) = read_block_header(f)
    metadata_length += 4
    if (block_ID == 0):
        (sample_rate,
         channel_count,
         bits_per_sample,
         PCM_frames) = parse_streaminfo(f.read(block_length))
        uncompressed_size = (channel_count *
                             (bits_per_sample / 8) *
                             PCM_frames)
        metadata_length += block_length
    else:
        raise ValueError("STREAMINFO not first block in stream")

    while (is_last != 1):
        (is_last, block_ID, block_length) = read_block_header(f)
        metadata_length += 4
        f.read(block_length)
        metadata_length += block_length

    return (metadata_length,
            os.path.getsize(filename) - metadata_length,
            uncompressed_size)

if (__name__ == "__main__"):
    total_metadata_length = 0
    total_frames_length = 0
    total_uncompressed_length = 0

    for (d, ds, fs) in os.walk(sys.argv[1]):
        for f in fs:
            if (f.lower().endswith(".flac")):
                path = os.path.join(d, f)
                try:
                    (metadata_length,
                     frames_length,
                     uncompressed_length) = FLAC_sizes(path)
                    total_metadata_length += metadata_length
                    total_frames_length += frames_length
                    total_uncompressed_length += uncompressed_length
                except (ValueError, IOError), err:
                    print "*** %s: %s" % (path, err)

    print "    metadata length : %s" % (total_metadata_length)
    print "total frames length : %s" % (total_frames_length)
    print "uncompressed length : %s" % (total_uncompressed_length)
    print "        compression : %2.2f%%" % \
        ((float(total_frames_length) * 100) /
         float(total_uncompressed_length))