MP3 repacker Options

[no404error]

Intel i7-3770, Lame 3.100a0, mp3packer64

CODE

a016-vbr.mp3 686.95x
a032-vbr.mp3 343.89x
a048-vbr.mp3 247.29x
a064-vbr.mp3 199.26x
a080-vbr.mp3 181.80x
a096-vbr.mp3 171.76x
a112-vbr.mp3 135.82x
a128-vbr.mp3 126.14x
a160-vbr.mp3 123.63x
a192-vbr.mp3 118.90x
a256-vbr.mp3 116.60x
a320-vbr.mp3 114.46x
c016-vbr.mp3 649.94x
c032-vbr.mp3 334.46x
c048-vbr.mp3 242.32x
c064-vbr.mp3 199.47x
c080-vbr.mp3 179.10x
c096-vbr.mp3 167.15x
c112-vbr.mp3 134.31x
c128-vbr.mp3 130.14x
c160-vbr.mp3 122.45x
c192-vbr.mp3 118.90x
c256-vbr.mp3 114.46x
c320-vbr.mp3 109.49x
vn00-vbr.mp3 113.45x
vn01-vbr.mp3 116.67x
vn02-vbr.mp3 117.74x
vn03-vbr.mp3 121.22x
vn04-vbr.mp3 123.63x
vn05-vbr.mp3 126.14x
vn06-vbr.mp3 131.47x
vn07-vbr.mp3 167.15x
vn08-vbr.mp3 193.27x
vn09-vbr.mp3 213.14x
vo00-vbr.mp3 115.56x
vo01-vbr.mp3 116.67x
vo02-vbr.mp3 116.60x
vo03-vbr.mp3 122.37x
vo04-vbr.mp3 124.91x
vo05-vbr.mp3 124.91x
vo06-vbr.mp3 128.75x
vo07-vbr.mp3 167.15x
vo08-vbr.mp3 176.64x
vo09-vbr.mp3 206.08x

This post has been edited by db1989: Jul 19 2012, 13:22

Reason for edit: For large items, please use [codebox] and not [code].

[Porcus]

I compared a troublesome .mp3 with 1.25.7 and 2.01. Both -z. The former warns against too many frequencies. The latter doesn't. Don't know if this is bug or improvement.

[Omion]

I compared a troublesome .mp3 with 1.25.7 and 2.01. Both -z. The former warns against too many frequencies. The latter doesn't. Don't know if this is bug or improvement.

It's a bit of a bug. I guess I forgot to move that warning to my new dequantization routine. I'll add it back in the next version, but it's a fairly minor warning anyway so it can be easily ignored. Thanks for the report!

In other news, I just made what seems to be a fairly generalized and deadlock-proof threadpool algorithm. I hope to process multiple frames in parallel with this, instead of just multiple granules as in 2.00/2.01. Hopefully I'll get 100% multi-core CPU usage and a linear speed increase with this.

[no404error]

Edit: Never mind It's fs fault.

This post has been edited by no404error: Jul 19 2012, 13:16

[Omion]

2.02 is out (mirror)

I changed how it handles the multithreaded stuff, so it should be able to use more CPU power if available. I also made it more efficient so it should go faster per CPU second anyway.
I also added CRC checking for the input files. Incorrect CRCs are reported with other errors, and frames with incorrect CRCs will skip -z processing.

One test I ran on my 8-logical-core processor with -z:
2.01 x64: 129.45x with ~17% CPU usage
2.02 x64: 364.06x with ~29% CPU usage

Files which report no errors will be bit-identical outside of the padding.


I would also like to request a bit of community feedback:
I set the default number of -z processing threads to 3 since that's the best on my processor (i7-2600k). I'd like some info from anybody who happens to use my program on other processors which --workers setting gives the best speed on a fairly long CD-quality file, along with the processor you have.

So run:

CODE

mp3packer -f -z --workers # path\to\some_file.mp3 NUL

changing the --workers number from 0 to however many cores the OS reports. Then PM me or post in this thread which --workers setting was the fastest. (Using NUL as the output will throw out the resulting file)

I'd especially be interested in anyone with an AMD Bulldozer CPU. Thanks!

[Destroid]

Processed MP3 file created with LAME 3.99 -b 320 (duration 39:32, 90813 frames), running i5-2500k, 8GB, WinXP x64 SP2.

CODE

MP3packer 2.01 (x86) -z 92.35x
MP3packer 2.01 (x64) -z 104.20x

MP3packer 2.02 (x86) -z --workers 0 121.65x
MP3packer 2.02 (x86) -z --workers 1 218.46x
MP3packer 2.02 (x86) -z --workers 2 247.68x
MP3packer 2.02 (x86) -z --workers 3 253.45x
MP3packer 2.02 (x86) -z --workers 4 246.08x

MP3packer 2.02 (x64) -z --workers 0 146.97x
MP3packer 2.02 (x64) -z --workers 1 261.32x
MP3packer 2.02 (x64) -z --workers 2 291.97x
MP3packer 2.02 (x64) -z --workers 3 308.57x
MP3packer 2.02 (x64) -z --workers 4 311.73x

I wasn't sure if worker parameter was in addition to the main program thread or not, so I ran it with "4" anyway.

Also, when using your suggested NUL parameter the program kept crashing at the same point in processing :shrug:

CODE

D:\audio>mp3packer -f -z --workers 1 btttol.mp3 NUL

*** 'btttol.mp3' -> 'NUL'
43% done on frame 39049

[no404error]

Intel i7-3770

1 - 252.44x/242.59x/242.59x/247.26x/1394.87x/757.33x*
2 - 294.45x/281.14x/287.42x/294.45x/1485.97x/825.57x*
3 - 281.14x/274.73x/281.14x/287.42x/1470.75x/835.87x*
4 - 287.85x/268.61x/268.98x/274.73x/1283.16x/774.27x*
5 - 268.98x/268.61x/268.61x/274.73x/1246.18x/743.85x*
6 - 268.98x/268.61x/274.73x/274.73x/1226.58x/753.90x*
7 - 268.98x/257.49x/263.11x/274.73x/1204.46x/744.11x*
8 - 242.59x/252.11x/247.26x/252.11x/1179.81x/738.09x*

1 - 287.42x/280.73x/287.42x/287.42x/1801.70x/648.19x
2 - 334.25x/325.23x/334.25x/333.67x/2099.14x/688.46x
3 - 363.89x/353.23x/353.23x/353.23x/2167.60x/761.62x
4 - 353.23x/343.78x/353.23x/353.23x/1840.04x/824.90x
5 - 343.17x/353.23x/343.17x/353.23x/1519.79x/744.40x
6 - 334.25x/343.17x/343.17x/353.23x/1392.56x/672.97x
7 - 325.23x/325.23x/325.23x/317.21x/1320.33x/633.84x
8 - 294.00x/301.35x/294.45x/301.35x/1235.45x/593.71x

* mp3packer86/test6.mp3 = Sync error on frame 513803, CRC error on frame 513803-1198044.

CODE

=== test1.mp3 ===
INFO:
MPEG1 layer 3
7383 frames
44100 Hz
38.281250 frames per second
192.862041 seconds
6688676 bytes in file (277.449143 kbps)
6688259 bytes in MP3 frames (277.431846 kbps) = current bitrate
51353241 bits of payload data (266.269302 kbps)
6422380 bytes of payload data (266.403071 kbps)
25799 bits wasted from partially-full bytes (0.133769 kbps)
6688168 bytes of MP3 data (277.428071 kbps) = minimum bitrate possible
91 bytes of padding (0.003775 kbps)
417 bytes outside MP3 frames (0.017297 kbps)
0 sync errors
Bitrate distribution:
160: 1,0
192: 3,0
224: 1489,0
256: 2640,0
320: 3250,0
Largest frame uses 9616 bits = 1202 bytes = 368.112500 kbps
Smallest bitrate for CBR is 320

=== test2.mp3 ===
INFO:
MPEG1 layer 3
7383 frames
44100 Hz
38.281250 frames per second
192.862041 seconds
7715525 bytes in file (320.043279 kbps)
7714481 bytes in MP3 frames (319.999974 kbps) = current bitrate
58088768 bits of payload data (301.193370 kbps)
7264287 bytes of payload data (301.325734 kbps)
25528 bits wasted from partially-full bytes (0.132364 kbps)
7530075 bytes of MP3 data (312.350734 kbps) = minimum bitrate possible
184406 bytes of padding (7.649240 kbps)
1044 bytes outside MP3 frames (0.043306 kbps)
0 sync errors
Bitrate distribution:
320: 754,6629
Largest frame uses 11735 bits = 1467 bytes = 449.230469 kbps
Smallest bitrate for CBR is 320

=== test3.mp3 ===
INFO:
MPEG1 layer 3
7383 frames
44100 Hz
38.281250 frames per second
192.862041 seconds
7667515 bytes in file (318.051804 kbps)
7667098 bytes in MP3 frames (318.034507 kbps) = current bitrate
57836089 bits of payload data (299.883216 kbps)
7232726 bytes of payload data (300.016570 kbps)
25719 bits wasted from partially-full bytes (0.133354 kbps)
7498514 bytes of MP3 data (311.041570 kbps) = minimum bitrate possible
168584 bytes of padding (6.992936 kbps)
417 bytes outside MP3 frames (0.017297 kbps)
0 sync errors
Bitrate distribution:
224: 2,0
256: 192,0
320: 7189,0
Largest frame uses 12044 bits = 1506 bytes = 461.059375 kbps
Smallest bitrate for CBR is 320

=== test4.mp3 ===
INFO:
MPEG1 layer 3
7383 frames
44100 Hz
38.281250 frames per second
192.862041 seconds
6203381 bytes in file (257.318899 kbps)
6202964 bytes in MP3 frames (257.301602 kbps) = current bitrate
47471625 bits of payload data (246.142915 kbps)
5937159 bytes of payload data (246.275896 kbps)
25647 bits wasted from partially-full bytes (0.132981 kbps)
6202947 bytes of MP3 data (257.300896 kbps) = minimum bitrate possible
17 bytes of padding (0.000705 kbps)
417 bytes outside MP3 frames (0.017297 kbps)
0 sync errors
Bitrate distribution:
64: 1,0
80: 13,0
96: 41,0
112: 25,0
128: 13,0
160: 58,0
192: 351,0
224: 2201,0
256: 2741,0
320: 1939,0
Largest frame uses 9533 bits = 1192 bytes = 364.935156 kbps
Smallest bitrate for CBR is 320

=== test5.mp3 ===
INFO:
MPEG2.5 layer 3
258229 frames
11025 Hz
19.140625 frames per second
13491.147755 seconds
94438176 bytes in file (56.000084 kbps)
94438034 bytes in MP3 frames (56.000000 kbps) = current bitrate
723951161 bits of payload data (53.661199 kbps)
90564336 bytes of payload data (53.702969 kbps)
563527 bits wasted from partially-full bytes (0.041770 kbps)
93921313 bytes of MP3 data (55.693594 kbps) = minimum bitrate possible
516721 bytes of padding (0.306406 kbps)
142 bytes outside MP3 frames (0.000084 kbps)
0 sync errors
Bitrate distribution:
56: 73780,184449
Largest frame uses 4095 bits = 512 bytes = 78.380859 kbps
Smallest bitrate for CBR is 56

=== test6.mp3 ===
INFO:
MPEG2 layer 3
1198046 frames
22050 Hz
38.281250 frames per second
31295.895510 seconds
625918038 bytes in file (160.000033 kbps)
625917910 bytes in MP3 frames (160.000000 kbps) = current bitrate
4861758484 bits of payload data (155.348118 kbps)
607946965 bytes of payload data (155.406185 kbps)
1817236 bits wasted from partially-full bytes (0.058066 kbps)
623521563 bytes of MP3 data (159.387435 kbps) = minimum bitrate possible
2396347 bytes of padding (0.612565 kbps)
128 bytes outside MP3 frames (0.000033 kbps)
0 sync errors
Bitrate distribution:
160: 660148,537898
Largest frame uses 4095 bits = 512 bytes = 156.761719 kbps
Smallest bitrate for CBR is 160

This post has been edited by db1989: Jul 30 2012, 12:55

Reason for edit: See post #551. It’s not [code]; it’s not [spoiler]; it’s [codebox].

[Omion]

Processed MP3 file created with LAME 3.99 -b 320 (duration 39:32, 90813 frames), running i5-2500k, 8GB, WinXP x64 SP2.

Thanks for the data! Your results are consistent with what I'm getting.

I wasn't sure if worker parameter was in addition to the main program thread or not, so I ran it with "4" anyway.

The worker threads are separate from the main thread. Whichever option you use for --workers, there should be 2 extra threads in the program - one for the OCaml GC and one for all of mp3packer's non-recompression stuff.

Also, when using your suggested NUL parameter the program kept crashing at the same point in processing :shrug:

CODE

D:\audio>mp3packer -f -z --workers 1 btttol.mp3 NUL

*** 'btttol.mp3' -> 'NUL'
43% done on frame 39049

Odd. Maybe Windows XP has some limitations on how the NUL device is implemented. Or maybe I added something that only crashes the program if the writing happens faster than normal.

Intel i7-3770

It's good to see an Ivy Bridge chip. Thanks for the response!

* mp3packer86/test6.mp3 = Sync error on frame 513803, CRC error on frame 513803-1198044.

Does this mean that the 64-bit version did not find the same CRC/sync errors as the 32-bit version? That would be... unexpected.

[no404error]

Does this mean that the 64-bit version did not find the same CRC/sync errors as the 32-bit version? That would be... unexpected.

log - https://dl.dropbox.com/u/5782249/log.7z (870kb)
file - uploaded (487Mb)

[BFG]

I know I'm still a n00b here, so you'll have to forgive me if this is a dumb question

I know that MP3Packer can reduce the size of CBR320 (and other CBRs) quite a bit. And, I suppose it would have a similar, but less extreme, effect on ABRs.
But I recently tried repacking a few random LAME VBR -V0 -q0, and the overall size reduction was only about 0.5%.
Is that similar to the experience others have had with LAME VBRs? If so, I won't bother trying to repack my entire VBR collection.

[Dynamic]

That's to be expected. Aside from brute force searching for the best data packing method, which takes too long for something like a 0.5% saving, LAME VBR uses the entire payload and any spare bits in frame N are carried over in the bit reservoir to encode some of the data of frame N+1, typically. (i.e. there's no waste, no padding bits, to be stripped out by mp3Packer)

This post has been edited by Dynamic: Jul 30 2012, 19:11

[djonline]

Bug in 2.03: --copy-time can't copy original file time when filename or filepath is cyrillic/non-latin.

[Omion]

Bug in 2.03: --copy-time can't copy original file time when filename or filepath is cyrillic/non-latin.

Sorry about the delay. I haven't checked this thread for a while.

Anyway, 2.04 is out, and it should fix your problem. For some reason I didn't use the Unicode-aware version of the file functions for the copy-time stuff.

[djonline]

Thank you, all work ok now!

[pwiecz]

Hi,
Is the source expected to compile on anything but Windows?
I've tried to compile version 2.04 on MacOSX and had to fix a few issues. The first of them should affect Linux as well.
1) mp3frameutils-c.c:1259 Non Windows variant of mfu_find_best_config_sse41 calls mfu_find_best_config which is undefined. Should it be mfu_find_best_config_base ?
2) c_part.c:189 gettid() is not defined on MacOSX, but apparently the whole function get_os_thread_self_id() is not used, so this call is unnecassary.
3) c_part.c:230 stat struct on MacOSX (and BSD I guess) is defined inside "sys/stat.h" which is not included. According to http://linux.die.net/man/2/stat this include should work on Linux as well.

With those fixes I've managed to compile it on MacOSX, but cannot tell if it works correctly (there are some differences with bitstream output form mpg123, but I can't tell if they are expected or not).

Best,
Piotr

[Omion]

Hi,
Is the source expected to compile on anything but Windows?
I've tried to compile version 2.04 on MacOSX and had to fix a few issues. The first of them should affect Linux as well.
1) mp3frameutils-c.c:1259 Non Windows variant of mfu_find_best_config_sse41 calls mfu_find_best_config which is undefined. Should it be mfu_find_best_config_base ?
2) c_part.c:189 gettid() is not defined on MacOSX, but apparently the whole function get_os_thread_self_id() is not used, so this call is unnecassary.
3) c_part.c:230 stat struct on MacOSX (and BSD I guess) is defined inside "sys/stat.h" which is not included. According to http://linux.die.net/man/2/stat this include should work on Linux as well.

With those fixes I've managed to compile it on MacOSX, but cannot tell if it works correctly (there are some differences with bitstream output form mpg123, but I can't tell if they are expected or not).

Best,
Piotr

I've tried to make it compilable on non-Windows platforms, but I don't actually test them. If it compiles with only those 3 changes, I'm impressed! However, note that the SSE optimization is not used outside of Windows.

It sounds like you made the right changes:
1) I renamed mfu_find_best_config to mfu_find_best_config_base a while back, so yes, that should change.
2) That function was used in an old version of the multithreaded code, but it was too slow so I used a different algorithm. I guess I left the old OS hooks in, though.
3) Since I don't compile the non-Windows ones I sometimes miss a few includes, and it clearly looks like that should have been included.

What sort of differences are there between the outputs? If everything works perfectly there should be no differences in the uncompressed output. However, one thing I learned very quickly in this project is that encoders sometimes do some quirky things that are handled differently by different decoders, so if they don't match up it could be one of many possibilities (most of which are benign).

[pwiecz]

What sort of differences are there between the outputs? If everything works perfectly there should be no differences in the uncompressed output. However, one thing I learned very quickly in this project is that encoders sometimes do some quirky things that are handled differently by different decoders, so if they don't match up it could be one of many possibilities (most of which are benign).

I'm not sure how to debug differences.
At first I've just compared pcm files output from mpg123 -O file.pcm file.mp3 and could see some differences.
They were very small when the file was packed without -z option, and very substantial when I'd used -z.

Later I ran foobar's bitcomparator under wine, and it said that the bitstreams from original mp3 and the mp3 produced without -z were identical.
It still claimed there was a difference between the original mp3 and mp3 produced using the -z option.

I've tried it on a file produced by lame 3.99.5 form a cd rip.

[Porcus]

Later I ran foobar's bitcomparator under wine, and it said that the bitstreams from original mp3 and the mp3 produced without -z were identical.
It still claimed there was a difference between the original mp3 and mp3 produced using the -z option.

I've experienced this as well. Don't bother to find the posting, but Omion pointed out to me that the peak difference was ... -150 dB I think? That is likely a roundoff thing. Annoying if you are looking for "No differences" and have to look up every one of them, but likely to be transparent

[hidn]

Great Job. It's pretty fast now.

[Ojay]

I don't get the compilation under 64-bit Linux (pure 64-bit) with Ocaml 3.12.1:

CODE

ocamldep *.mli *.ml > .depend
File "mp3frameutils.ml", line 127, characters 26-30:
Error: Syntax error
File "mp3info.ml", line 119, characters 28-32:
Error: Syntax error
File "mp3queue.ml", line 149, characters 1-5:
Error: Syntax error
File "mp3read.ml", line 93, characters 28-32:
Error: Syntax error
File "types.ml", line 414, characters 5-6:
Error: Syntax error
make: *** [depend] Error 2

So what could that be?

[bugbear]

I don't get the compilation under 64-bit Linux (pure 64-bit) with Ocaml 3.12.1:

CODE

ocamldep *.mli *.ml > .depend
File "mp3frameutils.ml", line 127, characters 26-30:
Error: Syntax error
File "mp3info.ml", line 119, characters 28-32:
Error: Syntax error
File "mp3queue.ml", line 149, characters 1-5:
Error: Syntax error
File "mp3read.ml", line 93, characters 28-32:
Error: Syntax error
File "types.ml", line 414, characters 5-6:
Error: Syntax error
make: *** [depend] Error 2

So what could that be?

I've just had this (compiling for Ubuntu). Solution: You need ocaml 4. To install this, I had to unistall all the ocaml packages, and compile/install from http://caml.inria.fr/.

There were also a couple of errors AFTER that, but earlier posts in this thread showed how to fix them.

Sadly for me, having done all this, my purpose was not served. I was hoping to use mp3 repacker in conjunction with projectX to get "real" mp3 files; projectX spawns mp2 files from DVB streams.

http://project-x.sourceforge.net/

But mp3 repacker does not repack mp2 (obvious in hindsight?)

BugBear

[bugbear]

Here are my diffs:

diff -r mp3packer-2.04/c_part.c pack_orig/mp3packer-2.04/c_part.c
21,23d20
< #include <sys/stat.h>
< #include <unistd.h>
<
192c189
< // out_val = Val_int(gettid());
---
> out_val = Val_int(gettid());
diff -r mp3packer-2.04/mp3framehuffman-c.c pack_orig/mp3packer-2.04/mp3framehuffman-c.c
337c337
< #if HAS_BYTESWAP
---
> #if 1
diff -r mp3packer-2.04/mp3frameutils-c.c pack_orig/mp3packer-2.04/mp3frameutils-c.c
1260c1260
< return mfu_find_best_config_base(quant_bits_ptr, quant_bits_count1_char_ptr, scf_bands_ptr, quant_raw_ptr, debug_val);
---
> return mfu_find_best_config(quant_bits_ptr, quant_bits_count1_char_ptr, scf_bands_ptr, quant_raw_ptr, debug_val);

Does anyone know a lossless converter from mp2 to mp3?

BugBear

[mjb2006]

(Question answered in separate thread.) Repacking MP2 frames to reallocate data among frames, as is done for MP3s, might be possible, but it is not possible to do it in a way that makes the MP2 audio data become MP3 audio data.

[timcupery]

I've found a possible bug that I haven't seen described anywhere.
When converting an mp3 file with max frame size of 112kbps, to CBR, mp3packer will nevertheless convert the file to 128kbps instead of 112kbps. It even works with files that start out as 112kbps CBR: if I process them to VBR, and then back to CBR, the resulting CBR file is 128kbps. It seems that 112kbps is "skipped over" amongst CBR options; I haven't noticed this happening for any other mp3 bitrate.

This happens with older versions of mp3packer, and also with 2.04.

Thread here

This post has been edited by timcupery: Feb 21 2013, 20:22

[JCZorkmid]

Here are my diffs:

Thanks for these! I was able to compile mp3packer64 on OS X 10.8.2.

I saw you said the code using 'gettid' wasn't used, but if it's needed this should work on Mac OS X (you'll need to include pthread.h):

CODE

out_val = pthread_mach_thread_np(pthread_self());

Anyone have a test file (before/after) I can compare the output with to be sure things are working as expected?