MP3 repacker

Topic: MP3 repacker (Read 594517 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Reply #175 – 2006-07-23 05:37:37

Grr.... It looks like Foobar has a much more lax idea of what a LAME header is supposed to look like. The problem with the first file (Scalar - Solar) is not really a problem in either mp3packer or Foobar, but how they interpret the LAME data.

All LAME headers have a CRC at the end, which identifies it as valid. mp3packer checks this before it uses the LAME tag data, but Foobar (apparently) does not. Therefore, Foobar will use the offsets stored in the LAME header whereas mp3packer will not.

The end result of this is that Foobar will start playing the output file 576 samples before the input file. I checked, and this was the only error on Scalar - Solar.

This is a bit of a problem, as I have no intention of making mp3packer read invalid LAME headers, whereas Foobar sees nothing wrong with it. I may just have to give in and make an exception for the offsets...

Probably the best fix right now is to run Foobar's "Fix MP3 Header" on the problem files. It should fix a large majority of the problems. (It is a bit time-consuming, though, as you have to "fix" each file separately)

~~I'm still checking on the Seatown song... that may be a genuine bug.~~
[ EDIT: I found out that the original Seatown has a little glitch right where the difference is occurring (25 seconds). If the side info is broken right there, mp3packer and Foobar may have different ways of dealing with it. Conclusion: not a bug in mp3packer (but a bug in the MP3) ]

MP3 repacker

Reply #176 – 2006-07-23 07:35:36

OK. 1.07 is out. There is now no reliance on the LAME/XING frame header for MP3 information. Also, I "fixed" the issue of the offsets.

I say "fix" because all I did was allow certian types of invalid LAME tags to slip through the CRC detection. If the LAME tag has nothing but offset info, it is treated as a valid LAME tag. If anything else is present and the CRC is incorrect, it is thrown out and only the XING part is used.

It should fix at least bukem's 83 "length mismatch" problems. I don't know about the others.

MP3 repacker

Reply #177 – 2006-07-23 16:08:17

@Omion:

1. I'll start uploading problematic mp3's at your server ASAP.
2. I used f2k 0.9.3 beta 2 for bit-comparing.
3. I didn't use -z switch for mp3packer.

I'll stay in touch - thanks for your time and work you put into mp3packer.

Edit:
I've found one but maybe very important thing (I don't know why I missed it...) - in most cases when differences where found during bit-comparing they started at 0.0000000 seconds and took around 3500 samples. Interesting, don't you think so?

MP3 repacker

Reply #178 – 2006-07-23 18:01:20

Quote from: bukem on 2006-07-23 16:08:17

Edit:
I've found one but maybe very important thing (I don't know why I missed it...) - in most cases when differences where found during bit-comparing they started at 0.0000000 seconds and took around 3500 samples. Interesting, don't you think so?

Hmm. That is interesting. It sounds like it may be due to the MP3 being split improperly, and mp3packer having to make up for that. I'll see what I can come up with.

MP3 repacker

Reply #179 – 2006-07-23 20:27:45

FYI I compiled an Mac OS X version of mp3packer. It can be downloaded from http://osx.iusethis.com/app/mp3packer

MP3 repacker

Reply #180 – 2006-07-27 22:39:54

1.08 is out.

I added the -w switch, which will throw out an entire frame even if a buffer error only affects one granule of the frame. The default mode is to only throw out the granule which is bad.
The new switch will change nothing for well-formed mp3 files, but it will match Foobar's broken frame handling for easier verification.

@bukem:
The version that I sent you is the same as this, but has the -w switch hardcoded. Don't upgrade to 1.08 unless you can coax WinMp3Packer to use the -w switch. It will help with the error reports. Thanks!

@KAMiKAZOW:
Thanks for the OSX build! I'm sure it will help Mac users out there. Just one question: did you have any problem compiling it?

MP3 repacker

Reply #181 – 2006-08-07 06:13:47

All righty. 1.09 is out.

mp3packer will now inform the user if there were any buffer or sync errors. Since the output has changed, WinMP3Packer may not interpret the results correctly (I haven't checked)

If buffer or sync errors occur, the output should look like this:

Code: [Select]

*** 'a.mp3' -> 'b.mp3'
WARNING: sync error; expected frame 4090 at 3018491
WARNING: buffer overflow; frame 4091 will be truncated
WARNING: sync error; expected frame 6726 at 5024182
100% done with 21686 frames

WARNING: There was 1 buffer error and 2 sync errors

The big news with this release is that I'm now quite certain that mp3packer is bit-identical for files with no errors. Thanks to bukem, mp3packer has been thouroughly tested with all sorts of input.

Synthesis of bukem's and my results on 4830 files tested with Foobar's Bit Compare Tracks: (bukem, correct me if I'm wrong )
4534 files returned no errors
This is the way it should be.
91 files returned 1 to 3 samples different with peak equal to 0.000000.
This can only be explained by a decoder rounding issue. Having thouroughly looked into the issue, I came to the conclusion that mp3packer could not have done anything to cause this. The different samples tended to be on the order of -140dB, or completely silent even with 24-bit output. All other decoders I tried were 16-bit, and decoded to identical WAV files.
199 files failed to compare due to a "length mismatch" error.
Generally this was due to the encoder not giving the proper number of frames in the LAME header. Foobar saw the different number of frames and concluded that they were different without actually checking the data.
Occasionally the LAME header's CRC was computed incorrectly. Foobar seems to treat it as a LAME header anyway, but mp3packer will throw it out (it is invalid, after all...)
In both cases, using Foobar's Fix MP3 Header before packing resulted in files with identical audio.
6 files actually returned different samples.
I checked every one of the files, and there were no errors due to mp3packer.
* 3 files only came back different in Foobar. No other MP3 decoder that I used showed any differences. They all had invalid big_values on the problem frames, which may have had something to do with it.
* 2 files have broken frames which are handled differently in Foobar and mp3packer.
* 1 file had a bizarre problem where two frames pointed to the same data. Foobar and mp3packer handled that case differently.

It must be noted that all the files which failed comparison were invalid (other than the 91 files which resulted in decoder rounding errors).
Also, the differences were local to the region of the error. Sometimes the offset was incorrect, but all the samples were the same. Sometimes a frame was invalid, but all the samples to either side were identical.

MP3 repacker

Reply #182 – 2006-08-07 18:10:47

All you've posted is correct - thank you very much Omion. And thank you for detailed error report support in mp3packer.

MP3 repacker

Reply #183 – 2006-08-11 09:29:13

Quote from: Omion on 2006-07-27 22:39:54

Just one question: did you have any problem compiling it?

None at all.

MP3 repacker

Reply #184 – 2006-08-20 16:57:53

i have tested the both the command line and GUI compile on a CBR of 160 bitrates with less than 1KB reduction of 10~18MB files . what could be the wrong

you can use http://media.libsyn.com/media/japanesepod1...906_jpod101.mp3

MP3 repacker

Reply #185 – 2006-08-20 18:52:43

It'll only really work on high (much higher than 160) bitrate CBR files. Or CBR files with a lot of silence.

MP3 repacker

Reply #186 – 2006-08-20 19:07:46

A lot of files don't compress very well, especially below ~250kbps. mp3packer depends on wasted bits, but there tend not to be any at 160kbps.

So nothing's wrong, but there's not much that can be done about some files.

MP3 repacker

Reply #187 – 2006-09-23 13:24:32

Thanks Omion for making such a useful program! Quite impressive!

MP3 repacker

Reply #188 – 2006-09-29 04:14:39

It's time for a BIG update!

After a month of stewing over decompressor outputs and tech papers, I finally learned enough about MP3s to make an actual compression optimizer (sort of like rehuff for Vorbis).

Using the new -z switch in 1.10 will enable a brute-force Huffman optimization which slightly improves overall compression at the loss of quite a bit of speed. How much it helps is highly dependant on the encoder. LAME encodes tend to be ~0.02% smaller (ie. don't bother ) but I've gotten a 10% improvement over the normal mp3packer output for some files (unknown encoders...) (*)

The -z switch will not help with MPEG2 / 2.5 audio, or short blocks (I don't really feel the need to include support for them) and it is fairly intolerant to errors. If you see a line that looks like this:

Code: [Select]

WARNING: recompression error on frame 10862

it means that the recompression of that frame was thrown out and the input data was used instead, as though -z was not specified.

Also added was support for directories (at last ) and support for deleting files.

The new switches for deleting files are:

Code: [Select]

--keep-ok (out | both)
--keep-bad (in | out | both)

The first is used for every file which did not report a sync or buffer error. If you set it to "out" it will only keep the output file, and delete the input file.
The second is used in the rest of the files (the ones which did have errors). Using "in" will leave only the input file, and not write an output.

A few examples:

Code: [Select]

--keep-ok both --keep-bad both

This is the default. It keeps both input and output files in all cases.

Code: [Select]

--keep-ok out --keep-bad in

This will essentially move and repack the error-free files, while leaving the files with errors alone.

Code: [Select]

-u --keep-ok out --keep-bad in

Same as previous, but it will move the files with errors and repack the other files in-place.

Code: [Select]

-u --keep-ok out --keep-bad out

This will replace every file with its repacked version.

Note that, as stated before, a recompression error is actually just a warning. A file with only recompression errors will be covered under the --keep-ok option.

(*) I actually got an 83% savings over the regular repacker with one file, but the file itself was mostly digital silence put through a braindead encoder. That file was why I originally made the -z switch in 1.04

MP3 repacker

Reply #189 – 2006-09-29 12:56:13

Very good work, Omion. Now I manage to save a few kbps on every 320 kbps file I have.

MP3 repacker

Reply #190 – 2006-09-29 14:11:16

Great job Omion! I'm going to test it tonight.

Edit: The links to the newest version of mp3packer doesn't work

MP3 repacker

Reply #191 – 2006-09-29 18:04:06

Quote from: bukem on 2006-09-29 14:11:16

The links to the newest version of mp3packer doesn't work

Sorry about the hosting issues. Looks like my router is down. Unfortunately I'm at work right now, so I can't do anything, but it should be up in a few hours.

MP3 repacker

Reply #192 – 2006-09-30 03:42:40

OK. I just got back. The links should work now.

It looks like my router didn't update the dyndns IP, so the DNS entry was wrong. Sorry about that!

MP3 repacker

Reply #193 – 2006-09-30 07:32:43

This is a really, really awesome update Omion thanks a lot. I appreciate all the effort you put into it, it really is a huge improvement. Many albums that yielded only a couple kb before will give up 20-50 now.

Have you tested the new -z on iTunes encodes? I realize that this isn't a particularly good MP3 encoder, but its output seems to be broken out of the box. As in, loads of recompression errors on a fresh file. EDIT - BUT the recompressed files do pass bit-compare

MP3 repacker

Reply #194 – 2006-09-30 07:47:06

Quote from: Gnerma on 2006-09-30 07:32:43

Have you tested the new -z on iTunes encodes? I realize that this isn't a particularly good MP3 encoder, but its output seems to be broken out of the box. As in, loads of recompression errors on a fresh file.

I haven't tested it yet. I'll get to it as soon as I can.
[edit: yeah, recompression errors galore. I'll fix it as soon as I can (probably late tomorrow)]

@all: it would probably be a good idea to use Foobar to make sure the output with the -z switch is the same as without, at least for now. I'm always surprised by how many encoders break my program

MP3 repacker

Reply #195 – 2006-09-30 09:33:02

Yes! Full directory packing

MP3 repacker

Reply #196 – 2006-09-30 11:26:29

@Omion: many Thanks for this program. For me it rapidly became the most important tool to directly edit MP3 files without needing to decode them. This has reasons and I just want to point you to a possibility how to use this program you probably haven't heard about.

I am a little bit of a 5.1 surround sound whore at the moment and I am converting all my MP3 files (mostly music radio shows especially from internet streaming) to MP3 Surround. That is done with the new Fraunhofer MP3SX converter and the final 5.1 quality is quite impressive (at least for me). The converter adds a third channel to any stereo MP3 file (the surround channel) without decoding just by rebuilding the MP3 bitstream. However, all the stereo files I own are CBR-encoded files only. This means that the third surround channel adds some bitrate to the file to store the surround info (actually always a constant amount of 15kbps for each file) but - unfortunately- the upgraded file needs to fit the CBR specifications (only 128,160,192,224,256 and 320kbps are defined for CBR). This means that a 192kbps CBR file will become a 224kbps CBR MP3 file after adding the surround channel. So 17kbps of these additional 32kbps are wasted as the surround data would fit into 15kbps.

At that point it is really useful to use mp3packer (before converting stereo MP3 to MP3 Surround). That removes any buffer and sync errors (required for the conversion as internet streams always show some errors and the converter does not accept MP3 streams with errors), makes the files slightly smaller (of course), and converts them to VBR. This conversion to VBR is especially charming as the MP3SX converter now does not need to increase the file size in 32kbps steps as before but just adds only the really needed 15kbps to the VBR file.

For a 2hour radio show that finally saves about 18MB (!) of hard disk space for a VBR file when compared to CBR.

So, many thanks again for this program!

(Edit: would be nice if mp3packer would recognize the MP3Surround structures in some way - maybe even smaller file sizes might be possible)

MP3 repacker

Reply #197 – 2006-09-30 16:31:44

@Ojay: Thanks for the response!

Quote from: Ojay on 2006-09-30 11:26:29

(Edit: would be nice if mp3packer would recognize the MP3Surround structures in some way - maybe even smaller file sizes might be possible)

I was thinking about adding that, but up until your post I didn't know if there was a free way to create MP3 surround bitstreams. I'll see if it's easy to make mp3packer not delete the extra info.

I also found out what was wrong with the iTunes encodes, although I'm not sure why it didn't happen with other encoders. I'll fix it when I get back from work. (yes, I work on Saturday...)

MP3 repacker

Reply #198 – 2006-10-01 07:51:42

1.11 is out, and should fix the iTunes issue.

The problem was that if the two highest frequency bands actually contained data, there was a 50% chance that writing that data would result in an out-of-bounds array access. It was fixed by forcing everything to the 50% that worked

Only iTunes triggered it because apparently only iTunes thinks that tiny values of >22KHz audio is worth saving. Everything with a working psy model throws it out

MP3 repacker

Reply #199 – 2006-10-01 11:06:08

@Omion

It seems that links doesn't work again

Notice