Unfortunately the file is 12MB, but I've managed to upload it to the RapidShare. I hope that it will help you debug the problem. BTW, I should mention it before - thank you for mp3packer, it's great tool!

Ps. and here comes the link -> suspicious mp3 file

Thanks for the file. It looks like the problem is a buffer overrun, right where I added the code to protect against a buffer underrun in 1.02. The odd thing is that the file looks perfectly OK, but it's requesting 9 bytes from the frame following the problem frame. Then the next frame wants 469 bytes from the bit reservoir... the file just seems to be screwed up right there, and my program is panicking.

I just checked, and Foobar will skip to the next song at 6:05, which is right where the problem frame occurs. [edit: I just checked again, and Foobar is no longer skipping to the next song there... odd]
No other mp3 checker programs would have caught it, because I don't think they check the bit reservoir for invalid stuff.

So I'll do a check for buffer overruns, and put up a warning and pad with garbage if that happens (just like I do with underruns)


@psyllium:
Your /dev/shm comment pushed me over the edge...
I got my diskless server running Ubuntu Linux, and it's quite awesome. I make it automatically mount a ramdisk on startup, fill it up with my web site, and synchronize new files every few hours. All without 3rd party tools. MAN, Linux is powerful.
The only thing I miss from Windows is filesystem-based compression like NTFS has. That would have saved me a lot of headaches. Well, it's only a matter of time before some Linux FS supports it.

Thanks Omion for your help. I'm looking forward for new version of mp3packer.

@Omion:

http://www.cenatek.com/product_ramdisk.cfm

ramdisks up to 3GB, there is a comprehensive Users manual to download from that web page...

Costs like $50 for a license, unfortunately.

Allrighty, 1.04's out.

Fixed bukem's issue, so it will now put up a warning that something is wrong instead of crashing. Note that there will still be a minor glitch where the error occurred.

Also tweaked the silent-frame detection algorithm, and made it optional. If you have a song which has a lot of silence in it, but mp3packer isn't doing anything to it, use the -z option. The reason I made it an option is that I don't know if my criteria for silent frames will actually include some non-silent frames.

There was another obscure LAME/XING tag related bug which is also squashed.


@kerminen:
Firon pointed out why I didn't consider that. I actually had a pretty good ramdisk for Windows, but it was also a trial to a $50 program. In the end, Linux is free.

I'm still having some issues with Linux (well, I think it's my CF card) and wireless support is no good, but I think I'll stick with Linux, at least on that computer. It just feels better.

@Omion:

Sorry for bothering you again, but I've found another problematic mp3 files:

1. Fatal error: exception Mp3types.Too_many_bytes case:

Again, the suspicious file seems to have problems with the bit reservoir, but nevertheless to implemented by you buffer checks it's crashing mp3packer v1.04-126. Here you have output:



2. Fatal error: exception End_of_file case:

That file this time crash the mp3packer completely.



FYI -> mp3trim doesn't report any problems with these files.

Is this feature planned to be included in the future?
Thank you very much.

@kevinsham (and others):
I should be able to add original-file renaming to the next version.


I think I know the problems with both files. The first one appears to have been cut improperly (hence the buffer underflow warning and negative padding). The problem is that whereas the original file could put those bytes in the previous (non-existant) frame, mp3packer has no such luxury and panics because there are too many bytes.

The second problem is due to the XING frame not having CRC data, but the rest of the frames do. My program reads the XING tag, assumes that all valid frames have no CRC data, and throws out the rest of the frames.

I should be able to fix both problems tomorrow.

All right. Got 1.05 out.

I added the -u switch to do original-file renaming as many have suggested. For example:
will rename file.mp3 to file-orig.mp3, then save the packed version as file.mp3.

Note that if you use the -u option and leave off the output file, the original file will have "-vbr" appended to it, instead of the new file. This is the default from before, but it doesn't make much sense with -u. So always either specify the output file or use something like "-a -orig" to override the appending.

I also sort of fixed the problems bukem was having. The second file now works flawlessly, but the first file is still a bit odd. There are at least 3 things wrong with it:

1. The XING tag has no CRC data, but the rest of the frames do. This was worked around in 1.05.
2. The XING tag has some LAME info that Foobar reads, but my program doesn't because the internal LAME CRC is not there (this is a different CRC from the one mentioned in #1) The result of this is that Foobar will trim 576 samples off the original file, but not from the repacked one, resulting in "bit compare tracks" resulting in quite a large failure.
3. Even if I manually fixed that in the file, the resulting files are a tiny bit different. I have NO idea where this is coming from. The maximum difference is -135dB, so it will be impossible to hear, but that means that something along the way is lossy for ONLY this one file. I really don't know why.

Everything else should be OK though.

@Omion:

Thanks again. I'm going to check new version of mp3packer ASAP and let you know the results.

Edit: [@Omion] I'm in the middle of the conversion process of ~30 GB mp3 test set. That's step one. Next step is to bit compare source tracks with converted tracks to find any inconsistency - if any I'll let you know.

Very cool. That would really help improve/confirm the stability of mp3packer. Unfortunately, I don't know of any way to automate the "Bit-compare tracks" command from Foobar, so it may take a while.

Also note that if a file reports an error in mp3packer, the conversion will probably not be lossless. However, if there was an error, then the file was "wrong" to begin with, so being bit-exact won't matter. (If you have too much time on your hands, though, you can check to make sure that the lossy parts are only concentrated around the error)

BTW, I found out that the non-exactness of the "I've got a cat" file was due to Foobar. Testing it with 0.9 reports no differences. I had previously used 0.8.3, which apparently had an extremely minor roundoff issue. Which is a relief, because I was pulling my hair out looking at the hex output of mp3packer, trying to see what went wrong. It's a good thing it's not my fault That means that, if you can, use 0.9 to do the comparison.

@Omion



Really good news . I was a bit afraid that mp3packer may lose some bits.



That's not a problem. With f2k you can bit-compare several tracks at once. Just add folder with source mp3's to the playlist and then add folder with converted mp3's to the same playlist, select all tracks, rightclick and choose bit-compare and voila!

BTW. It seems that I've found a small glitch in WinMp3Packer GUI - I have to write simple batch file to convert the test set again. Edit: My fault

Whoa. That is AWESOME. I just keep liking Foobar more and more.

@Omion
Yeah, F2K is the tool you can fall in love with . BTW, I just had to start the conversion process again because of my fault (too drunk?) so I'll update you about results tomorrow.

Yeah, it compares the first half of the tracks with the counterpart in the second half of the list. Works the same for the masstagger's copy tags function.

@Omion

I've found first broken mp3 file in my test set. It's negative padding case again. In fact all tracks from that particular album have the same issue.

Here's output from mp3packer -i:


Edit:
I've found solution in that case -> Fix MP3 Header from F2K did the trick! After that mp3packer has no problems with these tracks.



The strange thing, though, that mp3trim doesn't found any problems with these tracks?

Wow. There's 4MB of MP3 data which just isn't in the file! I'll find out what's going on. Just keep sending me the broken ones

In fact I just found and fixed another album which had the same symptoms like that one before. Fix MP3 Header helped again

Bleh. I found the problem. The XING frame is not parsed as an XING frame (it may be invalid... I'll have to check) and so something happens which causes mp3packer to think that the CRC in each frame is actually the bit-reservoir offset. Hilarity ensues.

I'll get this fixed soon.

EDIT: I fixed the errors all over the place, but now it throws out all the frames. Keep hacking...

I think I fixed the problem. Get 1.06.

The problem, if you really want to know, is that the XING tag's frame does not have a CRC on it, and it also does not specify an encoder. mp3packer sees that it doesn't have an encoder, assumes that it's not an XING frame, then assumes that all other valid frames don't have CRCs either. HOWEVER, I never actually check if the other frames have CRCs, so the CRC is used for the bit reservoir offset. This creates errors all over the place, until there is eventually too much data to even fit into a 320kbps frame. The program then croaks.

Aren't you glad you asked? (oh wait. You didn't ask...)

Omion, how about another nasty mp3 track ? . It fails in mp3packer with too_many_bytes error, but when treat with f2k's fix mp3 header magic happens and mp3packer has no problems with processing that file anymore.

Edit: Do anybody know what exactly f2k's fix mp3 header fuction do? Forget it - I've found answer

Bleh. It's that LAME tag being inconsistant again. The LAME tag header says that it's not original, but the rest of the frames are original. I fixed it, but I don't have time right now to release it, but here's an EXE that should be fixed.

Great! I've just finished mp3packing 37 531 877 KB of mp3 music (4862 tracks) and I saved 166 657 KB which is about 0.44 %. Now I've started bit-comparing source and mp3packed tracks to find any inconsistency (hope not ).

Edit 1:
I've found first inconsistent track ).



Edit 2:
Something is wrong here. Till now in 722 mp3 tracks, f2k bit-compare found 65 files with differences, 125 files failed to compare because of Comparing failed (length mismatch)

And here an example of mp3 with differences:


Edit 3:
Huston, we have a problem -> another 614 files processed and here are results:
83 files - Comparing failed (length mismatch)
109 files - Differences found

Run a few of the files that had errors through mp3packer again and look for any errors that may have occurred. There may have been over/underruns in the files which mp3packer threw out. Your example in "Edit 2" looks like there may have been a bad frame or two.

Also, I don't think I've ever run across a "Comparing failed (length mismatch)" error... What version of Foobar are you using? Could you upload an mp3 that does it? [ edit: Oops. You did upload one. Testing... ]


Actually, I just realized that mp3packer does not give a warning for sync errors, which may make it difficult to see if the error was in the original or in the conversion.

One more thing: you didn't use the -z switch, did you?

Grr.... It looks like Foobar has a much more lax idea of what a LAME header is supposed to look like. The problem with the first file (Scalar - Solar) is not really a problem in either mp3packer or Foobar, but how they interpret the LAME data.

All LAME headers have a CRC at the end, which identifies it as valid. mp3packer checks this before it uses the LAME tag data, but Foobar (apparently) does not. Therefore, Foobar will use the offsets stored in the LAME header whereas mp3packer will not.

The end result of this is that Foobar will start playing the output file 576 samples before the input file. I checked, and this was the only error on Scalar - Solar.


This is a bit of a problem, as I have no intention of making mp3packer read invalid LAME headers, whereas Foobar sees nothing wrong with it. I may just have to give in and make an exception for the offsets...

Probably the best fix right now is to run Foobar's "Fix MP3 Header" on the problem files. It should fix a large majority of the problems. (It is a bit time-consuming, though, as you have to "fix" each file separately)


I'm still checking on the Seatown song... that may be a genuine bug.
[ EDIT: I found out that the original Seatown has a little glitch right where the difference is occurring (25 seconds). If the side info is broken right there, mp3packer and Foobar may have different ways of dealing with it. Conclusion: not a bug in mp3packer (but a bug in the MP3) ]

OK. 1.07 is out. There is now no reliance on the LAME/XING frame header for MP3 information. Also, I "fixed" the issue of the offsets.

I say "fix" because all I did was allow certian types of invalid LAME tags to slip through the CRC detection. If the LAME tag has nothing but offset info, it is treated as a valid LAME tag. If anything else is present and the CRC is incorrect, it is thrown out and only the XING part is used.

It should fix at least bukem's 83 "length mismatch" problems. I don't know about the others.

@Omion:

1. I'll start uploading problematic mp3's at your server ASAP.
2. I used f2k 0.9.3 beta 2 for bit-comparing.
3. I didn't use -z switch for mp3packer.

I'll stay in touch - thanks for your time and work you put into mp3packer.

Edit:
I've found one but maybe very important thing (I don't know why I missed it...) - in most cases when differences where found during bit-comparing they started at 0.0000000 seconds and took around 3500 samples. Interesting, don't you think so?

Hmm. That is interesting. It sounds like it may be due to the MP3 being split improperly, and mp3packer having to make up for that. I'll see what I can come up with.

FYI I compiled an Mac OS X version of mp3packer. It can be downloaded from http://osx.iusethis.com/app/mp3packer

1.08 is out.

I added the -w switch, which will throw out an entire frame even if a buffer error only affects one granule of the frame. The default mode is to only throw out the granule which is bad.
The new switch will change nothing for well-formed mp3 files, but it will match Foobar's broken frame handling for easier verification.

@bukem:
The version that I sent you is the same as this, but has the -w switch hardcoded. Don't upgrade to 1.08 unless you can coax WinMp3Packer to use the -w switch. It will help with the error reports. Thanks!

@KAMiKAZOW:
Thanks for the OSX build! I'm sure it will help Mac users out there. Just one question: did you have any problem compiling it?

All righty. 1.09 is out.

mp3packer will now inform the user if there were any buffer or sync errors. Since the output has changed, WinMP3Packer may not interpret the results correctly (I haven't checked)

If buffer or sync errors occur, the output should look like this:


The big news with this release is that I'm now quite certain that mp3packer is bit-identical for files with no errors. Thanks to bukem, mp3packer has been thouroughly tested with all sorts of input.

Synthesis of bukem's and my results on 4830 files tested with Foobar's Bit Compare Tracks: (bukem, correct me if I'm wrong )
4534 files returned no errors
This is the way it should be.
91 files returned 1 to 3 samples different with peak equal to 0.000000.
This can only be explained by a decoder rounding issue. Having thouroughly looked into the issue, I came to the conclusion that mp3packer could not have done anything to cause this. The different samples tended to be on the order of -140dB, or completely silent even with 24-bit output. All other decoders I tried were 16-bit, and decoded to identical WAV files.
199 files failed to compare due to a "length mismatch" error.
Generally this was due to the encoder not giving the proper number of frames in the LAME header. Foobar saw the different number of frames and concluded that they were different without actually checking the data.
Occasionally the LAME header's CRC was computed incorrectly. Foobar seems to treat it as a LAME header anyway, but mp3packer will throw it out (it is invalid, after all...)
In both cases, using Foobar's Fix MP3 Header before packing resulted in files with identical audio.
6 files actually returned different samples.
I checked every one of the files, and there were no errors due to mp3packer.
* 3 files only came back different in Foobar. No other MP3 decoder that I used showed any differences. They all had invalid big_values on the problem frames, which may have had something to do with it.
* 2 files have broken frames which are handled differently in Foobar and mp3packer.
* 1 file had a bizarre problem where two frames pointed to the same data. Foobar and mp3packer handled that case differently.

It must be noted that all the files which failed comparison were invalid (other than the 91 files which resulted in decoder rounding errors).
Also, the differences were local to the region of the error. Sometimes the offset was incorrect, but all the samples were the same. Sometimes a frame was invalid, but all the samples to either side were identical.

All you've posted is correct - thank you very much Omion. And thank you for detailed error report support in mp3packer.

None at all.

i have tested the both the command line and GUI compile on a CBR of 160 bitrates with less than 1KB reduction of 10~18MB files . what could be the wrong

you can use http://media.libsyn.com/media/japanesepod1...906_jpod101.mp3

It'll only really work on high (much higher than 160) bitrate CBR files. Or CBR files with a lot of silence.

A lot of files don't compress very well, especially below ~250kbps. mp3packer depends on wasted bits, but there tend not to be any at 160kbps.

So nothing's wrong, but there's not much that can be done about some files.

Thanks Omion for making such a useful program! Quite impressive!

It's time for a BIG update!

After a month of stewing over decompressor outputs and tech papers, I finally learned enough about MP3s to make an actual compression optimizer (sort of like rehuff for Vorbis).

Using the new -z switch in 1.10 will enable a brute-force Huffman optimization which slightly improves overall compression at the loss of quite a bit of speed. How much it helps is highly dependant on the encoder. LAME encodes tend to be ~0.02% smaller (ie. don't bother ) but I've gotten a 10% improvement over the normal mp3packer output for some files (unknown encoders...) (*)

The -z switch will not help with MPEG2 / 2.5 audio, or short blocks (I don't really feel the need to include support for them) and it is fairly intolerant to errors. If you see a line that looks like this:
it means that the recompression of that frame was thrown out and the input data was used instead, as though -z was not specified.


Also added was support for directories (at last ) and support for deleting files.

The new switches for deleting files are:

The first is used for every file which did not report a sync or buffer error. If you set it to "out" it will only keep the output file, and delete the input file.
The second is used in the rest of the files (the ones which did have errors). Using "in" will leave only the input file, and not write an output.

A few examples:
This is the default. It keeps both input and output files in all cases.
This will essentially move and repack the error-free files, while leaving the files with errors alone.
Same as previous, but it will move the files with errors and repack the other files in-place.
This will replace every file with its repacked version.

Note that, as stated before, a recompression error is actually just a warning. A file with only recompression errors will be covered under the --keep-ok option.


(*) I actually got an 83% savings over the regular repacker with one file, but the file itself was mostly digital silence put through a braindead encoder. That file was why I originally made the -z switch in 1.04

Very good work, Omion. Now I manage to save a few kbps on every 320 kbps file I have.

Great job Omion! I'm going to test it tonight.

Edit: The links to the newest version of mp3packer doesn't work

Sorry about the hosting issues. Looks like my router is down. Unfortunately I'm at work right now, so I can't do anything, but it should be up in a few hours.

OK. I just got back. The links should work now.

It looks like my router didn't update the dyndns IP, so the DNS entry was wrong. Sorry about that!

This is a really, really awesome update Omion thanks a lot. I appreciate all the effort you put into it, it really is a huge improvement. Many albums that yielded only a couple kb before will give up 20-50 now.

Have you tested the new -z on iTunes encodes? I realize that this isn't a particularly good MP3 encoder, but its output seems to be broken out of the box. As in, loads of recompression errors on a fresh file. EDIT - BUT the recompressed files do pass bit-compare

I haven't tested it yet. I'll get to it as soon as I can.
[edit: yeah, recompression errors galore. I'll fix it as soon as I can (probably late tomorrow)]


@all: it would probably be a good idea to use Foobar to make sure the output with the -z switch is the same as without, at least for now. I'm always surprised by how many encoders break my program

Yes! Full directory packing

@Omion: many Thanks for this program. For me it rapidly became the most important tool to directly edit MP3 files without needing to decode them. This has reasons and I just want to point you to a possibility how to use this program you probably haven't heard about.

I am a little bit of a 5.1 surround sound whore at the moment and I am converting all my MP3 files (mostly music radio shows especially from internet streaming) to MP3 Surround. That is done with the new Fraunhofer MP3SX converter and the final 5.1 quality is quite impressive (at least for me). The converter adds a third channel to any stereo MP3 file (the surround channel) without decoding just by rebuilding the MP3 bitstream. However, all the stereo files I own are CBR-encoded files only. This means that the third surround channel adds some bitrate to the file to store the surround info (actually always a constant amount of 15kbps for each file) but - unfortunately- the upgraded file needs to fit the CBR specifications (only 128,160,192,224,256 and 320kbps are defined for CBR). This means that a 192kbps CBR file will become a 224kbps CBR MP3 file after adding the surround channel. So 17kbps of these additional 32kbps are wasted as the surround data would fit into 15kbps.

At that point it is really useful to use mp3packer (before converting stereo MP3 to MP3 Surround). That removes any buffer and sync errors (required for the conversion as internet streams always show some errors and the converter does not accept MP3 streams with errors), makes the files slightly smaller (of course), and converts them to VBR. This conversion to VBR is especially charming as the MP3SX converter now does not need to increase the file size in 32kbps steps as before but just adds only the really needed 15kbps to the VBR file.

For a 2hour radio show that finally saves about 18MB (!) of hard disk space for a VBR file when compared to CBR.

So, many thanks again for this program!

(Edit: would be nice if mp3packer would recognize the MP3Surround structures in some way - maybe even smaller file sizes might be possible)

@Ojay: Thanks for the response!


I was thinking about adding that, but up until your post I didn't know if there was a free way to create MP3 surround bitstreams. I'll see if it's easy to make mp3packer not delete the extra info.

I also found out what was wrong with the iTunes encodes, although I'm not sure why it didn't happen with other encoders. I'll fix it when I get back from work. (yes, I work on Saturday...)

1.11 is out, and should fix the iTunes issue.

The problem was that if the two highest frequency bands actually contained data, there was a 50% chance that writing that data would result in an out-of-bounds array access. It was fixed by forcing everything to the 50% that worked

Only iTunes triggered it because apparently only iTunes thinks that tiny values of >22KHz audio is worth saving. Everything with a working psy model throws it out

@Omion

It seems that links doesn't work again