MP3 repacker Options

[bukem]

Unfortunately the file is 12MB, but I've managed to upload it to the RapidShare. I hope that it will help you debug the problem. BTW, I should mention it before - thank you for mp3packer, it's great tool!

Ps. and here comes the link -> suspicious mp3 file

[Omion]

Thanks for the file. It looks like the problem is a buffer overrun, right where I added the code to protect against a buffer underrun in 1.02. The odd thing is that the file looks perfectly OK, but it's requesting 9 bytes from the frame following the problem frame. Then the next frame wants 469 bytes from the bit reservoir... the file just seems to be screwed up right there, and my program is panicking.

I just checked, and Foobar will skip to the next song at 6:05, which is right where the problem frame occurs. [edit: I just checked again, and Foobar is no longer skipping to the next song there... odd]
No other mp3 checker programs would have caught it, because I don't think they check the bit reservoir for invalid stuff.

So I'll do a check for buffer overruns, and put up a warning and pad with garbage if that happens (just like I do with underruns)


@psyllium:
Your /dev/shm comment pushed me over the edge...
I got my diskless server running Ubuntu Linux, and it's quite awesome. I make it automatically mount a ramdisk on startup, fill it up with my web site, and synchronize new files every few hours. All without 3rd party tools. MAN, Linux is powerful.
The only thing I miss from Windows is filesystem-based compression like NTFS has. That would have saved me a lot of headaches. Well, it's only a matter of time before some Linux FS supports it.

This post has been edited by Omion: Jun 25 2006, 10:26

[bukem]

Thanks Omion for your help. I'm looking forward for new version of mp3packer.

[kerminen]

@Omion:

http://www.cenatek.com/product_ramdisk.cfm

ramdisks up to 3GB, there is a comprehensive Users manual to download from that web page...

[Firon]

Costs like $50 for a license, unfortunately.

[Omion]

Allrighty, 1.04's out.

Fixed bukem's issue, so it will now put up a warning that something is wrong instead of crashing. Note that there will still be a minor glitch where the error occurred.

Also tweaked the silent-frame detection algorithm, and made it optional. If you have a song which has a lot of silence in it, but mp3packer isn't doing anything to it, use the -z option. The reason I made it an option is that I don't know if my criteria for silent frames will actually include some non-silent frames.

There was another obscure LAME/XING tag related bug which is also squashed.


@kerminen:
Firon pointed out why I didn't consider that. I actually had a pretty good ramdisk for Windows, but it was also a trial to a $50 program. In the end, Linux is free.

I'm still having some issues with Linux (well, I think it's my CF card) and wireless support is no good, but I think I'll stick with Linux, at least on that computer. It just feels better.

[bukem]

@Omion:

Sorry for bothering you again, but I've found another problematic mp3 files:

1. Fatal error: exception Mp3types.Too_many_bytes case:

Again, the suspicious file seems to have problems with the bit reservoir, but nevertheless to implemented by you buffer checks it's crashing mp3packer v1.04-126. Here you have output:

CODE

D:\tmp>mp3packer -i "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3'
INFO:
MPEG1 layer 3
7489 frames
44100 Hz
38.281250 frames per second
195.631020 seconds
6260727 bytes in file (256.021851 kbps)
6260192 bytes in MP3 frames (255.999973 kbps) = current bitrate
47901297 bits of payload data (244.855325 kbps)
5990964 bytes of payload data (244.990349 kbps)
26415 bits wasted from partially-full bytes (0.135025 kbps)
6260568 bytes of MP3 data (256.015349 kbps) = minimum bitrate possible
-376 bytes of padding (-0.015376 kbps)
535 bytes outside MP3 frames (0.021878 kbps)
Bitrate distribution:
  256: 612,6877
Largest frame uses 8704 bits = 1088 bytes = 333.200000 kbps
Smallest bitrate for CBR is 320

D:\tmp>mp3packer "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3' -> '02. the method - i've got a cat-vbr.mp3'
0% done on frame 0
Buffer underflow; added 451 zeros to the beginning of frame 0
Fatal error: exception Mp3types.Too_many_bytes

2. Fatal error: exception End_of_file case:

That file this time crash the mp3packer completely.

CODE

D:\tmp>mp3packer -i "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3'
Fatal error: exception End_of_file

D:\tmp>mp3packer "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3' -> '01. stylin' - drifting sun-vbr.mp3'
Fatal error: exception End_of_file

FYI -> mp3trim doesn't report any problems with these files.

[kevinsham]

A feature request:
Rename the original file and use the input file name as output.

Ah yes.
As in: shove '-orig' on the end of the original input file, and thus the resulting output file can have the original's original name (follow me!?).

Is this feature planned to be included in the future?
Thank you very much.

[Omion]

@kevinsham (and others):
I should be able to add original-file renaming to the next version.

CODE

D:\tmp>mp3packer -i "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3'
-376 bytes of padding (-0.015376 kbps)

D:\tmp>mp3packer "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3' -> '02. the method - i've got a cat-vbr.mp3'
0% done on frame 0
Buffer underflow; added 451 zeros to the beginning of frame 0
Fatal error: exception Mp3types.Too_many_bytes

CODE

D:\tmp>mp3packer -i "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3'
Fatal error: exception End_of_file

D:\tmp>mp3packer "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3' -> '01. stylin' - drifting sun-vbr.mp3'
Fatal error: exception End_of_file

I think I know the problems with both files. The first one appears to have been cut improperly (hence the buffer underflow warning and negative padding). The problem is that whereas the original file could put those bytes in the previous (non-existant) frame, mp3packer has no such luxury and panics because there are too many bytes.

The second problem is due to the XING frame not having CRC data, but the rest of the frames do. My program reads the XING tag, assumes that all valid frames have no CRC data, and throws out the rest of the frames.

I should be able to fix both problems tomorrow.

This post has been edited by Omion: Jul 15 2006, 16:01

[Omion]

All right. Got 1.05 out.

I added the -u switch to do original-file renaming as many have suggested. For example:

CODE

mp3packer -u file.mp3 file-orig.mp3

will rename file.mp3 to file-orig.mp3, then save the packed version as file.mp3.

Note that if you use the -u option and leave off the output file, the original file will have "-vbr" appended to it, instead of the new file. This is the default from before, but it doesn't make much sense with -u. So always either specify the output file or use something like "-a -orig" to override the appending.

I also sort of fixed the problems bukem was having. The second file now works flawlessly, but the first file is still a bit odd. There are at least 3 things wrong with it:

1. The XING tag has no CRC data, but the rest of the frames do. This was worked around in 1.05.
2. The XING tag has some LAME info that Foobar reads, but my program doesn't because the internal LAME CRC is not there (this is a different CRC from the one mentioned in #1) The result of this is that Foobar will trim 576 samples off the original file, but not from the repacked one, resulting in "bit compare tracks" resulting in quite a large failure.
3. Even if I manually fixed that in the file, the resulting files are a tiny bit different. I have NO idea where this is coming from. The maximum difference is -135dB, so it will be impossible to hear, but that means that something along the way is lossy for ONLY this one file. I really don't know why.

Everything else should be OK though.

This post has been edited by Omion: Jul 20 2006, 06:44

[bukem]

@Omion:

Thanks again. I'm going to check new version of mp3packer ASAP and let you know the results.

Edit: [@Omion] I'm in the middle of the conversion process of ~30 GB mp3 test set. That's step one. Next step is to bit compare source tracks with converted tracks to find any inconsistency - if any I'll let you know.

This post has been edited by bukem: Jul 20 2006, 18:50

[Omion]

[@Omion] I'm in the middle of the conversion process of ~30 GB mp3 test set. That's step one. Next step is to bit compare source tracks with converted tracks to find any inconsistency - if any I'll let you know.

Very cool. That would really help improve/confirm the stability of mp3packer. Unfortunately, I don't know of any way to automate the "Bit-compare tracks" command from Foobar, so it may take a while.

Also note that if a file reports an error in mp3packer, the conversion will probably not be lossless. However, if there was an error, then the file was "wrong" to begin with, so being bit-exact won't matter. (If you have too much time on your hands, though, you can check to make sure that the lossy parts are only concentrated around the error)

BTW, I found out that the non-exactness of the "I've got a cat" file was due to Foobar. Testing it with 0.9 reports no differences. I had previously used 0.8.3, which apparently had an extremely minor roundoff issue. Which is a relief, because I was pulling my hair out looking at the hex output of mp3packer, trying to see what went wrong. It's a good thing it's not my fault That means that, if you can, use 0.9 to do the comparison.

[bukem]

@Omion

BTW, I found out that the non-exactness of the "I've got a cat" file was due to Foobar. Testing it with 0.9 reports no differences. I had previously used 0.8.3, which apparently had an extremely minor roundoff issue.

Really good news . I was a bit afraid that mp3packer may lose some bits.

Unfortunately, I don't know of any way to automate the "Bit-compare tracks" command from Foobar, so it may take a while.

That's not a problem. With f2k you can bit-compare several tracks at once. Just add folder with source mp3's to the playlist and then add folder with converted mp3's to the same playlist, select all tracks, rightclick and choose bit-compare and voila!

BTW. It seems that I've found a small glitch in WinMp3Packer GUI - I have to write simple batch file to convert the test set again. Edit: My fault

This post has been edited by bukem: Jul 21 2006, 00:12

[Omion]

That's not a problem. With f2k you can bit-compare several tracks at once. Just add folder with source mp3's to the playlist and then add folder with converted mp3's to the same playlist, select all tracks, rightclick and choose bit-compare and voila!

Whoa. That is AWESOME. I just keep liking Foobar more and more.

[bukem]

@Omion
Yeah, F2K is the tool you can fall in love with . BTW, I just had to start the conversion process again because of my fault (too drunk?) so I'll update you about results tomorrow.

This post has been edited by bukem: Jul 21 2006, 00:17

[Firon]

Yeah, it compares the first half of the tracks with the counterpart in the second half of the list. Works the same for the masstagger's copy tags function.

[bukem]

@Omion

I've found first broken mp3 file in my test set. It's negative padding case again. In fact all tracks from that particular album have the same issue.

Here's output from mp3packer -i:

CODE

*** '01. fatboy slim - right here, right now.mp3'
INFO:
MPEG1 layer 3
14851 frames
44100 Hz
38.281250 frames per second
387.944490 seconds
9091943 bytes in file (187.489566 kbps)
9091413 bytes in MP3 frames (187.478637 kbps) = current bitrate
106667278 bits of payload data (274.955002 kbps)
13339229 bytes of payload data (275.075004 kbps)
46554 bits wasted from partially-full bytes (0.120002 kbps)
13873865 bytes of MP3 data (286.100004 kbps) = minimum bitrate possible
-4782452 bytes of padding (-98.621367 kbps)
530 bytes outside MP3 frames (0.010929 kbps)
Bitrate distribution:
   32: 9,0
   48: 1,0
  128: 898,0
  160: 5798,0
  192: 5435,0
  224: 1253,0
  256: 718,0
  320: 739,0
Largest frame uses 15488 bits = 1936 bytes = 592.900000 kbps
Fatal error: exception Mp3types.Too_many_bytes

Edit:
I've found solution in that case -> Fix MP3 Header from F2K did the trick! After that mp3packer has no problems with these tracks.

CODE

*** '01. fatboy slim - right here, right now.mp3.mp3'
INFO:
MPEG1 layer 3
14850 frames
44100 Hz
38.281250 frames per second
387.918367 seconds
9091995 bytes in file (187.503264 kbps)
9091257 bytes in MP3 frames (187.488044 kbps) = current bitrate
68159667 bits of payload data (175.706212 kbps)
8526466 bytes of payload data (175.840418 kbps)
52061 bits wasted from partially-full bytes (0.134206 kbps)
9061066 bytes of MP3 data (186.865418 kbps) = minimum bitrate possible
30191 bytes of padding (0.622626 kbps)
738 bytes outside MP3 frames (0.015220 kbps)
Bitrate distribution:
   32: 9,0
  128: 898,0
  160: 5798,0
  192: 5435,0
  224: 1253,0
  256: 718,0
  320: 739,0
Largest frame uses 8918 bits = 1115 bytes = 341.392187 kbps
Smallest bitrate for CBR is 320

The strange thing, though, that mp3trim doesn't found any problems with these tracks?

This post has been edited by bukem: Jul 21 2006, 02:03

[Omion]

Wow. There's 4MB of MP3 data which just isn't in the file! I'll find out what's going on. Just keep sending me the broken ones

This post has been edited by Omion: Jul 21 2006, 02:27

[bukem]

In fact I just found and fixed another album which had the same symptoms like that one before. Fix MP3 Header helped again

This post has been edited by bukem: Jul 21 2006, 02:33

[Omion]

Bleh. I found the problem. The XING frame is not parsed as an XING frame (it may be invalid... I'll have to check) and so something happens which causes mp3packer to think that the CRC in each frame is actually the bit-reservoir offset. Hilarity ensues.

I'll get this fixed soon.

EDIT: I fixed the errors all over the place, but now it throws out all the frames. Keep hacking...

This post has been edited by Omion: Jul 21 2006, 03:07

[Omion]

I think I fixed the problem. Get 1.06.

The problem, if you really want to know, is that the XING tag's frame does not have a CRC on it, and it also does not specify an encoder. mp3packer sees that it doesn't have an encoder, assumes that it's not an XING frame, then assumes that all other valid frames don't have CRCs either. HOWEVER, I never actually check if the other frames have CRCs, so the CRC is used for the bit reservoir offset. This creates errors all over the place, until there is eventually too much data to even fit into a 320kbps frame. The program then croaks.

Aren't you glad you asked? (oh wait. You didn't ask...)

[bukem]

Omion, how about another nasty mp3 track ? . It fails in mp3packer with too_many_bytes error, but when treat with f2k's fix mp3 header magic happens and mp3packer has no problems with processing that file anymore.

Edit: Do anybody know what exactly f2k's fix mp3 header fuction do? Forget it - I've found answer

This post has been edited by bukem: Jul 21 2006, 14:58

[Omion]

Bleh. It's that LAME tag being inconsistant again. The LAME tag header says that it's not original, but the rest of the frames are original. I fixed it, but I don't have time right now to release it, but here's an EXE that should be fixed.

[bukem]

Great! I've just finished mp3packing 37 531 877 KB of mp3 music (4862 tracks) and I saved 166 657 KB which is about 0.44 %. Now I've started bit-comparing source and mp3packed tracks to find any inconsistency (hope not ).

Edit 1:
I've found first inconsistent track ).

CODE

Comparing:
"R:\music\..good looking records\.other\studio x\01. scalar - solar.mp3"
"D:\documents\music\.streams\music\..good looking records\.other\studio x\01. scalar - solar.mp3"
Comparing failed (length mismatch).

Edit 2:
Something is wrong here. Till now in 722 mp3 tracks, f2k bit-compare found 65 files with differences, 125 files failed to compare because of Comparing failed (length mismatch)

And here an example of mp3 with differences:

CODE

Comparing:
"R:\music\..good looking records\.other\.singles\looking good - (lgr019) 01. moonchild - seatown.mp3"
"D:\documents\music\.streams\music\..good looking records\.other\.singles\looking good - (lgr019) 01. moonchild - seatown.mp3"
differences found: 3255 sample(s), starting at 25.5749887 second(s), peak: 0.1141328 at 25.5886621 second(s), 2ch

Edit 3:
Huston, we have a problem -> another 614 files processed and here are results:
83 files - Comparing failed (length mismatch)
109 files - Differences found

This post has been edited by bukem: Jul 22 2006, 20:43

[Omion]

Huston, we have a problem -> another 614 files processed and here are results:
83 files - Comparing failed (length mismatch)
109 files - Differences found

Run a few of the files that had errors through mp3packer again and look for any errors that may have occurred. There may have been over/underruns in the files which mp3packer threw out. Your example in "Edit 2" looks like there may have been a bad frame or two.

Also, I don't think I've ever run across a "Comparing failed (length mismatch)" error... What version of Foobar are you using? Could you upload an mp3 that does it? [ edit: Oops. You did upload one. Testing... ]


Actually, I just realized that mp3packer does not give a warning for sync errors, which may make it difficult to see if the error was in the original or in the conversion.

One more thing: you didn't use the -z switch, did you?

This post has been edited by Omion: Jul 23 2006, 04:10