How did you produce this graph? Very interesting to me as I'm interested a lot in the maximum actual frame size producable by encoders especially Lame 3.90.3 api and -b320. You show already that actual frame size can be more than the size of a 320 kbps frame (minus frame header minus side info).
(I'm interested in differences between Lame 3.90.3 and 3.96.1/3.97b when using cbr320. There is some evidence that 3.90.3 cbr320 provides better quality. 3.96+ doesn't use bit reservoir which restricts actual frame size to the size of a 320 kbps frame minus frame header minus side info. 3.90.3 however does use the bit reservoir but it is unclear with which restriction concerning maximum actual frame size.)

Omion may have used his very own mp3repacker tool with some of the DEBUG options enabled to output the numbers. You can try that yourself, simply edit the file mp3.pm accordingly.


Keep that discussion out this thread, please.
(for other readers, there is a thread about halb27's search for the ultimate way to create "[...] a universally usable mp3 archive not needing any transcoding any more" - using very high bitrates)

I made a Perl script which I use as a debugger for my repacker. It outputs a bunch of garbage about each frame, then makes a few Mathematica-readable lists. I then used Mathematica to create the graphs.

@smack: The debug options in mp3.pm do a similar thing, but they don't output a nice list. It's more of a random spamming .


I don't know about the quality difference, but the fact that newer LAMEs don't use the reservoir for API is why I made the repacker in the first place.For every frame that is less than 320kbps, some space is wasted.

PS. The Perl script I used is here.

Thank you very much. I think it will help me a lot.

Hmm, I seem to be having a strange issue with the repacker.
I was trying to repack some 192kbps CBR files to see the actual frame sizes (not to save any space, I know it won't help) and make it look like nifty VBR files.
However, when I ran it through the repacker, it spit out this error:
ERROR: Can't run frame location on layer 1 or 2 data at mp3.pm line 936.

Not sure what could be causing it (since it's layer III according to foobar and everything else decodes 'em just fine...)
Well, I got the first 4KiB of the file and uploaded it. If you need a bigger clip, I can give you one.
http://firon.site.voila.fr/mp3clip.mp3

I didn't encode these myself, but according to some data at the end of the file, it was made by LAME3.89 (beta)

Very strange. I'll take a look. (and see if I still remember what I was thinking when I wrote that )

[edit]
HA! That's hilarious! It looks like it's all ID3v2's fault... again. Whoever tagged this shoulda used the "unsynchronization" feature...

Short answer: delete (or otherwise mangle) the tag!

Long answer:
The MP3 header sync pattern is 0xFFF. (normally doesn't occur in text)
There is an ID3v2 tag at the beginning of the file. (so far, so good)
That tag uses UTF-16 encoding. (still good)
The Unicode strings all have a 16-bit Unicode "byte order mark" ("BOM", Unicode character U+FEFF)
However, it's little-endian UTF-16 so the character U+FEFF is encoded to the bytes 0xFFFE. (BIG problem)

Now, since the ID3 tag is before everything else in the file, my program scans it for the first sync header. It sees the 0xFFF from the Unicode BOM and parses it as a frame header.

The tag was parsed as an MP1 frame, and since I couldn't find anything on how MP1 or MP2 files store the data I just kill the program when one is encountered. (Nobody here uses MP1 anyway...)

This whole problem could have been avoided if whoever tagged the file used the ID3 unsynchronization scheme. That outputs a 0x00 byte after all 0xFF bytes to break up any false sync headers PRECISELY because of this problem.

Most other programs at least know how to skip over ID3 data, which is why no other programs fail on this. mp3repacker, however, has the attitude that if it ain't an MP3 frame it's junk (and I don't plan on changing that)

So the problem came from a combination of things:
* The file uses an ID3v2 tag
* The tag does not use the feature that breaks up confusing bytes
* The tag uses the EXACT Unicode encoding that results in lots of 0xFFFE strings
* My program has no idea what a "tag" is, or how to skip one when it sees one

My debugger program gives this as output:


The first frame is the one from the header, and you can see that it is different from all the rest.

Well, the tags were made by foobar2000 (I'd retagged the files with it), so blame it I guess.
I use ID3v2 and ID3v1 for compatibility reasons.

I'm not sure what to do. Turn off the BOM and rewrite the tags? It's a lot of files so I can't really delete or mangle the tags.

Which version of the ID3v2-wrinting component do you have? I just downloaded the 1.17 and it only writes the BOM if it's actually using non-ASCII characters. I noticed that the file you uploaded has no reason to be in Unicode, so getting the newest version of the component and re-saving the tag will help in this case.

I would recommend against disabling the BOM, as it looks like it's the only way a program can tell if the data is actually Unicode and it's technically required (*)

You have a few options:
1. Get version 1.17 of the ID3v2 component and hope you don't run into any Unicode
2. Don't use ID3v2 anymore (ID3v1 and APE tags are both OK)
3. Don't use mp3repacker! It probably won't help much on a CBR 192 file
4. Pester whoever made the Foobar ID3v2 component (probably Peter) to support one of the following:
. a. Big-endian Unicode
. b. Unsynchronization padding

Note that although #4 would be the best solution, it is the least likely to go through, as evidenced from the ID3v2 'about' box:

Good luck!

(*) Also, my version puts up a hilarious warning message if you try to disable the BOM. So don't do it.

I've got version 1.20, which I nabbed off kode54's site. Perhaps I'll bother him about #4 on IRC next time I see him
Anyway, I rolled back to 1.17, no more problems. Thanks for the tip.

I don't expect the mp3repacker to really help on these particular CBR files, I mostly wanted to use it for fun (and to see what impact the bit reservoir has on so-called CBR files).

TAKE OFF EVERY 'BOM'!!
YOU KNOW WHAT YOU DOING?

I am really impressed by this program. It converts so much faster than EAC. I am proud of myself simply for figuring how to run PERL. I wish I could contribute to to the cause for batch processing. for now all I can do though is wait for someone really smart to save the day.

Well, it converts faster than EAC because it doesn't really do anything . All my program does is re-arrange the bytes in an MP3 file (and might throw out a few unused ones). Although I am also impressed at how fast it goes. Perl is not known for speed.

As for batch processing, asonicboom posted a way to use an external program to do batch processing in this post. As for actually supporting it in my program, I might do something about it. I haven't programmed Perl for a while, so I'll have to dust off Ye Olde Camle Booke and see how best to support multiple files.

I can't work on it right now because I'm in the middle of an OCaml program, and using 2 programming languages at once just leads to confusion (and many, many bugs).

over at poineerprodjforums they are telling everyone that the only way to convert a VBR mp3 to a CBR version (for use on the pioneer CDJ-200) is if they que them all up in Easy CD Creator and do it all in there.

I sooo want to chime in and say You know what, not only is that slow as hell, but it's just flat out shitty!. If only we had a system for batch-processing a list, or a directory mp3s; and maybe even a GUI.... Oh boy, would I love to go over there and say SHOVE IT!! WE FIGURED OUT A WORK-AROUND FOR YOUR DAMN MACHINES THAT DON'T SUPPORT VBRMP3!!!

for now I will check out asonicboom's method. good job! we'll get there some day.

What are the specifics for getting this 'inserted' into mp3tag? I'd liek to do this, but a wee bit more info would be great.

jaybee:

Make sure you've got Omion's Perl tool in a permanent folder, and get the latest version of Mp3Tag, install it and open it. Go Tools --> Options. Click on the last option in the tree menu 'Tools'. Now click on the icon to the right on the text box. And enter this:

Name: MP3 Packer (or whatever you want)
Path: Navigate to C:\Windows\System32\cmd.exe
Parameter: /c "cd /d "C:\Program Files\Encoding\mp3repack" && perl mp3packer.pl -mst "%_path%" "%_folderpath%\%_filename%-vbr.mp3""

N.B Where mine says 'C:\Program Files\Encoding\mp3repack', just change it to whatever folder you've UnRared the MP3 Packer to.

And check 'For all Selected Files'. Press OK and come out of the menu.

Now change the directory (press CTRL+D) and navigate to a folder of mp3's you want to repack. Select the MP3's, right click on them, go Tools --> MP3 Packer. You'll then get loads of command line terminals come up. Just leave it until it's all finished.

Your new files will have -vbr at the end of the filename.


And thanks Omion, you've solved a problem for me or cutting MP3's gaplessly. Now thanks to Sebi's PCutMP3 and this tool, my MP3's play on my Karma without the faintest gap.

That was a completely unintended benifit, but I suppose it works. I find it kind of funny that I made the tool to turn CBR into VBR files, and now it seems that most people are using it the other way around!

About the batch processing: I think I'll make it so that if you give it a directory as input, it will process all the .mp3 files within it. Then it'll add a specified string to the end of the filenames, like "-vbr". I should have it ready for testing in a few days.

I do turn my CBR's into VBR

The reason I use repacker is because when PCut (and I'm not bashing Sebi's prog here, it's superb) cut's CBR MP3's it actually makes them VBR (the first frame is smaller than the rest CBR bitrate) but doesn't put any Xing header on. This messes up my Karma as it can't figure out the length of the file and gapless playback using the Lame tag padding.

Your MP3 packer, aswell as shedding KB's off, makes the file completely VBR compliant meaning gapless playback works like a charm.



Batch processing sounds good too...

Many thanks senab. Works like a charm.

And thanks for the interesting tool Omion. Just tried it on a 185,809kb 320kbps file and it got it down to a 183,263kb 316kbps file. Not as much as I thought it might, but I'll play with some other files. Mind you, 2.5mb ain't too bad is it.

Oh. Right. I knew that. It's still not quite what I had in mind...

Batch processing is coming along nicely, but I need to get to work soon. Should be finished by tomorrow night.

Do you know what was used to encode it? What kind of music is it? I may have to revise my "~10% off" statement...

[ edit: Just did a big test of ~6 hours of music and I got the files 3% smaller. Time to change the claim on the front page... ]

LAME 3.96 (according to Mr QM)
Hip-hop from DJ Yoda.

Kay, 0.07's out, and adds full-directory processing.

Basically, if you specify a directory as the anput file, it will do all MP3s that are in it (NOT recursive).

By default, it will append "-vbr" to the filename right before the extension, so a.b.c.mp3 will turn into a.b.c-vbr.mp3. Change the appended string with the -a option.

Also, it will skip files which already have the appended string on them. So if you specify "-a monkeys" it will skip the file "lots_of_monkeys.mp3". The idea is that such files have already been processed. You can force processing of all files by using the -A option.

Use the -X option to delete files after they are processed. USE WITH CAUTION! Only do this if you trust me to not delete your music. (I wouldn't! )


The batch processing is by no means perfect. If you cancel processing in the middle and re-run it, it will probably throw an error about "conflicting filenames". In this case. you will need to delete EITHER the previous output files OR the input files which have already benn successfully processed.

[edit]
Just realized, some options don't work when they probably should:
-f doesn't work with batch processing
-X doesn't work with single-file processing
(-a and -A don't work with single-file either, but they're not supposed to)
[/edit]

I think that's about all the warnings I can give out. Good luck!



@jaybeee:
Hmm... I think I'll have to revise my statement about taking off ~10% from API files. Maybe my testing isn't extensive enough.

this is an amazing tool!! I think everyone is too busy converting their mp3s (which takes a lot less time now!) and actually playing the music in their player of choice to make posts about how pleased they are. I expect my pioneer cdj-200 unit in the mail tomorrow, and if it plays all these files that I've converted to CBR from VBR.... you have no idea how happy I will be. My grin will stretch from ear to ear, all day long.

I am also super excited about the GUI for this program. Makes it so easy! Now an even bigger range of folks can use your software. I'm sure that some companies who sell mp3 cd players, or portable mp3 players... whatever... wish they had thought like this beforehand! I'm sure there's money in a market like this. You guys are so rad!

where would we be, on the internet- without long-distance collaberation and open-source software like this? NOT FAR AT ALL!

^^ I'm confused: I thought it made CBR files VBR, not the other way round. At least it does with mine.

Primarily, you are correct. According to Omion's initial post: "Also, using this program on VBR files, or anything below ~200kbps is probably useless."

The single exception I know of, is converting any file (including VBR) to 320k CBR:

perl mp3packer.pl -b320 infile.mp3 outfile.mp3

Regards,
Madrigal

That's the main reason it exists. But as Madrigal says, by forcing the minimum bitrate to 320, you can turn any MP3 into CBR 320. The reason one would do this is to play an MP3 on something which only supports CBR, like, apparently, most MP3 DJ mixing thingies.

Also, ABR files may be able to be turned into CBR at a lower bitrate, but there's no easy way to find out what the lowest CBR bitrate is.

PS. I went on the Pioneer DJ forums to get some more information about this, and all I discovered was how very glad I am that HA exists. A choice quote:

I was so sad when I first read that pioneer forum last week.

now, I am sooooooo glad this thing works!!

if it could deal with sub-folders inside of the main folder... that would be even better. but now I'm just being picky

My GUI front-end for this script is available here: http://www.hydrogenaudio.org/forums/index....showtopic=40780
Features batch processing and folder recursion

Just released 0.08. If you don't use the -b switch, you don't care

Basically there was a discrepancy with handling of CBR files. psyllium noted that inputting a 128kbps file with the "-b 128" switch resulted in a VBR file, not the CBR one would expect. The vast majority of the frames were 128kbps, but ~1% were 160.

As a fix, 0.08 now assumes padded frames for valid bitrates, and unpadded frames for invalid ones. For example:
"-b 129" -> UNpadded 160
"-b 128" -> PADDED 128
"-b 127" -> UNpadded 128
"-b 126" -> UNpadded 128
"-b 113" -> UNpadded 128
"-b 112" -> PADDED 112
"-b 111" -> UNpadded 112
etc.

The problem was that the -b switch (before 0.08) assumed unpadded frames, so using "-b 128" REALLY meant "-b 127.706", which opened the door for a 160kbps frame to sneak through (remember that lowering the minimum bitrate tends to raise the maximum bitrate). Now using "-b 128" assumes padded frames, so turns into "-b 128.013". This is enough to guarantee a CBR128 file stays a CBR128 file, but at the expense of a little more wasted space (On the order of 1 bit per frame)

padding is only needed for CBR at 44.1, 22.05 or 11.025 kHz. it should be done such that the actual bitrate is always less or equal to the desired bitrate. you may look into the LAME sources or read about it in MPEG-Layer3 Bitstream Syntax and Decoding. You can find that document at Gabriel's "mp3-tech" site: http://www.mp3-tech.org/programmer/docs/96-05.pdf

A new version, 0.3 beta, of WinMP3Packer has been released which caters for version 0.08 of mp3packer.pl.

Check the thread here

Released 0.09.

Better support for whole directories. Specifically, the -f switch now works on directories, and the -X switch works with single files. Plus, added support for an output directory.

The other addition is the -i switch, which should be of interest to psyllium. It outputs a whole bunch of info, then exits. An example of the output:

The filst few lines should be self-explanatory. All the bitrates are calculated slightly differently:
1st bitrate: bytes in file / second. Includes the tags and non-frame data
2nd: This is the "normal" bitrate, consisting of all the frame data
3rd: The amount of MP3 payload data, without the header, side info, or padding
4th: Payload data + header + side info. This is the theoretical minimum bitrate that mp3packer can pack to.
5th and 6th: How much is used up by padding and non-MP3 data, respectively

Then comes the bitrate distribution: The number of unpadded, padded frames at each bitrate (The example file didn't use padded frames)

Then comes the really interesting part: It calculates the smallest possible CBR bitrate. Like the GUI's approach, it is calculated by iterating through the bitrates and seeing if they will produce a CBR file. However, everything happens in memory and is optimized for just printing out info, so it goes MUCH faster.

Sweet as dude I'll add support for that when my brain returns from holidays... in a few days I reckon...

Getting this error:


I've tried via both the GUI & commanline and same. Ran a basic command of: 'mp3packer.pl -a -vbr'

Interestingly, the GUI shows this file as being 128kbps, whereas MRQ shows 320kbps FhG.

Any ideas?

That sounds like what Firon's problem on post 55 was. The problem was Unicode tags before the real MP3 data that look like the beginning of an MP1 or MP2 frame.

Make a copy of the file, and strip the tags off the copy (using Foobar or whatever), then try running mp3packer on the copy. If it works, then the problem is a false sync header in the tag. If it still doesn't work, then it's something else

I am constantly getting this error message:

Wide character in print at C:\mp3packer/mp3.pm line 866.

The output file does not show correct length in foobar.

Hmmm... that's where the program writes the LAME header. I have a lot of chr(....) statements which could result in a Unicode character slipping by once in a while, although obviously it shouldn't.

What kind of files make this error? Are they massively long? What command line are you using (or are you using psylium's GUI)? Does the file play OK? What if you use Foobar's "fix mp3 header"? (I think that's enough questions for now )

Omion, I love your program, but I've been having some problems with it.

One is a minor annoyance: assuming a is the original CBR 320kbps mp3 without any ID3 (or other) tags, and b is a VBR by mp3packer 0.09, b - a != 0, which, I assume, it should. I did these operations in Audacity. The differences are minor, and generally inaudible, but it's still not a straight line

The other one is a bigger problem: I tested a CBR 96kbps tag-free mp3 with mp3packer, and got a VBR that had the following properties:

• The copy was 3456 samples "behind" the original file, meaning that every single sample was shifted 3 frames.
• The copy omits last 2304 samples of the original file altogether, even if not using -s. (This also happened with the b-a test above).
• Approximately the first two frames of the copy contain modifications to the actual waveform, distorting the audio data in those first ~50 ms.

I do not know what might cause these problems, as I have removed tags from my input files... So that's why I'm turning to you for help.

Thanks,

UED77

Does Audacity dither the audio? That would be the most obvious explanation. My program doesn't decode any of the data, so if it messed it up it would probably be a LOT more noticabe than "generally inaudible". It's just about impossible for mp3packer to change the data a little bit. It'll either be perfect, offset by a multiple of 1152, or complete garbage. Try Foobar's "bit-compare files" feature; I know it works (and doesn't dither).


That's a bit bizarre. Even if the program is misinterpreting the XING tag, it should still only be off by 1152 samples. It sounds like something in the file is confusing my proggie. What version of Perl do you have?

It seems that mp3packer is chopping the file. See the following output for running mp3packer for multiple times.

C:\>mp3packer 5.mp3 5a.mp3
Pre-parsing frame 3836
On frame 3836 of 3836
Output file uses frame sizes from 32 to 320
3836 frames in 1520131 bytes

C:\>mp3packer 5a.mp3 5b.mp3
Pre-parsing frame 3825
On frame 3825 of 3825
Output file uses frame sizes from 32 to 320
3825 frames in 1519171 bytes

C:\>mp3packer 5b.mp3 5c.mp3
Pre-parsing frame 3815
On frame 3815 of 3815
Output file uses frame sizes from 32 to 320
3815 frames in 1518211 bytes

C:\>mp3packer 5c.mp3 5d.mp3
Pre-parsing frame 3805
On frame 3805 of 3805
Output file uses frame sizes from 32 to 320
3805 frames in 1517251 bytes

O_O
You're right. Looks like that started on version 0.08. What an odd thing for the program to be doing... testing...

[edit]
WOW. I must have been completely insane when I was working on 0.08.

FIX:
Un-comment line 804 of mp3.pm. Delete the initial "#" character at the beginning of the line:


I broke the download link in the first post so nobody else downloads it. I'll release 0.10 as soon as I hammer out the other problems brought to light today. Unfortunately, I am quite tired right now, so I'll work on them in the morning.

Sorry for not getting back earlier.
Thought I'd let you know the outcome since you'll be looking at v0.10 soon.

If I update any of the tags (actually deleted a couple) in foobar, then it works ok. So, seems it's the same problem as you mentioned. However, when I then run mp3packer I get this:

Command run: mp3packer.pl -M 40 -a -vbr

Output from mp3packer:

Foobar properties of 1.mp3:

Foobar properties of 1-vbr.mp3:



All fine so far, but...
1-vbr.mp3 when put into mp3packer shows the file is 56kbps CBR!
1-vbr.mp3 is displayed as 56kbps CBR by AudioShell 1.1 and the length as 27:43
1-vbr.mp3 is displayed as 56kbps CBR by Mr QuestionMan v0.701 and the length as 27:44
1-vbr.mp3 is displayed as 56kbps CBR by Mp3tag v2.35 and length as 27:52

Now, I'm not bothered about the differences in the lengths that different progs display, but obviously the 27mins'ish time is just not correct. Also, the 56kbps is surely not correct.

Any ideas?

PM me if you want some more details about the file.

This fix has a side effect: the line 866 error is gone too!

Okay, I commented out line 804, like you suggested. The cause of my first problem was probably Audacity's dithering, as foobar2000's bitcompare says the two files's outputs are identical -- albeit of different lengths. Commenting out the line fixes this problem.

As for the second problem issue, commenting out the line seems to fix this too.

Thanks for your help, and keep up the great work!

UED77

I think I might know the problem. mp3repacker uses a small frame for the XING header (unless the -b option is given) That space is reserved before the file is written. However, the file that you have was repacked to a CBR file, so the small XING frame says it's CBR. I guess programs see that and think it's actually 56kbps CBR. The problem that I see is that the frame is usually a 64kbps frame, so I don't really see why anything would mistake that as 56kbps.

The timing being off is another effect of the bitrate being completely wrong. 56/320 = 4:51/27:44. Fixing the bitrate will fix the timing.

I think I can make sure that the XING tag only says it's CBR if the initial frame is the same bitrate as the rest of them.


PS. What was the initial encoder? It looks like the encoder didn't use the bit reservoir, and just about completely filled up every frame. It's very rare for my program to output a CBR file without the use of the -b switch.

see post post 83 - FhG

Thanks for looking at this btw

0.10 out. Just a bugfix. All the recent gripes should be quelched

Many thanks Omion.

I reckon that for most peeps using your prog to reduce the size of files and thus make vbr files, it will be done on non-LAME mp3s. Cos most peeps that use LAME will probably use vbr or cbr 192kbps. Thus, the mp3 files might not be created in the nicest way... just like the file that I'm having problems with, and as you said yourself (via PM). Just goes to show how good LAME really is.

A feature request:
Rename the original file and use the input file name as output.

Ah yes.
As in: shove '-orig' on the end of the original input file, and thus the resulting output file can have the original's original name (follow me!?).

Hmm, when using this I get the error message "Can't locate object method 'close' via package 'mp3' (perhaps you forgot to load 'mp3'?) at mp3packer.pl line 171." It will still work when just using the command prompt, but the frontend will cancel the process (or something, it stops and deletes the output file). I was hoping version 0.10 would fix this (I've only had trouble with version 0.9 and now 0.10). Any idea what's wrong?