Hydrogenaudio Forums > MP3 repacker

Omion

May 24 2007, 22:19

QUOTE(Jojo @ May 24 2007, 15:10)

I assume neither of the changes affect WinMP3Packer, so that I just have to replace the .exe?

Yup. There weren't any changes in the latest version that would affect winmp3packer. Well... it may not report the errors correctly, if there were any. I changed around the error reporting in 1.16, and I don't know how winmp3packer handles it.

QUOTE(gottkaiser @ May 24 2007, 17:54)

Could you add a option to keep the time stamps of the old mp3 files?

It looks like I can easily add the option to change the last accessed and last modified times for the files, but to change the created time would require an external app.

What did you have in mind for using the original times? Would just changing the last modified time work?

gottkaiser

May 25 2007, 03:13

QUOTE(Omion @ May 25 2007, 06:19)

It looks like I can easily add the option to change the last accessed and last modified times for the files, but to change the created time would require an external app.

What did you have in mind for using the original times? Would just changing the last modified time work?

If it is not possible to keep all the time stamps inclusive created time. Then it's all right.

Thanks for trying! :-)

Polar

May 25 2007, 04:42

QUOTE(gottkaiser)

@Omion

Thanks for developing this great tool!

I can only second that. Wiki article anyone?

j7n

May 25 2007, 11:15

QUOTE

The -z switch now does something with MPEG2 / MPEG2.5 files (should fix j7n's issue)

Thank you for the update, those -r -R switches look promising. It wasn't me who tried repacking MPEG 2, but another poster above (Martin F.).

gottkaiser

May 25 2007, 12:35

@Omion

I created a Wiki article. Maybe you can check it that it's ok like that.

decayed.cell

May 26 2007, 02:03

Would it be possible to add Unicode filename support?

Omion

May 26 2007, 19:07

QUOTE(decayed.cell @ May 26 2007, 01:03)

Would it be possible to add Unicode filename support?

Unfortunately, that's not easy to do. The problem is that the programming language I use (OCaml) has no idea what Unicode is, so it refuses to open Unicode files. I've been thinking of writing a patch around that, but it probably won't be fixed for a while.

That's one of the few things I really hate about OCaml (the other is lack of multi-processor support), and it looks like it won't be officially supported for a while...

j7n

May 27 2007, 18:06

@decayed.cell
Using special characters in filenames is a very bad idea. You risk losing data. Ocaml is but a single example of non-Unicode aware software.

(1) I've encountered several files where Mp3Packer messes up non-mp3 data at the end (APETAGEX/ID3). How non-standard non-fb9 the APETAGEX might be, this should not happen. Here is one sample processed with v1.17, it has happened before with other versions. It would seem that mp3packer chokes on the repeated "ALBUM" for some reason. Copy the code blocks to notepad in order to view them properly.

Original tag:

CODE

00000000   41 50 45 54 41 47 45 58  D0 07 00 00 1C 01 00 00  08 00 00 00 00 00 00 A0  00 00 00 00 00 00 00 00   APETAGEXŠ
00000020   0C 00 00 00 00 00 00 00  54 69 74 6C 65 00 41 64  69 6F 73 20 4E 6F 6E 69  6E 6F 1D 00 00 00 00 00           Title Adios Nonino
00000040   00 00 41 72 74 69 73 74  00 53 65 78 74 65 74 6F  20 4D 61 6A 6F 72 20 54  61 6E 67 6F 20 4F 72 63     Artist Sexteto Major Tango Orc
00000060   68 65 73 74 72 61 13 00  00 00 00 00 00 00 41 6C  62 75 6D 00 41 20 50 61  73 73 69 6F 6E 20 66 6F   hestra        Album A Passion fo
00000080   72 20 54 61 6E 67 6F 1F  00 00 00 00 00 00 00 41  4C 42 55 4D 32 00 41 75  74 68 65 6E 74 69 63 20   r Tango        ALBUM2 Authentic
000000A0   54 61 6E 67 6F 73 20 66  72 6F 6D 20 41 72 67 65  6E 74 69 6E 61 0A 00 00  00 00 00 00 00 41 4C 42   Tangos from Argentina        ALB
000000C0   55 4D 44 41 54 45 00 31  39 39 34 2D 30 37 2D 31  39 0B 00 00 00 00 00 00  00 45 4E 47 49 4E 45 45   UMDATE 1994-07-19        ENGINEE
000000E0   52 00 4C 61 75 72 61 20  46 6F 6E 7A 6F 0E 00 00  00 00 00 00 00 50 52 4F  44 55 43 45 52 00 45 74   R Laura Fonzo        PRODUCER Et
00000100   74 6F 72 65 20 53 74 72  61 74 74 61 02 00 00 00  00 00 00 00 54 72 61 63  6B 00 31 38 41 50 45 54   tore Stratta        Track 18APET
00000120   41 47 45 58 D0 07 00 00  1C 01 00 00 08 00 00 00  00 00 00 80 00 00 00 00  00 00 00 00 54 41 47 41   AGEXŠ              €        TAGA
00000140   64 69 6F 73 20 4E 6F 6E  69 6E 6F 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 53 65 78   dios Nonino                  Sex
00000160   74 65 74 6F 20 4D 61 6A  6F 72 20 54 61 6E 67 6F  20 4F 72 63 68 65 73 74  72 61 00 41 20 50 61 73   teto Major Tango Orchestra A Pas
00000180   73 69 6F 6E 20 66 6F 72  20 54 61 6E 67 6F 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   sion for Tango
000001A0   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 12 FF                                          ˙

Bad tag:

CODE

00000000   41 50 45 54 41 47 45 58  D0 07 00 00 1C 01 00 00  08 00 00 00 00 00 00 A0  00 00 00 00 00 00 00 00   APETAGEXŠ
00000020   0C 00 00 00 00 00 00 00  54 69 74 6C 65 00 41 64  69 6F 73 20 4E 6F 6E 69  6E 6F 1D 00 00 00 00 00           Title Adios Nonino
00000040   00 00 41 72 74 69 73 74  00 53 65 78 74 65 74 6F  20 4D 61 6A 6F 72 20 54  61 6E 67 6F 20 4F 72 63     Artist Sexteto Major Tango Orc
00000060   68 65 73 74 72 61 13 00  00 00 00 00 00 00 41 6C  62 75 6D 00 41 20 50 61  73 73 69 6F 6E 20 66 6F   hestra        Album A Passion fo
00000080   72 20 54 61 6E 67 6F 1F  00 00 00 00 00 00 00 41  4C 42 55 4D 32 00 41 75  74 68 65 6E 74 69 63 20   r Tango        ALBUM2 Authentic
000000A0   54 61 6E 67 6F 73 20 66  72 6F 6D 20 41 72 67 65  61 20 46 6F 6E 7A 6F 0E  00 00 00 00 00 00 00 50   Tangos from Argea Fonzo        P
000000C0   52 4F 44 55 43 45 52 00  45 74 74 6F 72 65 20 53  74 72 61 74 74 61 02 00  00 00 00 00 00 00 54 72   RODUCER Ettore Stratta        Tr
000000E0   61 63 6B 00 31 38 41 50  45 54 41 47 45 58 D0 07  00 00 1C 01 00 00 08 00  00 00 00 00 00 80 00 00   ack 18APETAGEXŠ              €
00000100   00 00 00 00 00 00 54 41  47 41 64 69 6F 73 20 4E  6F 6E 69 6E 6F 00 00 00  00 00 00 00 00 00 00 00         TAGAdios Nonino
00000120   00 00 00 00 00 00 00 53  65 78 74 65 74 6F 20 4D  61 6A 6F 72 20 54 61 6E  67 6F 20 4F 72 63 68 65          Sexteto Major Tango Orche
00000140   73 74 72 61 00 41 20 50  61 73 73 69 6F 6E 20 66  6F 72 20 54 61 6E 67 6F  00 00 00 00 00 00 00 00   stra A Passion for Tango
00000160   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
00000180   00 00 00 00 12 FF                                                                                         ˙

(2) How about setting the process priority to low by default? The processing with Z takes a long time and the comp becomes pretty much unresponsive during this process.

Omion

May 27 2007, 18:32

QUOTE(j7n @ May 27 2007, 17:06)

I've encountered several files where Mp3Packer messes up non-mp3 data at the end (APETAGEX/ID3). How non-standard non-fb9 the APETAGEX might be, this should not happen. Here is one sample processed with v1.17, it has happened before with other versions. It would seem that mp3packer chokes on the repeated "ALBUM" for some reason. Code blocks require 1280*960 to view properly using the default style.

I don't think it's a problem in the tag itself. mp3packer doesn't care or know about tags at all. It just sees a bunch of stuff which isn't MP3 data and copies it over to the new file. It's more likely a problem in the last mp3 frame which is causing mp3packer to screw up the tag.

There is a problem with MP3 files which have the last frame cut off. If a tag is added to the end of one of these files, it will actually start in the middle of the last MP3 frame. mp3packer will then "optimize" the last frame and throw out the beginning of the tag (since it is non-audio data in an mp3 frame)

That doesn't quite look like what's happening here, but something's definitely screwing up.

QUOTE

How about setting the process priority to low by default? The processing with Z takes a long time and the comp becomes pretty much unresponsive during this process.

I'll look into it. OCaml doesn't have a way to do that in Windows, but I could hook it into a custom C function that would do it. It shouldn't be too hard (assuming I remember how to do that

)

j7n

May 28 2007, 10:39

I can upload the whole file, but it's copyrighted.

It's been encoded using LAME 3.91, with some frames at the beginning chopped off that produces buffer underflow. Last frame is ok.

Ojay

May 30 2007, 06:47

Hm, I have a problem with the newest version (1.17). For all versions prior to 1.17, when using the flag -b 192 (and no other flags) I had the final vbr file within 10 seconds. With version 1.17, it took 16 minutes for the same operation. MP3 file size was 80MB.

Is this a bug or a "feature", and why?

Edit: I checked the same operation but now with the flags: "-b 192 -R" - this resulted in the old, very fast process (10s instead of 16m) and the final file size was EXACTLY the same (for the very slow "-b 192" and the fast "-b 192 -R")

Omion

May 30 2007, 13:29

@Ojay:

The new way is supposed to be a feature, but obviously 16 minutes is far too long for any feature. And yes, the file size is always identical (-r and -R don't change the frame sizes at all, just the location of the data within the frame).

The -r switch (enabled by default if you use -b) adds another frame queue. Usually, one of two things happens with this:
1. If the -b switch is larger than the average input bitrate, the extra queue will be very small (1-4 frames)
2. If the -b switch is smaller than the average input bitrate, the extra queue will average ~50 frames

For a nasty enough input file, it is technically possible for the extra queue to have to store every frame in the file, which is quite a bit for an 80MB file. And each new frame has to check every frame in that queue, so the time it takes is proportional to the square of the number of frames! I didn't think that would happen so I didn't code for it, but I suppose Murphy's Law has a firm grasp on mp3 assumptions

I'd like to take a look at the file, but obviously it would be difficult to email an 80MB file. So instead, run this program on the file, like this:

CODE

mp3reader problemfile.mp3 > output.txt

Then zip and email the output.txt file to me.

Thanks for the report!

nemoW

May 31 2007, 13:46

Could MP3 repacker process MPEG-1 layer 2 files?

Omion

May 31 2007, 22:00

QUOTE(nemoW @ May 31 2007, 12:46)

Could MP3 repacker process MPEG-1 layer 2 files?

Nope. It just does layer 3. There is very little demand for it, and even less documentation, so I didn't bother. Unfortunately, it would be fairly difficult to wedge in support for layer 2 files right now.

Also, I'm going over the LAME layer 2 decoder, and it looks like there's no side info like there is for layer 3. If that's the case, it would imply that I'd have to actually decode every frame to see how long the data is, which would be way too much work given the difference in popularity between layer 2 and layer 3.

radorn

Jun 1 2007, 01:02

hi.

I see the tool has been up for quite a while, but I just discovered it now, so, for starters, congratulations for making such a nice tool.

Now, I would like to ask something that maybe is a bit stupid but I ought to.
I'm starting to check some old mp3's I have to rehaul them a bit (retaggin with more accurate data, maybe a rename, adding replaygain, and, of course, repacking it).
Normally I would take extra time to check that the decoded sound is identical to the original after repacking, because my music is important to me (obviously

), but it's pretty time consuming and heavy.
Since I haven't experienced the program to fuck up my music yet, I've put a little cheap batch file in my PATH for mp3packer with my preferred comandline options (wich include -z, and deleting the original upon successful repack), and that's what I tend to use.
You seem to put out new versions quite fast, and that's good, but it makes me feel uneasy as to if the version I'm using (1.17-171 right now, wich is the last, if no new one has been posted while I write this) can be considered stable and working right, because, of course, my much loved music would be in danger.

So, could you, as the coder of this great tool, say whether it's advisable (withing reasonable limits) to just use the last version, or is there some earlier version wich is considered safer?

Polar

Jun 1 2007, 03:24

What's the compression ratio gain one can expect from running MP3packer 1.17 over 320 kbps CBR LAME 3.97 encodes, realistically speaking?

Edit: realistically speaking implies I know about the Can make --preset insane files up to 10% smaller front page claim

j7n

Jun 1 2007, 04:28

Radorn, due to the reasons I wrote about a few posts above, consider tagging your files after they've been processed by Mp3Packer, since in some cases your tags might get corrupted. However, I've never encountered the program outputting corrupted MPEG data.

In my opinion, there is no need to test the safety of Mp3Packer on all your collection. Mp3Packer is by no means a magic wand. I'd use this tool only if

- In a hex editor I see many null, 0xFF or otherwise repeating patterns of data.
- I have cut my files disregarding the bit reservoir. (Some hardware devices don't like these.)

radorn

Jun 1 2007, 07:56

Polar and j7n

I appreciate you reply to my post, but you haven't really answered y question, but questioned my need for such an answer.

My interest from this tool, and the use I make of it, is beyond the scope, I believe.
I'm aware (though I haven't experienced) of the tag corrupting thing, wich is logical if you think about it. I really didn't imply an ordered procedure when I enumerated the things I do to my files. It was just a list, so repacking is not necesarily the last step.
Also, I know that big space savings are rare, and normally you just get a few kb, maybe less. That's not a problem to me.

What I mean is you should not question my reasons (and assume they are due to a lack of knowledge) and just give the best answer possible to the question. Or do both thinks if you must, but at least do the real answering.
That gives people (me in this case) the oportunity to decide what to do with the information, instead of just mindlessly accepting what others think is best.

I don't pretend to sound like lecturing anyone, it's just that I use to ask uncommon things (common ones are already answered, aren't they) and getting to get this kind of replies instead of a real answer, which don't solve me anything.

So, in the end.
Is there a policy or recommendation about what version to use regarding it's reliability? reliability meaning it won't fuck your files up.

Polar

Jun 1 2007, 08:01

QUOTE(radorn)

Polar and j7n

I appreciate you reply to my post, but you haven't really answered y question, but questioned my need for such an answer.

Well, to be entirely honest, my post wasn't quite meant as a reply to yours, but fwiw, you're welcome

radorn

Jun 1 2007, 08:22

QUOTE(Polar @ Jun 1 2007, 15:01)

QUOTE(radorn)

Polar and j7n

I appreciate you reply to my post, but you haven't really answered y question, but questioned my need for such an answer.

Well, to be entirely honest, my post wasn't quite meant as a reply to yours, but fwiw, you're welcome

Oh, haha, sorry. I assumed it was for me since it seemed to be so given the content, timing and lack of clear adressing. my apologies.

Architectonical

Jun 1 2007, 09:21

Omion - one 'feature' that I don't like is the automatic 'fixing' of sync errors. So there is still an audible skip, but the errors are no longer reported as the frame has been repaired.

eg on this track (0:13) : http://www.archive.org/download/xgn014_-_s...nown_galaxy.mp3

Also, I'm not sure if this has been mentioned yet, but is it possible to have a feature which allows the automatic removal of digital silence from the start and end of each track?

QUOTE(Polar @ Jun 1 2007, 01:24)

What's the compression ratio gain one can expect from running MP3packer 1.17 over 320 kbps CBR LAME 3.97 encodes, realistically speaking?

Edit: realistically speaking implies I know about the Can make --preset insane files up to 10% smaller front page claim

It depends on the bitrate, encoder, track etc. But I typically get around 1-5% size reduction. Non-lame encodes can often be reduced a little more than lame encodes. Occasionally I notice some reduction even with 128-192kbit/s tracks, but it depends on the track and the encoder used. Edit - this is with using the -z switch.

Jojo

Jun 1 2007, 14:39

I noticed that files that contain ID3v2 and APEv2 tags are identified as 160kbps, even though they are 320kbps CBR. Also, I tried to shrink some Lame 3.96 encoded files using 320kbps and regular Stereo (no Joint-Stereo) and although I used the -z switch, it only saved 3kbps. I've tried several files.

gib

Jun 1 2007, 17:50

QUOTE(radorn @ Jun 1 2007, 03:56)

Is there a policy or recommendation about what version to use regarding it's reliability? reliability meaning it won't fuck your files up.

Simple. After packing the mp3s, use foobar to do a bit compare between the packed and the original. If they match, you're good.

radorn

Jun 1 2007, 18:32

QUOTE(gib @ Jun 2 2007, 00:50)

QUOTE(radorn @ Jun 1 2007, 03:56)

Is there a policy or recommendation about what version to use regarding it's reliability? reliability meaning it won't fuck your files up.

Simple. After packing the mp3s, use foobar to do a bit compare between the packed and the original. If they match, you're good.

If you had read my posts you'd know I already knew and did that, but it gets a bit tiresome after a while you know, so that's one reason I'm asking this.

"Normally I would take extra time to check that the decoded sound is identical to the original after repacking, because my music is important to me (obviously ), but it's pretty time consuming and heavy."
That I said, which you didn't read. Also, I made a batch file for repacking and deleting backups if mp3packer thinks everything went OK. It saves time and clicks you know... but only if you have the certainty that your files aren't getting fubar in the process. So, I need to know, to decide whether or not to use that "script", among other things.

Please, someone, answer to my question, instead of giving me alternatives.
I know my way arround these things and can make decissions for myself, but for that I need information, as concise as posible, and that's what I'm asking for, nothing more, nothing less.
Would someone be kind enough to do that? I beg. (and, no, I'm not being sarcastic, even though it could seem so)

Omion

Jun 1 2007, 22:23

QUOTE(radorn @ Jun 1 2007, 00:02)

hi.

I see the tool has been up for quite a while, but I just discovered it now, so, for starters, congratulations for making such a nice tool.

Now, I would like to ask something that maybe is a bit stupid but I ought to.
I'm starting to check some old mp3's I have to rehaul them a bit (retaggin with more accurate data, maybe a rename, adding replaygain, and, of course, repacking it).
Normally I would take extra time to check that the decoded sound is identical to the original after repacking, because my music is important to me (obviously

), but it's pretty time consuming and heavy.
Since I haven't experienced the program to fuck up my music yet, I've put a little cheap batch file in my PATH for mp3packer with my preferred comandline options (wich include -z, and deleting the original upon successful repack), and that's what I tend to use.
You seem to put out new versions quite fast, and that's good, but it makes me feel uneasy as to if the version I'm using (1.17-171 right now, wich is the last, if no new one has been posted while I write this) can be considered stable and working right, because, of course, my much loved music would be in danger.

So, could you, as the coder of this great tool, say whether it's advisable (withing reasonable limits) to just use the last version, or is there some earlier version wich is considered safer?

Starting with the last question:
Currently, 1.16 or 1.17 are the best bet. They are functionally the same when not using the -z switch. If you are using the -z switch, 1.17 will give slightly higher compression at a fairly significant time increase (up to 40% longer). If you are worried about new features being less tested, you should stick with 1.16. Otherwise use 1.17. All previous builds are available here.

That being said, I don't remember any well-formed mp3 files that did not test exact (without using -z) since 0.10 over a year ago. In the interest of fairness, here are the current issues that can happen:

If the input file was part of an improperly-split file, the first few frames may be gone
Obviously, random data errors can cause confusion, perhaps making a non-bit-identical file depending on how broken frames are handled by the other program.
If the XING/LAME frame is somehow invalid, mp3packer will throw it out whereas Foobar will still use the invalid information. This may make Foobar think the new file is a tiny bit longer and therefore not bit-identical.
j7n just reported an issue where an invalid last frame would cause the tags to get screwed up. This doesn't affect the audio at all, though.
mp3packer will only think that a frame is valid if it has the same samplerate, CRC existance, channel mode, copyright, and emphasis. Input files which are merged from two other files which differ on these conditions may only have the first part copied. (the one exception is if the first frame is LAME/XING. Then that frame only needs to have the same samplerate as the rest)
When using the -z switch, mp3packer handles a certain type of overflow one way whereas another program may handle it a different way. The details are a bit obscure, but it has to be handled one way or another.

Note, again, that all of these issues are due to incorrectly-formed input files.

QUOTE(Polar @ Jun 1 2007, 02:24)

What's the compression ratio gain one can expect from running MP3packer 1.17 over 320 kbps CBR LAME 3.97 encodes, realistically speaking?

Edit: realistically speaking implies I know about the Can make --preset insane files up to 10% smaller front page claim

Yeah... --preset insane won't compress that much as of 3.97, unless you're using -z and get extremely lucky

I don't know any exact numbers, but I seem to remember it's generally in the 1-3% range. -z is maybe ~5%. I would be happy to be corrected on this point, though, if anybody else has done any other tests.

@Jojo:
What program are you using to identify the file? Was this the input or the output file that you checked?

nemoW

Jun 2 2007, 07:26

In my tests any files coded with Lame show slight compression even with -z (~4% maximum for some 320CBR). But some high-bitrate CBR files encoded with other tools (iTunes etc.) can be compressed much better (up to 13%).

Jojo

Jun 2 2007, 16:49

QUOTE(Omion @ Jun 1 2007, 21:23)

@Jojo:
What program are you using to identify the file? Was this the input or the output file that you checked?

to identify what tags a file contains I use mp3Tag, which I also used for tagging the file.
in terms of identifying what codec has been used I use Audio Identifier and double checked with foobar.

gib

Jun 2 2007, 21:11

QUOTE(radorn @ Jun 1 2007, 14:32)

If you had read my posts you'd know I already knew and did that, but it gets a bit tiresome after a while you know, so that's one reason I'm asking this.

"Normally I would take extra time to check that the decoded sound is identical to the original after repacking, because my music is important to me (obviously ), but it's pretty time consuming and heavy."
That I said, which you didn't read.

Wow, thanks for the snarky response. Not only is it unnecessary, but it also makes you sound like a complete jerk since you aren't even in the right. I read your posts, and saying, "...check that the decoded sound is identical to the original..." does not imply you used foobar2000 to bit-compare the files - it implies you listened to the files, because that's what you do with sound.

In the future it's a good policy not to be pissy to someone who took a moment of time to post a possibly useful reply. This is especially true when the initial failure is on your end due to not being clear in your posts.

radorn

Jun 3 2007, 16:56

Thanks Omion, that mostly solves my doubts.

Also, I apologize for littering your program's thread.
I already moved the discussion with that "gib" guy to the private domain.

QUOTE(gib @ Jun 3 2007, 04:11)

[...] saying "...check that the decoded sound is identical to the original..." does not imply you used foobar2000 to bit-compare the files - it implies you listened to the files, because that's what you do with sound.

In the future it's a good policy not to be pissy to someone who took a moment of time to post a possibly useful reply. This is especially true when the initial failure is on your end due to not being clear in your posts.

Whatever, man.

I'll reply to your nonsense privatelly, because we have littered this thread enough so far.

Omion

Jun 5 2007, 18:35

1.18 is out.

The main difference is the addition of the --nice switch. Use it to change mp3packer's priority.

The --nice switch has a similar range as the Unix "nice" utility. 0 is normal, 19 is idle, and negative numbers are higher priority. The exact mapping from Unix-range numbers to Windows priorities is listed in the mp3packer HTML file.

mp3packer will default to "--nice 10" which will show up as "below normal" in the task manager.

I also fixed some bugs which may or may not have existed.

Polar

Jun 6 2007, 02:49

It's refreshing to see this tool being developed so actively, with updates and bug fixes issued at such short intervals.

That said, what about running MP3packer over freeformat MP3s encoded at, say, 325 kbps? I was wondering: if you're lucky enough to squeeze out enough bits to get them down to 320 kbps (that's just 1.5%, so using the -z switch you should, right?), could this be a nice 'n' clean way to get the highest possible encoding quality for 16 bit stereo MP3s, without breaking compatibility with just about any MP3 software or hardware on the planet?

I have little to no experience with freeformat MP3 files, so it may very well be that I'm daydreaming here. So if I am, sorry in advance.

Evidently, anyone being that pedantic about audio quality willing to go through the fuss of encoding to freeformat 325 kbps in stead of plain standard CBR 320 kbps, (edit:) without any ABXable improvements, should be referred to any lossless codec of choice. So my question is purely from a point of view of interest in the format, and in MP3packer. After all, MP3 having the pile driver ascendancy over any other format it has, for the marginal audience looking for maximum compatibility along with maximum encoding quality, IMHO, this exercise might prove to be worth a small second of attention.

QUOTE(sven_Bent @ Mar 18 2005)

couldn't this funktion be put into lame ? some thing like -repack

Any news on this? Worth contacting the LAME developers, if not done so already?

Omion

Jun 6 2007, 12:57

QUOTE(Polar @ Jun 6 2007, 01:49)

It's refreshing to see this tool being developed so actively, with updates and bug fixes issued at such short intervals.

That said, what about running MP3packer over freeformat MP3s encoded at, say, 325 kbps? I was wondering: if you're lucky enough to squeeze out enough bits to get them down to 320 kbps (that's just 1.5%, so using the -z switch you should, right?), could this be a nice 'n' clean way to get the highest possible encoding quality for 16 bit stereo MP3s, without breaking compatibility with just about any MP3 software or hardware on the planet?

I have little to no experience with freeformat MP3 files, so it may very well be that I'm daydreaming here. So if I am, sorry in advance.

Evidently, anyone being that pedantic about audio quality willing to go through the fuss of encoding to freeformat 325 kbps in stead of plain standard CBR 320 kbps, (edit:) without any ABXable improvements, should be referred to any lossless codec of choice. So my question is purely from a point of view of interest in the format, and in MP3packer. After all, MP3 having the pile driver ascendancy over any other format it has, for the marginal audience looking for maximum compatibility along with maximum encoding quality, IMHO, this exercise might prove to be worth a small second of attention.

It would be possible to make it accept freeformat MP3s, and I looked into it a while ago. The problem is that I don't think anybody uses freeformat at 325kbps (do they?

), and the possibility of turning freeformat into something useful decreases rapidly above 320kbps.

I had wanted to support converting from allofmp3's 384kbps to 320kbps, since I figured that would be the most popular source of freeformat files. However, in the tests that I did, there were simply too many bits to cram into 320kbps.

I made a program a while ago which can tell the smallest frame size required for a freeformat stream, assuming that the bit reservoir can be used. Run this program on a freeformat mp3 and it will tell you the minimum frame size needed to store all the bits. For 44.1KHz files, if this number is larger than 1009 then the file must be freeformat.

QUOTE

QUOTE(sven_Bent @ Mar 18 2005)

couldn't this funktion be put into lame ? some thing like -repack

Any news on this? Worth contacting the LAME developers, if not done so already?

I don't think it's worth putting into LAME. LAME mp3 files don't get much benefit from repacking, and if you want to convert to CBR, why not call LAME with CBR options...

The -z switch is somewhat useful, but the code required for it is fairly big. Plus, it's all written in OCaml, which is somewhat difficult to integrate into C programs.

All in all, I don't see much benefit from combining mp3packer with any encoders.

Polar

Jun 7 2007, 02:29

QUOTE(Omion)

It would be possible to make it accept freeformat MP3s, and I looked into it a while ago. The problem is that I don't think anybody uses freeformat at 325kbps (do they?

) ...

Not yet

QUOTE(Omion)

... and the possibility of turning freeformat into something useful decreases rapidly above 320kbps.

I had wanted to support converting from allofmp3's 384kbps to 320kbps, since I figured that would be the most popular source of freeformat files. However, in the tests that I did, there were simply too many bits to cram into 320kbps.

So if I get it right, that means MP3packer currently doesn't support freeformat at all? I'll agree that it would be of very marginal use anyway.

QUOTE(Omion)

I don't think it's worth putting into LAME. LAME mp3 files don't get much benefit from repacking, and if you want to convert to CBR, why not call LAME with CBR options...

Imho, having LAME offer the opportunity to repack any high-bitrate CBR encode out-of-the-box, especially --preset insane, could have its use though.

Edit: fixed quotes.

nemoW

Jun 7 2007, 12:21

QUOTE(Omion @ Jun 6 2007, 20:57)

All in all, I don't see much benefit from combining mp3packer with any encoders.

It is known, that mp3 cannot be compressed with any archivers. Only SoundSlimmer can losslessly compress (not repack!) mp3 - partially decode/unpack mp3 and compress unpacked stream with more strong algorithm than mp3 use. But SoundSlimmer is shareware, cmd version absent, no winamp plug-in etc.

QUOTE

-z
This option makes mp3packer partially decode the audio in the file, optimize the data, then recode it.

Adjacent question: could mp3packer produce partially decoded output (neither mp3, nor wav) that might be compressed with general purpose archivers (and losslessly reverted to mp3)?

Excuse for my bad English.

Omion

Jun 8 2007, 00:49

QUOTE(Polar @ Jun 7 2007, 01:29)

So if I get it right, that means MP3packer currently doesn't support freeformat at all? I'll agree that it would be of very marginal use anyway.

Right. A freeformat mp3 will pass straight through mp3packer as though it were invalid.

QUOTE

Imho, having LAME offer the opportunity to repack any high-bitrate CBR encode out-of-the-box, especially --preset insane, could have its use though.

That would make sense, but I don't really think it'd help. I have the feeling that many people who use --preset insane are using it because it is "the highest quality that mp3 can go". If they ended up with a VBR 312kbps file, they'd complain that it was not maxing out the potential of the mp3 file format. Some people would understand it, but they could download the standalone program if they needed.

QUOTE(nemoW @ Jun 7 2007, 11:21)

Adjacent question: could mp3packer produce partially decoded output (neither mp3, nor wav) that might be compressed with general purpose archivers (and losslessly reverted to mp3)?

It's unlikely that a general-purpose archiver would be able to do better with raw, decoded mp3 data than mp3 itself. However, I'll keep it in mind and make a test when I'm less busy.

Firon

Jun 8 2007, 00:57

QUOTE(Omion @ Jun 8 2007, 02:49)

QUOTE(nemoW @ Jun 7 2007, 11:21)

Adjacent question: could mp3packer produce partially decoded output (neither mp3, nor wav) that might be compressed with general purpose archivers (and losslessly reverted to mp3)?

It's unlikely that a general-purpose archiver would be able to do better with raw, decoded mp3 data than mp3 itself. However, I'll keep it in mind and make a test when I'm less busy.

Using arithmetic coding instead of huffman would probably improve the compression ratio noticeably.

smack

Jun 8 2007, 02:35

QUOTE(Firon @ Jun 8 2007, 07:57)

QUOTE(Omion @ Jun 8 2007, 02:49)

QUOTE(nemoW @ Jun 7 2007, 11:21)

Adjacent question: could mp3packer produce partially decoded output (neither mp3, nor wav) that might be compressed with general purpose archivers (and losslessly reverted to mp3)?

It's unlikely that a general-purpose archiver would be able to do better with raw, decoded mp3 data than mp3 itself. However, I'll keep it in mind and make a test when I'm less busy.

Using arithmetic coding instead of huffman would probably improve the compression ratio noticeably.

@Firon
It's true that arithmetic coding is more efficient than Huffman coding so it wouldn't hurt to test it for mp3 repacking. However, the more significant improvements in compression ratio usually come from a better "understanding" of the data to be compressed in order to apply special algorithms for certain types of data. (context modeling / context mixing, see also: Wikipedia: PAQ)

@Omion
It should be possible to losslessly compress mp3 better using a more sophisticated algorithm. For comparison just look at the results of the Lossless JPEG compression test.

Rather than implementing de-packing in mp3packer and using another program (for instance a general-purpose archiver) to compress the de-packed mp3, IMO it would be more useful to implement a new context model for mp3 data in an existing program like PAQ.

j7n

Jun 8 2007, 03:29

And what would be the practical use of the efficiently packed MP3s? Even arithmetic coded JPEGs are not safe for archival because of little support in programs.

smack

Jun 8 2007, 04:01

QUOTE(j7n @ Jun 8 2007, 10:29)

And what would be the practical use of the efficiently packed MP3s? Even arithmetic coded JPEGs are not safe for archival because of little support in programs.

We're talking about an archiver that could be able to compress mp3's losslessly. Just think of it like zip-/rar-ing your collection of mp3 files in order to save some storage space, with the difference that the hypothetical archiver would compress mp3's much better than zip or rar. These archive files wouldn't be supported by existing mp3 players, of course.

btw. nemoW has provided a link to SoundSlimmer which appears to do exactly what we're discussing here. So it already exists, the question is: when will the first open-source program surface that works even better than that?

psyllium

Jun 11 2007, 01:04

First of all, I have to say I am very impressed with the performance of the '-z' option! This is particularly true for files that have been recorded on a device doing its own hardware encoding (in this case, an iKey Plus recorder).

The original file: 320kbps
Converted to vbr without -z: 306kbps
Converted to vbr with -z: 165kbps

I have checked the decoded WAV files and did a bit compare on them, and of course the output files are perfectly identical.

That's nearly HALF the size! This leads me to believe the encoder on this portable device isn't really up to scratch! I had a feeling this was the case anyway, and had been using WAV recording on the device for more important applications.

Anyway this means the 'super-squeeze' option will come in handy for compressing mp3s recorded on the device so they can be uploaded/downloaded faster.

psyllium

Jun 11 2007, 06:00

QUOTE(psyllium @ Jun 11 2007, 17:04)

The original file: 320kbps
Converted to vbr without -z: 306kbps
Converted to vbr with -z: 165kbps

For the hell of it, I tried squeezing the same file with version 1.16 of mp3packer. It gave me a file of pretty much the same size, I'm not sure whether this is symptomatic of the recording device or its encoding algorithm though. For the 4h30m VBR mp3 file, it took 37 minutes to process in version 1.18 instead of 25 minutes in 1.16. Do you think we could have an option for a 'normal' squeeze (1.16 behaviour), as well as a 'super' squeeze (1.18 behaviour)? No big deal if not, I can use 1.16 still

.

Chris.

Omion

Jun 11 2007, 23:58

QUOTE(psyllium @ Jun 11 2007, 05:00)

QUOTE(psyllium @ Jun 11 2007, 17:04)

The original file: 320kbps
Converted to vbr without -z: 306kbps
Converted to vbr with -z: 165kbps

For the hell of it, I tried squeezing the same file with version 1.16 of mp3packer. It gave me a file of pretty much the same size, I'm not sure whether this is symptomatic of the recording device or its encoding algorithm though. For the 4h30m VBR mp3 file, it took 37 minutes to process in version 1.18 instead of 25 minutes in 1.16. Do you think we could have an option for a 'normal' squeeze (1.16 behaviour), as well as a 'super' squeeze (1.18 behaviour)? No big deal if not, I can use 1.16 still

.

Chris.

Blarg. I knew someone would ask for that. For right now, the answer is no, because I changed around a lot of the data types when moving to the new -z switch, so everything would have to be re-written. I was thinking of doing sort of a -z1 -z2 -z3 thing, but that may take a while.

By the way, what were the actual final file sizes?

psyllium

Jun 13 2007, 06:01

QUOTE(Omion @ Jun 12 2007, 15:58)

Blarg. I knew someone would ask for that. For right now, the answer is no, because I changed around a lot of the data types when moving to the new -z switch, so everything would have to be re-written. I was thinking of doing sort of a -z1 -z2 -z3 thing, but that may take a while.

By the way, what were the actual final file sizes?

I've just recorded a smaller snippet this time around to test it out:
Original file: 1319706 bytes (320kbps CBR)
'mp3packer': 1242296 bytes (301kbps VBR) 94.1%
'mp3packer -z 1.16': 628042 bytes (152kbps VBR) 47.6%
'mp3packer -z 1.18': 627689 bytes (152kbps VBR) 47.6%

As you can see, this recording device is wasting a lot of space! It must be the packet compressing algorithm? But either way the 1.18 -z option doesn't seem to squeeze it that much more. I can provide a sample if you need it.

Ojay

Jun 30 2007, 09:59

Hmmm, I just tried MP3packer with the new LAME3.98b4. The new vbr-code seems to produce a lot of padding (in contrast to LAME3.97, I checked that). I was able to use mp3packer to reduce the bitrate from 231 to 227 (lame parameters: -V0 --vbr-new -b32).

4kbps padding removed with mp3packer :-)

PHOYO

Jul 2 2007, 14:33

QUOTE(Ojay @ Jun 30 2007, 18:59)

Hmmm, I just tried MP3packer with the new LAME3.98b4. The new vbr-code seems to produce a lot of padding (in contrast to LAME3.97, I checked that). I was able to use mp3packer to reduce the bitrate from 231 to 227 (lame parameters: -V0 --vbr-new -b32).

4kbps padding removed with mp3packer :-)

Yes, I noticed that too. That's pretty strange. I think LAME's vbr routine should be as efficient as possible. Is there any LAME developer here to explain this behaviour?

OT: --vbr-new isn't needed anymore in 3.98. -b32 is also unnecessary.

Jojo

Jul 4 2007, 12:39

it might have something to do with the strict ISO enforcement Lame 3.98 uses

numbskull

Jul 11 2007, 00:00

There is any advantage in converting cbr 192 to vbr (using -preset standard)?

Also, there is no loss depending on LAME version used in the original encoded file or the LAME used in the new file? I'm asking this because I still use LAME 3.90.3 --preset standard, and I'm yet scared to repack my 320kbps files, don't want any quality loss or other kind of problem.

Omion

Jul 11 2007, 00:10

QUOTE(numbskull @ Jul 10 2007, 23:00)

There is any advantage in converting cbr 192 to vbr (using -preset standard)?

If you transcode CBR-192 to --preset standard, there will be quality loss. However, that is not something that mp3packer does.

QUOTE

Also, there is no loss depending on LAME version used in the original encoded file or the LAME used in the new file? I'm asking this because I still use LAME 3.90.3 --preset standard, and I'm yet scared to repack my 320kbps files, don't want any quality loss or other kind of problem.

There will be no quality loss repacking to CBR-320. However, there will be no quality gain either. Unless you absolutely need CBR, don't repack to CBR-320.

I think you may be misunderstanding exactly what mp3packer does. The main thing to realize is that it does not change quality at all; it only changes file size. If you want higher quality MP3 files, re-rip from the source. If you want slightly smaller MP3 files, that's what mp3packer is good at.

MatMaul

Jul 12 2007, 05:42

hello !

thanks to you Omion for this great tool !

I propose 2 new functionalities :

--avoid-fhgdec-bug to generate mp3 file "compatibles" with the FHG decoder like lame 3.98b4

--strictly-enforce-ISO to do like the switch in lame too

I don't know if it is possible to do that losslessly, but if it is possible I think it can be useful.

Omion

Jul 13 2007, 21:27

Those are good ideas, but only the encoder can control either of them.

The FHG bug deals with the maximum Huffman value. mp3packer does not change these values (and it is generally impossible to do losslessly), it only changes how they are encoded.

The strictly-enforce-iso switch in LAME will lower the maximum amount of data in one frame. However, mp3packer will either not change the amount of data in a frame (without -z), or it will lower it (with -z). This means that a file repacked with -z may have the issue fixed, but it's impossible to always fix it.

The good news is that if the input file does not suffer from either of these issues, the output file won't either.