Help - Search - Members - Calendar
Full Version: Matroska Chapter Issues...
Hydrogenaudio Forums > Digital Audio/Video > General A/V
Canar
I'm attempting to use mkvmerge to link together several MP3 files (an album) in matroska to "solidify" all the seperate files in a way playable by foobar2000. The problem is that I can't seem to get it to work properly. I've messed around with the command-line options for quite a while and no matter what I do it won't work properly. Any suggestions that could help?

I've successfully converted some MP3/CUE pairs... I'm assuming the problem lies in the fact that I don't have defined chapters, but is there some option that will insert chapter markers between tracks automatically or something?

I'm also trying to do this with Ogg Vorbis as well. I've tried appending the files to each other and telling mkvmerge to convert that, to a similar result as the multiple MP3: You end up with a file that's as long as the longest, but only plays the first track.
Peter
I use this commandline:
mkvmerge -o <outfile.mka> --chapters <file.cue> <file.mp3>

I'll have to be cruel here, but current mkvmerge has serious issues with timecode accuracy - chapter durations you get won't be sample-accurate, and seeking isn't accurate either (convert albums that require gapless playback and you will get pops between tracks). I strongly recommend not to use current matroska tools and to wait till they get this fixed.
Canar
Okay, thanks for the recommendation. From my (rather limited) test, I had a perfect transition (according to my ears anyhow) between two adjacent tracks in several places in various different situations, so I assumed that there was sample-accuracy there. Apparently not though.

The command-line you wrote was exactly what I was using to convert my MP3/CUE pairs to Matroska files. But it doesn't seem to work to convert a group of individual, sequential tracks to a single file, which was what I was trying to do.
Peter
Chapter division glitches aren't audible all the time, I had to use Cool Edit to make sure whether they're present or not (seeking inaccurate by +- 50 samples).
I don't think you can merge multiple files into one MKA with current tools, but I may be wrong.
Pamel
You can import several files into Matroska as different tracks. I don't know if the fb2k plugin supports reading them properly, but it is easy. Using MMG, just keep adding files.

The current Matroska tools use the default accuracy of 1ms for timecodes. Because audio packets are basically never an exact multiple of 1ms, this leads to small rounding errors for the timecodes used to indicate the start time of of an audio packet. In practice this presents no issues with Audio/Video synch because it is impossible for anyone to notice 10ms off, let alone 1ms. However, this does mean that seeking to a specific sample becomes impossible. For 'normal' 48kHz audio, your accuracy is going to be +/- 0.5ms or +/- 24 samples.

It should be noted though that Matroska uses a base time accuracy of 1 nanosecond. But everything is scaled from the value stored in the TimecodeScale element allowing the overhead to be reduced. The default value of the TimecodeScale element is 1,000,000. And one million nanoseconds(ns) is one millisecond(ms). So, basically all timecodes are stored as ms.

There are several ways this can be used. For instance, with 96kHz audio you could store 10,417 in the TimecodeScale element and then all timecodes would be accurate to within 1ns, and offer accurate sample seeking. ((1/96000)x1000000000)=~10417. Unfortunately this would also increase overhead, but how much it would increase it remains to be seen from testing. Probably prohibitive for low bitrates, and minimal for high bitrates.

It should be noted that DirectShow only allows accuracy to 100ns.

@zZzZzZz:
Could you provide me a sample that shows the clipping caused by seeking to the wrong sample with the chapters?
Peter
I heard audible pops with Jean Michel-Jarre - Oxygene, but since the glitch depends on seeking operations made before before playing the track transition, it's total pain to reproduce (obviously visible in Cool Edit though).
Anyway, MKA with 1ms timecode accuracy doesn't allow sample-exact seeking, while its competitors (MP4, Ogg) do. MKA's current specs apparently don't allow gapless playback of MP3/AAC albums split into multiple files either.
ChristianHJW
QUOTE(zZzZzZz @ Jan 7 2004, 07:49 AM)
Anyway, MKA with 1ms timecode accuracy doesn't allow sample-exact seeking, while its competitors (MP4, Ogg) do. MKA's current specs apparently don't allow gapless playback of MP3/AAC albums split into multiple files either.

Our default scaling allows precisions of 1 ms ( millisecond ), as Pamel was noting above. The max. cluster lenght for this precision, however, can be 64 seconds, and we noticed that all current muxing tools will limit cluster length to a max. of 5 s anyhow, so we are currently wasting precision without any need.

We never had complaints from any side about missing timecode precision in matroska, in contrary many capturing fans are using matroska more and more because of its excellent sync compared to AVI ( especially for analog captrung ), but we have understood meanwhile that matroska has to be improved here for the specific needs of the audio fans.

Currently we investigate several possible solutions, as extensions to the existing specs :

- Using the time necessary for one sample ( 1 / sampling rate ) as timecode basis. This is well possible in the current specs, and will solve the issue completely if the sampling rate is accurate ( 44.1 KHz and 48 Khz from commercial CDs/DVDs ). We wont even need a spec change for that, its well possible with current specs and all spec compliant players should be able to handle that, only video editing, when several different tracks with different timecode bases are present, becomes a pain then.

- adding a new, optional element to increase timestamp accuracy within a block. Players would have to be updated to be able to read and use the element, old players would simply skip it, but still be able to play the track ( but without sample accurate seeking ). This method, unfortunately, would add some significant overhead, like 3 bytes / block.

- adding a complete new way to write timestamps, specifically for audio tracks and on a sample basis. This method would make audio tracks using it unplayable in old players/parsers, but would be overhead neutral compared to the 'normal' timestamps.

The discussion is currently ongoing on our mailing list, if you want to participate get a newsreader ( like mozilla ) and point it to news.gmane.org , group gmane.comp.multimedia.matroska.devel .
ChristianHJW
QUOTE(Canar @ Jan 6 2004, 11:33 PM)
I'm attempting to use mkvmerge to link together several MP3 files (an album) in matroska to "solidify" all the seperate files in a way playable by foobar2000. The problem is that I can't seem to get it to work properly. I've messed around with the command-line options for quite a while and no matter what I do it won't work properly. Any suggestions that could help?
I've successfully converted some MP3/CUE pairs... I'm assuming the problem lies in the fact that I don't have defined chapters, but is there some option that will insert chapter markers between tracks automatically or something?
I'm also trying to do this with Ogg Vorbis as well. I've tried appending the files to each other and telling mkvmerge to convert that, to a similar result as the multiple MP3: You end up with a file that's as long as the longest, but only plays the first track.

There are 2 different ways to 'collect' several songs into a single MKA file :

1. creating one big track, and chapters dividing each song. This is only possible if all songs are using the same codec, like MP3 in your case.

2. Putting each song into the file as a new track. In principal, you can then use any supported audio codec for each of the songs, say MP3, AAC, Vorbis, FLAC, AC3, DTS, PCM, MP2.

For example, if you have 12 songs ( 5 x 3 mins, 6 x 4 mins, 1 x 5 mins ), and you want to put them into a single MKA file you get

1. one track, with the total length being the sum of the lengths of the 12 songs, plus the gaps between them ( = 15 + 24 + 5 + gaps = 44 mins + gaps )

2. 12 tracks, each starting at timecode 0:00:00 s : the total length of the file is the same as the length of the longest of the 12 songs ( = 5 mins )

For a player, both files have to be treated differently as for

1. it will play the track like it was one, very long song. Chapters can be used to define where a new song is starting, and fb2k will display each chapter entry as a song of its own. But in mkvmerge, if you load the file, you will only see one audio track.

2. Means the playlist will show all available tracks in the file, and the player has to advise the matroska parser/input plugin to play the right track from the file. In mkvmerge, you will see 12 different audio tracks in your MKA file.


If you want to work with a CUE sheet from a CD, i highly recommend to use the 1. way and to concatenate all the MP3's you have into a single, long MP3 file. You can also do this using matroska ( dont know of a good MP3 concatenator tool ), by coverting each of the MP3's into MKA's with mkvmerge, and then concatenate all the MKA's with either avi-mux GUI or Pamel's 'matroska stream editor' into a single, long MKA file. Load this long MP3 or MKA in mkvmerge, convert the CUE sheet into a chapter file and load this chapter file for the muxing process in the 'Global' tab of mmg.exe .

As for using chapters in a file made like 2., this isnt possible. Chapters will only allow to jump to a specific timestamp in a single track, they do not allow to jump from one track to the next. This will be possible with 'control tracks' , which will allow to define exactly how to play a file, starting with track 1 at timestamp x, play until y, then proceed to track 4, timestamp z , etc .....
Canar
Thank you all for your detailed replies and I apologize for any tensions that this thread initiated.

@ChristianHJW: Your method 2 is kind of what I wanted to do. Kind of. I was hoping that there would be a method of concatenating each individual file (edit: into a single file), and automatically inserting the chapter point at the point of concatenation. The problem is that each file that's concatenated will have to be pared down to the proper length by the decoder (assuming it's not inherently gapless, like MP3) which adds a little more complexity.
fenterbug
Okay, I finally found mkvmerge and have started playing with it. Like Canar, I'd prefer method #2, but I'll settle for method #1 until player support arrives. However, I can't get my .mkv to play in Foobar.

I have separate .ogg files for each track in the album. (They each play perfectly in FB2K.) I append them into a single .ogg using copy /b 1.ogg + 2.ogg all.ogg. (The resulting file plays perfectly in FB2K, even distinguishing the individual tracks.) I then use the command posted by Peter above to make the mkv. When I load it into FB2K, I get, for each track, this:
ERROR (foo_matroska) : Matroska: no decodable streams found.

Is this an mkvmerge issue? A foo_matroska issue? A Foobar2000 issue?
wiflye81
Hi all,
I have just try the second method but it don't work, media players doesn't seem to support this function, if it's not the case can anyone tell me how to do it please ?

Sorry for my bad english and matroska is the best container biggrin.gif .
Canar
QUOTE(fenterbug @ Jan 7 2004, 08:33 AM)
I have separate .ogg files for each track in the album. (They each play perfectly in FB2K.) I append them into a single .ogg using copy /b 1.ogg + 2.ogg all.ogg. (The resulting file plays perfectly in FB2K, even distinguishing the individual tracks.) I then use the command posted by Peter above to make the mkv. When I load it into FB2K, I get, for each track, this:
ERROR (foo_matroska) : Matroska: no decodable streams found.

Is this an mkvmerge issue? A foo_matroska issue? A Foobar2000 issue?

From what I gather the problem is that Matroska doesn't understand that you want all three files to be appended to each other with chapter entries in between each. Instead, (looking at it from a movie encoding perspective) mkvmerge thinks that each individual track is a different "language" for the same movie.

The problem is in targeting. Matroska's made to handle movies; it's not yet been properly tuned and set up to handle audio nicely. There are some "essential" audio handling techniques that Matroska has yet to be made to handle nicely for album containment.

AFAIK, it's a combination of a mkvmerge and a foo_matroska issue. mkvmerge won't put chapter marks in between, and foo_matroska won't read the overlapping audio tracks as seperate tracks.
ChristianHJW
QUOTE(Canar @ Jan 7 2004, 03:24 PM)
@ChristianHJW: Your method 2 is kind of what I wanted to do. Kind of. I was hoping that there would be a method of concatenating each individual file (edit: into a single file)

There is, i described it in my post above. But, its time consuming. You first have to convert all your songs into single MKA's, and then use the matroska stream editor to concatenate them.
QUOTE
and automatically inserting the chapter point at the point of concatenation.
.... sounds good smile.gif .... let me pass this suggestion to the devs....
QUOTE
The problem is that each file that's concatenated will have to be pared down to the proper length by the decoder (assuming it's not inherently gapless, like MP3) which adds a little more complexity.
.... ??? Sorry, i dont understand you here ohmy.gif ....
Canar
QUOTE(ChristianHJW @ Jan 8 2004, 01:01 PM)
.... ??? Sorry, i dont understand you here ohmy.gif ....

Thanks for pointing the trick you did out again... I remember reading that, but it must not have sunk in until after I posted the response. wink.gif

For proper gapless MP3 playback (a la foobar2000), there have to be a certain number of samples stripped from the beginning and end of each individual MP3 stream. I believe that AAC is similar. I don't know precisely how many and/or how it works other than this basic explanation, but I'm sure Peter (or someone else knowledgable) would be available to elucidate that further.

I'm not sure if the Matroska spec supports this, as it doesn't seem to be directly applicable to anything other than this specific usage.

BTW, I can't seem to find a mirror site for the stream editor... a working link would be nice. wink.gif
ChristianHJW
QUOTE(fenterbug @ Jan 7 2004, 04:33 PM)
Okay, I finally found mkvmerge and have started playing with it. Like Canar, I'd prefer method #2, but I'll settle for method #1 until player support arrives. However, I can't get my .mkv to play in Foobar.
I have separate .ogg files for each track in the album. (They each play perfectly in FB2K.) I append them into a single .ogg using copy /b 1.ogg + 2.ogg all.ogg. (The resulting file plays perfectly in FB2K, even distinguishing the individual tracks.) I then use the command posted by Peter above to make the mkv. When I load it into FB2K, I get, for each track, this:
ERROR (foo_matroska) : Matroska: no decodable streams found.
Is this an mkvmerge issue? A foo_matroska issue? A Foobar2000 issue?

I talked to Mosu and he confirmed that this is indeed a mkvmerge issue. While the Ogg specs officially allow to concatenate Ogg files the way you do it, mkvmerge will not be able to handle the screwed page numbers in such files. Mosu also made clear to me that this issue is very low on his priority list, sorry sad.gif .....
ChristianHJW
QUOTE(Canar @ Jan 7 2004, 10:38 PM)
From what I gather the problem is that Matroska doesn't understand that you want all three files to be appended to each other with chapter entries in between each. Instead, (looking at it from a movie encoding perspective) mkvmerge thinks that each individual track is a different "language" for the same movie.

In his case the problem is that mkvmerge cant handle the big Ogg file he was trying to load, as it cant handle concatenated Oggs.

QUOTE
The problem is in targeting. Matroska's made to handle movies; it's not yet been properly tuned and set up to handle audio nicely. There are some "essential" audio handling techniques that Matroska has yet to be made to handle nicely for album containment. AFAIK, it's a combination of a mkvmerge and a foo_matroska issue. mkvmerge won't put chapter marks in between, and foo_matroska won't read the overlapping audio tracks as seperate tracks.


Please dont swob the container and the currently existing tools to create matroska files wink.gif . The tools, i agree here, are targeted onto video yet. But this is not a container limitation.

The only existing limitation for proper audio use seems to be the sample precise seeking, but as you can read above we are adressing this issue right now already. About the proper tools to make matroska easier usable for audio, well, we'll see if it gets accepted from the audio world, and if it will then i am sure we will see nice tools for matroska creation and handling soon wink.gif ....
ChristianHJW
QUOTE(Canar @ Jan 8 2004, 09:22 PM)
Thanks for pointing the trick you did out again... I remember reading that, but it must not have sunk in until after I posted the response. wink.gif
BTW, I can't seem to find a mirror site for the stream editor... a working link would be nice. wink.gif


Here is another tool that will allow to concatenate various matroska files into a single, long file :

http://www-user.tu-chemnitz.de/%7Enoe/Vide.../index-eng.html

QUOTE
For proper gapless MP3 playback (a la foobar2000), there have to be a certain number of samples stripped from the beginning and end of each individual MP3 stream. I believe that AAC is similar. I don't know precisely how many and/or how it works other than this basic explanation, but I'm sure Peter (or someone else knowledgable) would be available to elucidate that further.


Peter already found a workaround for this he told me, like reading 10 frames in advance or such. BTW, if you make one single big track from your songs, transition from one to the next will allways be gapless as it is a single file, of course wink.gif ....

QUOTE
I'm not sure if the Matroska spec supports this, as it doesn't seem to be directly applicable to anything other than this specific usage.

.. this is a player issue, not a matroska container thing that must be specified .....
Peter
QUOTE(ChristianHJW @ Jan 9 2004, 02:31 AM)
QUOTE
For proper gapless MP3 playback (a la foobar2000), there have to be a certain number of samples stripped from the beginning and end of each individual MP3 stream. I believe that AAC is similar. I don't know precisely how many and/or how it works other than this basic explanation, but I'm sure Peter (or someone else knowledgable) would be available to elucidate that further.


Peter already found a workaround for this he told me, like reading 10 frames in advance or such. BTW, if you make one single big track from your songs, transition from one to the next will allways be gapless as it is a single file, of course wink.gif ....

I've never said such thing, seeking 10 frames in advance is needed to avoid issues with MP3 bit reservoir when seeking in MP3 stream.
With current matroska tools ( == inaccurate seeking), track transitions will be "gapless" only in a player that plays entire big matroska file as one track.
Canar
QUOTE(ChristianHJW @ Jan 8 2004, 02:00 PM)
Please dont swob the container and the currently existing tools to create matroska files wink.gif . The tools, i agree here, are targeted onto video yet. But this is not a container limitation.

I wasn't intending to do this at all. I meant to imply that the tools have yet to be tweaked for these purposes, and I was uncertain whether or not Matroska was capable of it.

QUOTE(ChristianHJW)
.. this is a player issue, not a matroska container thing that must be specified .....


Ehm... I waffle a little on this one. How will the player know to do this unless it's specified in the container? If you were playing the entire stream straight through, the container would need to tell the decoder/player to chop off the last "X" samples of the previous MP3 stream as well as the first "Y" samples of the next MP3 stream for there to be a completely gapless transition. If the player was reading chapter-by-chapter, like the current FB2K implementation does, you could tell it only to play the gapless sections, but if it was reading the entire file as a single entity, you'd have to truncate the two streams like I said.

The player won't know to do this unless that information is contained within the container's metadata.
Pamel
QUOTE(Canar @ Jan 8 2004, 04:10 PM)
Ehm... I waffle a little on this one. How will the player know to do this unless it's specified in the container?

It is accurate though. As I mentioned earlier, it is certainly possible to create sample accurate seeking with the current specs. There is nothing in the specs that would prevent it, and there is provision to allow it, so I fail to see the issue as the container side. It is an "issue" with the available tools.

Also, although I didn't mention it above, it should be more than simple to get sample accurate seeking in the files using the 1ms precision for any codec that uses a constant number of samples per packet (anything except Vorbis). Because you know the samplerate, the number of samples per packet, and the aproximate time, calculating the exact time becomes extremely simple.

For an example, lets use MP3 at 44100Hz. Because its MP3, we know its 1152 samples per block. So, each Block takes around 26.12ms, (1152/44100=~0.02612sec). Lets assume that we have a Block with a timecode of 3997ms. We know that this number is going to be a little off because the the time length of each sample isn't a multiple of 1ms. This is what everyone has been complaining is the big problem. And, using simple mathematics that I learned as a 10 year old I will show you how to solve this.
3.997 / (1152 / 44100) = 153.0

So, its block 153. Now to find out the amount of rounding error.
3.997 - (153 * (1152 / 44100)) = 0.000265

This means the exact time of the Block should be
3996.734694ms

And, because the Chapter time is accurate to 0.000001ms, you have sample accurate seeking. Note that you would only have to do this calculation on a seek, not on every block you read.


As a side note, any 24, 48, 96, or 192kHz audio in MP3 will be sample accurate because they take up time amounts that are increments of 1ms. Also, calculating for Vorbis is similar, but just a little more complex.
Canar
I must not be coming across very clearly. I see that sample-accurate seeking is probably possible using Matroska. The issue I'm trying to get across is gapless MP3 playback when several independently encoded MP3s are concatenated into a single stream by Matroska.

The problem is that every individual MP3 needs to have a certain number of samples "chopped off" at the beginning and end of the file before it can be gaplessly appended to the output of the previous and next streams. The number of samples for each MP3 varies, so the decoder cannot guess in any way. The only way for this to happen decoder-side that I can see is if the Matroska stream contains some metadata telling the decoder to do this. Does the container presently hold this information? Can it?

To continue what you said though, what happens when you're attempting to get sample-accurate streaming with Vorbis? biggrin.gif

Note: I don't intend to flame Matroska here. I'm excited by the possibilities and want to help out by working through these problems with you guys. That's my intent, just so I don't get misconstrued here as well. wink.gif
Pamel
QUOTE(Canar @ Jan 9 2004, 07:52 AM)
To continue what you said though, what happens when you're attempting to get sample-accurate streaming with Vorbis? biggrin.gif

Each vorbis packet has a duration of 2^i samples, where 6<=i<=14. Vorbis only allows two values for i in a stream, typically 8 and 11. This means that the number of samples for each packet can range from

64 samples (2^6)
to
16384 samples (2^14)

But most streams will just have
256 samples (2^8)
and
2048 samples (2^11)

Because 256 is a multiple of 2048, you can just assume 256 samples per packet to make things easier. Using this, you can use about 5.8ms per packet (or some multiple of this). So, you basically follow the process that I showed above, but use 256 samples instead of 1152. Of course, this would bread if i=6 or 7. If you wanted to you could use 64 samples just to be safe, but the rounding becomes very close and I'm not sure it would be wise.
Peter
I can implement that but I don't like per-codec hacks. Is there a generic way to determine this number for each codec ? What about FLAC or AC3 ?
If I apply that to all timecodes, chapter durations will be still inaccurate because they're always multiply of 588 samples at 44100Hz for tracks ripped from CD (not that any end user going to notice it but it makes testing problematic because you can't simply bitcompare results to MP3+CUE output).
Pamel
For generic support you have two options.

1. Decode one Block and see how many samples are returned. Quick, but you have to decode a little bit of data first. Also, you may have to decode more than one block and average if there is anything like a bit resevoir.

2. Read the timecode of some set amount of Blocks into the stream. For example, Count 200 Blocks and see the timecode of the last one. Then you do
(X / Y) / Z

X = Timecode in nanoseconds.
Y = Duration of a single audio sample in nanoseconds. (1,000,000,000 / samplerate)
Z = Number of Blocks
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2008 Invision Power Services, Inc.