I was trying to work out the proper way of parsing MPEG data and ran against the following questions (i would be glad if those who answer provide links to official documents (standards, technical specifications) or quotes from documents with names of the documents):

• When trying to synchronize the parser against MPEG frame header syncbits, the usual way is to read up until a certain pattern is met (some bit sequence). Then check the bitrate, sampling rate and other options from the header. Then there are two codepath options:
  1. If the bitrate is not 'free' calculate the frame length and check to see if there's a MPEG frame sync at the calculated offset (and better do it several times for at least 3 consequent frames to ensure proper synching)
  2. If the bitrate is 'free' - find the next MPEG frame sync, check if it is a valid MPEG header and calculate the distance between the synchbit sequences (frame size in bytes), so that the actual 'free' bitrate could be calculated too (and again do it in a loop for at least 3 iterations)

Things are clear so far, but the following problems and questions arise:

• What is a good behaviour for a code if the bitrate is a fixed value from the table and there's no MPEG frame synch at the calculated offset (broken frame sequence probably)?
• Should i try and find the next frame after the 'garbage' is encountered in a file?
• Is it allowed for MPEG frame sequences to be broken (to include 'garbage' between frames)? What's a good behaviour for the decoder then? In other words, should frames lie one right after another in a file? Is the sequence broken in the other case and the program should abort processing?
• What to do if there's only one or two consequent legal frames in the file (VERY short file) ? is it ok to handle these frames, or is there a minimum size and maximum size requirement a MPEG frame sequence should meet?

I'd suggest you to split you post into several small questions, otherwise lazy people (like me) would perhaps not take the time to read your long list of questions.

Problems and questions regarding 'free' bitrates come up next. There are at least three descriptions of size of a frame with a 'free' bitrate:

1. According to the ISO 11172-3 specification:
   QUOTE
   The all zero value indicates the 'free format' condition, in which a fixed bitrate which does not need to be in the list can be used. Fixed means that a frame contains either N or N+1 slots, depending on the value of the padding bit. The bit_rate_index is an index to a table, which is different for the different Layers.
   
   So, 'free' bitrates can be from 1 bit per second to an infinite value, but must remain constant within all of the frames in the sequence. No restrictions on the bitrate value.
2. Next we go to <a href="http://www.mp3-tech.org/programmer/frame_header.html" target="_blank">http://www.mp3-tech.org/programmer/frame_header.html</a> (I'd say it is the most visited site for programmers coding mpeg-related stuff):
   QUOTE
   "free" means free format. The free bitrate must remain constant, an must be lower than the maximum allowed bitrate. Decoders are not required to support decoding of free bitrate streams.
   
   Again, 'free' bitrate must remain constant, but in this case some restrictions on the value are mentioned - a 'free' bitrate must be less than the maximum allowed bitrate for the certain MPEG version, layer, etc...
3. I do not remember the links, but was surfing in search for info and found several board discussions, where it has been said that a 'free' bitrate should fall in the range of minimum and maximum allowed values for current MPEG version, layer, etc...

Now, regarding 'free' bitrates, please answer:

• Which of the three statements lie and which one is true?
• How do i calculate the actual 'free' bitrate?
• Are there any requirements a 'free' bitrate value should meet (size restrictions)? Please, comment on that.
• What if there's only one MPEG frame with a 'free' bitrate in a short file? How do you determine the frame size in the following cases:
  • It is a short file (one MPEG frame)?
  • A decoder is not ID3v1-aware, and does not recognize appended tags at all - should it read from a short file (one frame) until the EOF is met and consider the whole read data to be a single frame?
• What is a condition to stop frame scanning? Zero bytes encounter? 'Garbage' bytes encounter? Stop, when the maximum frame length (if any) is reached and sync is not met?
• Is the sequence required to include at least two consequent frames with 'free' bitrate for proper parsing?

Going further, i would propose to develop an algorithm for MP3 file parsing, as this would be useful for the readers of this forum. The proposed algorithm is a WIP and requires modification, bugfixing and checking against standard compliance and conformance to technical specifications:

• First, cut off all the prepended (id3v2, etc...) / appended (lyrics3, id3v1) tags - could parse the tags (that is documented enough), or could just throw the tags away.
  
• Read up to N bytes from the file to a buffer, where N is suitable for most of encoders. This could be 32Kbytes, 64Kbytes or greater (depends on the tolerance of the parser). A value of 32Kbytes is enough, i suppose.
  
• If there's no MPEG frame sync in the first 32Kbytes - this is not a MPEG file, abort the execution.
  
• If a MPEG frame sync is found within the first N bytes:
  
  • If it is not a valid MPEG frame header (validate against allowed combinations of version, layer, sampling rate and other data) - continue searching for the next frame sync
    
  • If it is a valid MPEG frame header - check out the bitrate:
    
    • If it is a fixed value from the table - calculate the size (easily done) and see if there's enough data in the buffer to check for the next synch, otherwise try searching for the next valid MPEG frame header in currently buffered data and calculate the frame size and the bitrate.
      
      • If bitrate is 'free' for the first MPEG frame, and the second MPEG header is not found in the buffered data - we could try to read up to next N bytes and search there (probably, an extremely high bitrate? need to work this out, if there are size limitations on a MPEG frame with a 'free' bitrate). If EOF is reached (or size limit is reached, if exists), and the second MPEG is not found in buffered data - we could abort with 'first N bytes contain an incomplete MPEG frame' message, or consider the whole data to be a single frame and process it as is.
        
      • If bitrate is a fixed tabular value (index from a table) and data is not enough and EOF is not reached, buffer more data until up to size_of_current_frame + 4 consequent bytes are buffered (the size of MPEG frame header is four bytes). If EOF is reached during buffering before all insufficient bytes are read, then we have a broken first frame or a false synch, and the file is quite short, so abort the parsing and report of the incomplete / invalid MPEG sequence.
        
      • If data is enough, repeat checking for at least 3 consequent frames (including the first one), calculating the bitrate and frame size on each iteration (might be a VBR without FHG or LAME VBR header, or 'free' bitrate) and buffering more data if needed:
        
        • If there is only one frame in the buffer, EOF is reached and there was no garbage / zero bytes appended - (the only frame ends up right at the end of the file) - this is a valid MPEG sequence, process the frame as it was the first frame of any legal MPEG frame sequence. If the first frame has 'free' bitrate - we know what to do.
          
        • If there is only one frame, but with some data after it - check the data. If the four successive bytes represent a valid MPEG frame header - parse the first frame, but report of the probably incomplete sequence. If the four successive bytes are garbage or zeroes - process the first frame, but report about the garbage at the end of the sequence.
          
        • If there are two frames in the buffer, EOF is reached and there was no garbage / zero bytes appended - (the second frame ended right at the end of the file) - this is a valid MPEG sequence of two frames, process both. First of all, we check bitrates in both frames. Those should be both 'free', and frame sizes of both frames should be equal in this case. If the first frame is free, and the second one is not - we got a broken sequence ('free' bitrate must remain constant (sizes should also be equal) for all frames) - abort and report. In other cases the frame sizes are well-known (constant or vbr).
          
        • If there are only two frames with some data after the second frame - check if the succeding data has a valid MPEG frame header following the second frame (incomplete third frame), process the first two frames (and mind the 'free' bitrate validation) and report of a probably broken sequence. If the data after the second frame does not begin with a valid MPEG frame header - process the first two frames (again, remember to check for 'free' bitrates and sizes) and report about the garbage at the end of the sequence.
      • If three consequent valid frames are found - we're almost definitely synched. Do parsing.
        
      • If the first frame was of 'free' bitrate all subsequent frames should be of the same 'free' bitrate and size. Otherwise we are dealing with a broken sequence.
        
      • If the first frame is long enough and there's a FHG VBR header present in it - the VBR and duration can be carried out of that header.
        
      • If the first frame is long enough and there's a Xing Tag/Info Tag/LAME header - the VBR/ABR and duration can be carried out of that header.
        
      • If first M frames are all CBR - the whole data is most likely CBR, we can calculate bitrate and duration of the sequence (M can be anything and should also be tuned as a 'tolerance' parameter of the parser, i would propose to set M to 32 or 64 frames - this should be enough, but for those who do not need any optimization, do not care about the parsing time and want a precise calculation - M should be 'all of the frames in sequence')

For now, this should be enough and failproof to determine the bitrate / duration of the MP3 file (the most common task). If not, please correct this. And if additional parsing is needed - this can also be implemented, making a parser to be a decoder. Plus additional checks for sequence integrity could me implemented for a parser, involving the byte reservoir.

I'm going to implement this in a day or few in php (can be easily converted to anything else), publish the code and test results here. Meanwhile, i would be glad if anyone in this forum could check and comment on the algorithm, reply on this whole post and answer the questions from the top of this post.


Tried to do that, but the invision board engine concatenates consequent posts if from same author... the first post is now short enough, but can't split the rest of text...

Perhaps the sources of getID3() can be useful for you.
getID3() is a PHP script that extracts useful information from MP3s & other multimedia file formats. It is available under the terms of GNU GPL.

I know what getID3 is, but don't think that getID3 is a good example of MPEG handling to look at, as I have reported on bugs in getID3 code for several times... Anyway, I've examined getID3 code in search for a proper solution before posting on this forum, and did not find answers for my questions in their code. And in fact getID3 code inspired me to dive deeper in the subject and perform such a research.

I have a little experience with this, atm. I'm using the following to re-sync:
- Check for an end-tag header @ the calculated position: end-of-stream or stream-concatenation;
- If no tag-header or in case there's bytes after the end-tag: scan for a new MPEG header AFTER the calculated position.

I tried scanning from an arbitrary position inside the last found frame, but that gave me too much rubbish. That was because I wasn't using a validation of the next frame header after a newly found one.
So atm. I'm using validation of four bytes from the next frame in scanning for / validation of a new MPEG header.
I'm also using a preliminary scan-area which is two frames wide in case the stream is predicted to be a CBR one.


I would suggest so.


I guess a nice-&-neat MPEG audio stream does not contain any rubbish. I think it depends on the decoder to qualify part of a stream as rubbish or out-of-sync. This might for example depend on how the decoder calculates the amount of bytes in a frame. Now while this seems fairly straightforward, I've seen MPEG-1 L.I streams encoded by the MainConcept encoder containing frames which seem too long, or frame headers having the first (sync) byte chopped off / overwritten. Not nice if you want your decoder to detect that, but on the other hand, skipping one frame will not do much harm.
Anyway, a lot of redundancy in a decoder will slow it down, so I guess a trade-off has to be made.


I have never read anything about a minimum or maximum number of frames. Again, if you want to be able to decode a one-frame or a two-frame MPEG audio stream you'll probably have to take these situations into account. Especially in case you're using a fixed size window / buffer for reading the stream. I think a decoder can distinguish itself by using a flexible stream-reading window, maybe even adapting to such exotic situations as you point out. In practice however, I've seen decoders which simply crash @ a one-frame stream.