IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Vorbis stream with no Ogg header
stick
post Feb 25 2006, 13:36
Post #1





Group: Members
Posts: 3
Joined: 25-February 06
Member No.: 28054



Hi!
Last night I broadcasted signal grabbed from my soundcard's line-in. I used OddCast+IceCast. OddCast was set to save archive of stream to disk. After hour IceCast crashed resulting in OddCast reconnect. After the broadcast I found two dumped OGG files, but first of them had no OGG container header (no "vorbis" text present in header). Second file is OK. If I'm getting it right, Vorbis within Ogg container has the following format:

CODE
OggS <ogg_header> OggS <vorbis_header> OggS <vorbis_data> OggS <vorbis_data> OggS <vorbis_data> ...


but I ended only with

CODE
OggS <vorbis_data> OggS <vorbis_data> OggS <vorbis_data> ...


I would like to know if there is any chance to reconstruct header of the first file (put it again in another Ogg container) or how to play it or convert it to another format.

Thanks in advance ...
Go to the top of the page
+Quote Post
HotshotGG
post Feb 25 2006, 22:12
Post #2





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



If the stream has no header then you are probably not going to be able to salvage it. The header is the most important part of the stream due to the fact that it contains all of the setup information including codebooks, granulepos, vendor string, etc. Try running the file through ogginfo.exe and see what it tells you about the file.


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
stick
post Feb 26 2006, 00:55
Post #3





Group: Members
Posts: 3
Joined: 25-February 06
Member No.: 28054



QUOTE (HotshotGG @ Feb 25 2006, 11:12 PM)
Try running the file through ogginfo.exe and see what it tells you about the file.
Warning: Invalid header page, no packet found
Warning: Invalid header page in stream 1, contains multiple packets
New logical stream (#1, serial: 000041d0): type invalid
Warning: stream start flag not set on stream 1
Warning: sequence number gap in stream 1. Got page 166557 when expecting page 0.
Indicates missing data.
Warning: EOS not set on stream 1
Go to the top of the page
+Quote Post
HotshotGG
post Feb 26 2006, 03:21
Post #4





Group: Members
Posts: 1593
Joined: 24-March 02
From: Revere, MA
Member No.: 1607



QUOTE
Warning: Invalid header page, no packet found
Warning: Invalid header page in stream 1, contains multiple packets
New logical stream (#1, serial: 000041d0): type invalid
Warning: stream start flag not set on stream 1
Warning: sequence number gap in stream 1. Got page 166557 when expecting page 0.
Indicates missing data.
Warning: EOS not set on stream 1


A lot of people who are streaming from an icecast or oddcast server have this problem. Basically what I am going to tell you to do is attempt to use vcut.exe to try and cut the stream at specific granulepos. Seeking with an Ogg container is relative to wherever the granulepos is set within a page. You might be able to salvage or whatever is left of it if you truncate it down past a specific position. This is the specific problem that of people who streaming with Vorbis complain about. Why was the Ogg bitstream designed with all of the codebooks and other specific packet information at the start of the stream? the logical and physical bitstream structure though was very clever. I can't think of anything else , although that might be worth a shot.

This post has been edited by HotshotGG: Feb 26 2006, 03:22


--------------------
College student/IT Assistant
Go to the top of the page
+Quote Post
stick
post Feb 26 2006, 20:11
Post #5





Group: Members
Posts: 3
Joined: 25-February 06
Member No.: 28054



QUOTE (HotshotGG @ Feb 26 2006, 04:21 AM)
Basically what I am going to tell you to do is attempt to use vcut.exe to try and cut the stream at specific granulepos.

CODE
>vcut.exe k.ogg 1.ogg 2.ogg 10

WARNING: vcut is still experimental code.
Check that the output files are correct before deleting sources.

Processing: Cutting at 10 samples
Input not ogg.
Error reading headers
Processing failed


didn't work crying.gif any other ideas?
Go to the top of the page
+Quote Post
illiminable
post Mar 12 2006, 02:56
Post #6





Group: Members
Posts: 36
Joined: 1-July 04
From: Australia
Member No.: 15011



QUOTE (stick @ Feb 27 2006, 03:11 AM)
QUOTE (HotshotGG @ Feb 26 2006, 04:21 AM)
Basically what I am going to tell you to do is attempt to use vcut.exe to try and cut the stream at specific granulepos.

CODE
>vcut.exe k.ogg 1.ogg 2.ogg 10

WARNING: vcut is still experimental code.
Check that the output files are correct before deleting sources.

Processing: Cutting at 10 samples
Input not ogg.
Error reading headers
Processing failed


didn't work crying.gif any other ideas?
*



You can probably save it... but you'll have to do it manually.

Your post indicates you know how the headers should be at the start of the file. ie Main header, comment header, code book header.

In order to replace the headers you need to know what specifically they were... this might be a bit of trial and error, however there's a fairly good chance that they were a fairly standard configuration. There's a few things that can vary... the stuff in the first header is fairly likely to be easy to guess, ie it's probably 44100 stereo, grab the first page from some other vorbis file that is the same. The comment header is irrelevant, you can put any old comment header in there, you can always use a tool to change the comment header later.

As to the third header, this will depend on the encoder settings use to produce the file. You may be lucky and it's a fairly "default" encode, in which case you will probably be able to cut this header from a file that was encoded in a similar way, ie with the same encoder settings. There's a fixed set of codebooks, if this file was made by something other than the reference encoder, you may have to do a lot of trial and error to find one that will work.

Basically, what i would do is... get a hex editor, make a backup copy of your file. Then from the start of the file, look forward until you see the first OggS, if there's one right at the start, do nothing, if not, cut everything away upto the first one... that small bit of data is not likely to be salvagable.

Now you have a file with a clean page start. Now try inserting the headers... this will be the first 2 or sometimes 3 pages you will have to do some mental decoding to see this, though if the third page has granule pos 0, it has the continuation flag set, has only packet and it's fairly small, chances are this 3rd page is part of the headers. Cut this section from some other file, which you beleive was similarly encoded.

Save it into it's own file, then make another copy of this little file.

Now look in your broken file, look at the serial number field, now, in your little file with the headers, you need to edit the serial number field. Now you have a problem, the checksum on the header page will be wrong. There is no easy way around this unless you can do some programming to recalculate what it should be. Most player will reject the file if the checksums are incorrect.

However, i know my directshow filters have a bug and they don't check the checksum... so at this point you could maybe try and play the file, and you will at least know if you guessed the right headers, and then you can worry about fixing the checksum later.

On the downside, my filters scan the whole file at load time, so if there is something funky later on in the file... or if this file is chained (which my codecs don't handle yet), it may barf right there. A workaround, is to put it on the web, and use my filter to load url..., since it does not scan the whole file in a network stream, so it will play (assuming the headers are right) until it reaches something that is not right, and then either stop unexpectedly, or possibly crash smile.gif

Once you know you have he right headers, then you fix the header page checksums... and possibly fix up any garbage that is at the end of the file.

So, all in all, it's almost definately possible to save this file... but it requires some knowledge of the bitstream, and some programming skill, there is nothing automated that i know of that will fix this, and it's unlikely there ever will be, because there's no easy way to guess which headers should be used. Though someone with a very in depth knowledge of the vorbis packets could possibly make some reasonably educated guesses.

So ask yourself... how bad do you want to save this file... and how much time are you willing to expend to do so.

If you send me the file, use www.yousendit.com and enter my email address ogg@illiminable.com after you upload it there, it will send me a link to download it from the yousendit server. I will try to have a look at it, but i can't make any promises that i will actually be able to fix it, or that i will be able to get around to quickly, so if you what i wrote above sounds like you can do it, then i'd suggest you do it, as you are obviously far more motivated to recover this file than me.

In order for this to even be possible i would need as much information about what settings you think it was encded at, also another file (probably another file from this ssame stream) that you suspect was encoded the same way, what version of icecast/vorbis encoder/other software used etc. Even in the best case this is time consuming. You should hope that it was made with the reference encoder(ie libvorbis_, if it was made with any other encoder (ie AoTuV) it may not be possible.
Go to the top of the page
+Quote Post
lubot
post Aug 22 2007, 01:16
Post #7





Group: Members
Posts: 5
Joined: 23-December 02
Member No.: 4208



I know this posting is rather old, but you can fix broken Ogg Vorbis streams with Avi-Mux GUI. These fixed files can be cut wit vcut.exe.

EDIT: My fault, it doesn't work if the header is missing.

This post has been edited by lubot: Aug 23 2007, 19:25
Go to the top of the page
+Quote Post
SebastianG
post Aug 22 2007, 08:25
Post #8





Group: Developer
Posts: 1317
Joined: 20-March 04
From: Göttingen (DE)
Member No.: 12875



Havn't tried Avi-Mux but I'd be verysurprised if it could generate proper headers out of nothing


commenting on post #6:

One could also encode another sample (of the same format) with the same settings (including the serial number -- it can be forced afaik). Since the serial number is now the same you can just try to copy the first OggS pages (upto the setup header) and place'em in front of your recorded stream. No manual CRC calculation necessary.

Cheers!
SG

This post has been edited by SebastianG: Aug 22 2007, 08:26
Go to the top of the page
+Quote Post
j7n
post Aug 29 2008, 22:28
Post #9





Group: Members
Posts: 809
Joined: 26-April 04
Member No.: 13720



Why was Ogg container designed so complex? The inability to cut Vorbis streams is a serious disadvantage compared to MPEG formats. Would it seriously hurt quality if each Ogg frame was self sufficient? To minimize overhead frames (Ogg pages?) could simply be longer.
Go to the top of the page
+Quote Post
Yirkha
post Aug 30 2008, 01:45
Post #10





Group: FB2K Moderator
Posts: 2359
Joined: 30-November 07
Member No.: 49158



Unlike 4 Bytes MPEG frame header, Vorbis stream header is ~2 kB long, so including it in every frame would really be an overhead. I'm not such an expert to say why, it might contain some dynamic codebooks or whatever.

The claim about "inability to cut Vorbis streams" is obviously false. All what is needed is that programs which save Ogg Vorbis streams save them properly. I don't know why it was broken in this case regardless of the crash, Ogg is supposed to be written linearly.

However I wonder why nobody suggested to the OP to take the headers from the second file from the same session, which was fine. The headers would be the same because it was apparently one encoding session.

Edit: Some less BS.

This post has been edited by Yirkha: Aug 30 2008, 01:46


--------------------
Full-quoting makes you scroll past the same junk over and over.
Go to the top of the page
+Quote Post
Canar
post Sep 27 2009, 15:52
Post #11





Group: Super Moderator
Posts: 3327
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



I have the same problem as stick. I used wget to grab a chunk of a stream, but no tool I can find will let me cut out the interesting bits. They all crash or hang. foobar2000 plays the file beautifully.

There's no way to rebuild a stream, hey?

This post has been edited by Canar: Sep 27 2009, 15:53


--------------------
∑:<
Go to the top of the page
+Quote Post
Yirkha
post Sep 27 2009, 21:27
Post #12





Group: FB2K Moderator
Posts: 2359
Joined: 30-November 07
Member No.: 49158



Just for reference, I've heard Canar later used this beautiful tool made by some great individual with great success.


--------------------
Full-quoting makes you scroll past the same junk over and over.
Go to the top of the page
+Quote Post
Canar
post Sep 27 2009, 22:44
Post #13





Group: Super Moderator
Posts: 3327
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



QUOTE (Yirkha @ Sep 27 2009, 16:27) *
Just for reference, I've heard Canar later used this beautiful tool made by some great individual with great success.
And then promptly forgot to give said great individual some great props.

Seriously Yirkha, thanks a ton!


--------------------
∑:<
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 12:05