Error Handling, How to test? Options

[kanak]

Hi

I'm planning to run some damage tests to check TAK's error resilience and see how it fares against existing codecs. I plan on using TBeck's Damage tool to damage the files. My questions are:

0. How was the error handling column of the Lossless Comparison Wiki made? What does "yes" in the error handling column- that the file is still playable?

1. How do i quantitatively check for damage?
I was planning to see if the files are playable with foobar2000, to see if they decode to wav without any problems (using the commandline decoder e.g. wavunpack for wv files). Is there a more "scientific" way to do it?

TBeck had this to say about TAK:

Tak's decoding error protocol will show you something like this:

Header frames: 321
Valid frames: 320

Of the 321 frames of the file only 320 frames could be restored, hence one frame is lost.

You could compare the count of errors you have applied to the file with the count of lost frames.

Additionally you could calculate a total count of lost samples by multiplying the lost frames count with the frame size (in samples) which can be taken from the "Skipped data blocks" table:

CODE

No       Source pos  Size        Sample pos  Count
       1     1069674       30665      429936        5512
       2     2014491       34040      804752        5512

"Count" usually equals the frame size. Exceptions: Two succeeding frames have been damaged (then it's twice the frame size) or the last frame has been damaged (most of the time the last frame is smaller). Well, i should have added a total count of lost samples to the table...

Important: If you get an "Undecodable"-error you have to disable the "Restore wave file meta data" option to decode the damaged file: -wm0

Is there a way to do something similar for other codecs?



2. Do i need to check all settings of a codec to get the representative picture?
I was planning on checking the "extremes" only: e.g. Flac 0 & Flac 8, Wv Fast and HHx6, Monkey at Fast and Extra High. Will i get significantly different results if i compare the entire set (Flac 0 through 8 etc)?

TBeck had this to say about TAK:

There are two important structures within a Tak file:

1) The container is the same for every preset. Here it's enough to test one preset.
2) The encoded audio frames. Here it could be important, that the preset used for damage testing is using any of the internal compression methods of Tak. This is true for presets 3 and 4.

I would try -p3 or p3m. This should be sufficient.

Similarly, is testing say Flac 0 and Flac 8 "sufficient" (i.e. is it enough to establish a trend)?

3. Could anyone share their experiences in damage testing/ give tips or suggestions or anything?
i did check rjamorim, guruboolez and greynol's discussion about monkey's audio's error handling in the discussion for "which lossless codec is the best?" But it seemed like the tests that were done there mainly checked if the files were playable after damage.


Thanks
Kanak

[TBeck]

TBeck had this to say about TAK:

In earlier discussions the size of the audio data loss caused by a single error seemed to be relevant. Usually it should depend on the frame size the encoder is using.

Two remarks:

1) The frames have to be totally independend, otherwise errors in one frame will affect the next frame too. In Mpeg4Als for instance only the random access frames can be independently decoded.

2) It's possible that a single error affects two independend frames, if it hits the end of the first and the start of the next.

For Tak the frame size depends on the preset and the sample rate. For 44 KHz it lies between 4134 and 11025 (94 to 250 ms) samples. In any case the upper limit is 16384 samples.

[TBeck]

Well, i want to try a bit more systematic approach (still a brain storming):

Possible test scenarios

Test 1: Damage file header

We want to know, if damage or loss of the file header makes the whole file undecodable. Without knowing the internals of the codec, we can not tell how large the file header will be. Therefore we probably should damage for instance the first 4 KB of the file. Cut it off or overwrite it with zero.

Hm, there can be a seektable in the header. Because the size of the seektable often depends on the file size, we probably have to damage more than 4 KB to be sure to have damaged the whole header.

Another possible rule: Alternatively damage 0.25 percent of the file size bytes at the beginning, if this value is greater than 4 KB.

Result: Can the file still be decoded? Yes/No

Test 2: Check the size of data loss caused by a single error:

5 runs per file. In each run apply exactly one error to the file. Distribute it over the whole file. Calculate the average data loss in samples over all runs.

Standardization is always a good thing to make results comparable, hence we should possibly damage file positions defined as 10, 30, 50, 70 and 90 percent of the file size.

Here we don't want to damage the header or footer of the file. With the damage start at 10 and the end at 90 percent we should be save.

Result: Average data loss in samples per error.

Test 3: Severe Damage / Multiple errors

15 errors per file.

Lets use the same positions as in 2): 10, 30, 50, 70 and 90 percent of the file size.

At each position create 3 errors with a distance of about 50 bytes. Some kind of a burst error (not quite right). Possibly the decoder can handle a single error in a frame, but not 3...

Result: Sum of data loss in samples.

Which presets should be tested?

Often different presets will use different compression methods. Because some compression methods may be more sensitive to data damage, we should use a preset, that uses all compression methods of the codec. Unfortunatly we often will not know enough about the codec to find such a preset. And sometimes there will be no preset that combines all methods.

Options for a selection:

1) Check any preset.
2) Check the fastest and the strongest preset.
3) Check the stongest preset.
4) Check the most commonly used preset.

In Tak the frame size varies depending on the preset. Therefore "Test 2: Check the size of data loss caused by a single error" will give different results for the presets.

How many files should be tested?

Many! If you are testing only a couple of files, there is a fair chance, that you hit some weak point of one unlucky codec and miss any weakness of another lucky codec.

I would use at least 20 files.

If we would like to go up to 46 files, we could use the nice files hosted at rarewares which i am calling my primary test corpus. Hm, possibly a good idea to use this as a reference... Unfortunately it does not contain big files. We should have some of them too.

Well, some quick thoughts as a base for further discussion.

[TBeck]

Well, i want to try a bit more systematic approach (still a brain storming):

Important: The post above was not meant as a guideline for the current Tak alpha testing!

The idea was to provide a starting point for the development of a comprehensive testing methodology. This will need far more thinking and discussion and hence take some time.

If we should find a good testing paradigm, i possibly will write a specific tool to support it.