CUETools DB Options

[Eli]

I have only now become aware of Gregory S. Chudov's effort to develop CTDB (CUETools DB). I am very excited about this as I have actually been suggesting this to spoon (dbpoweramp's developer for over 5 years)

http://db.cuetools.net/about.php

for others that have missed it:

What's it for?
You probably heard about AccurateRip, a wonderfull database of CD rip checksums, which helps you make sure your CD rip is an exact copy of original CD. What it can tell you is how many other people got the same data when copying this CD. CUETools Database is an extension of this idea.
What are the advantages?

* The most important feature is the ability not only to detect, but also correct small amounts of errors that occured in the ripping process.
* It's free of the offset problems. You don't even need to set up offset correction for your CD drive to be able to verify and what's more important, submit rips to the database. Different pressings of the same CD are treated as the same disc by the database, it doesn't care.
* Verification results are easier to deal with. There are exactly three possible outcomes: rip is correct, rip contains correctable errors, rip is unknown (or contains errors beyond repair).
* If there's a match, you can be certain it's really a match, because in addition to recovery record database uses a well-known CRC32 checksum of the whole CD image (except for 10*588 offset samples in the first and last seconds of the disc). This checksum is used as a rip ID in CTDB.

What are the downsides and limitations?

* CUETools DB doesn't bother with tracks. Your rip as a whole is either good/correctable, or it isn't. If one of the tracks is damaged beyound repair, CTDB cannot tell which one.
* If your rip contains errors, verification/correction process will involve downloading about 200kb of data, which is much more than it takes for AccurateRp.
* Verification process is slower than with AR.
* Database was just born and at the moment contains much less CDs than AR.

How many errors can a rip contain and still be repairable?

* That depends. The best case scenario is when there's one continuous damaged area up to 30-40 sectors (about half a second) long.
* The worst case scenario is 4 non-continuous damaged sectors in (very) unlucky positions.

What information does the database contain per each submission?

* CD TOC (Table Of Contents), i.e. length of every track.
* Offset-finding checksum, i.e. small (16 byte) recovery record for a set of samples throughout the CD, which allows to detect the offset difference between the rip in database and your rip, even if your rip contains some errors.
* CRC32 of the whole disc (except for some leadin/leadout samples).
* Submission date, artist, title.
* 180kb recovery record, which is stored separately and accessed only when verifying a broken rip or repairing it.

[Teknojnky]

methinks the record companies would have a little issue with the sampling and reconstruction of their copywritten audio

[Eli]

methinks the record companies would have a little issue with the sampling and reconstruction of their copywritten audio

There is no sampling. There is no audio storage. Without the nearly complete audio a repair file is worthless. It can't be played. There is no copyright issue. This database is live and already happening. It may be below the radar. There is no legal violation. Thats not to say big business couldn't bury it financially, but there would really be little to no incentive to attack it. This is the main reason that spoon gave for not going forward with this type of idea when I suggested it years ago though.

@Eli
How much does such a HD drive cost? You answer that.
How many new CD releases per month? 1000? Recovery data submission bandwidth is under control.

There is also no need to have a track based DB (10x size) for archival purposes.

Clearly the current DB is reasonable for 1 person to host. The idea was to pay for a larger, more robust DB, in order to cover those costs. As I suggested, maybe the best solution would be to offer free access to the current DB structure, and a for fee access to the larger track based DB.



This post has been edited by Eli: Apr 20 2010, 21:57

[Gregory S. Chudo...]

methinks the record companies would have a little issue with the sampling and reconstruction of their copywritten audio

I don't expect any problems on this front. There are many databases which store various information about CDs - freedb, musicbrainz, discogs and AccurateRip to name a few. And now CTDB. None of them ever had any legal problems, because none of them store any actual audio, they all store various meta-information - CD TOCs, artist names, crcs, etc. Recovery records obviously fall under the same category.

Clearly the current DB is reasonable for 1 person to host. The idea was to pay for a larger, more robust DB, in order to cover those costs. As I suggested, maybe the best solution would be to offer free access to the current DB structure, and a for fee access to the larger track based DB.

I'd like to keep this project absolutely free. Current database contents was submitted by the users and it belongs to the users. If i'm not mistaken, Musicbrainz uses Creative Commons license for it's content. CTDB will use the same, if there are no objections. If hosting costs will become significant at some point, i think we will work something out without compromising the database freedom.

This post has been edited by Gregory S. Chudov: May 7 2010, 21:26