Detecting whether a 24-bit file has been upconverted from 16-bit? Options

[vincefalks]

I downloaded a FLAC album from online web store and it appears that the files I've downloaded are 24-bit. it's not a hd web store like hdtracks, it's generally a CD 16-bit 44.1kHz store with either FLAC or MP3 to choose from in its offerings.

I'm wondering whether the 24-bit files are real and were supplied as 24-bit from the label and they just didn't label it correctly on the download page, or whether there was an encoding error or just somehow it made its way from 16-bit files to 24-bit FLACs.

Is there a way I can definitively tell or analyze it (looking at spectral maybe) to see if it's been upconverted? Are there certain algorithms and 'good jobs' that can be done to make its 16->24 upconversion undetectable?

happy to upload a sample if it helps.

thanks
vince

[nu774]

Try something like the following to see if wasted bits is equal to or greater than 8 for each FLAC sub-frames.

CODE

flac -ac foo.flac | findstr wasted_bits (for windows)
flac -ac foo.flac | grep wasted_bits (for Unix like OS)

[bandpass]

Are there certain algorithms and 'good jobs' that can be done to make its 16->24 upconversion undetectable?

You could do it with a compander, and some all-pass filtering for good measure, but you'd probably not like the results: the quiet bits would likely be too quiet.

[spoon]

>Are there certain algorithms and 'good jobs' that can be done to make its 16->24 upconversion undetectable?

I think if they converted frequencies up and down it would be difficult to spot as there would not be 00000's in the lower byte.

[skamp]

Try something like the following to see if wasted bits is equal to or greater than 8 for each FLAC sub-frames.

Thanks! I used your idea to make a BASH script that prints out the average number of effective bits of FLAC files.

fbits:

CODE

#!/bin/bash

me="${0##*/}"

if [ -w "$TMPDIR" ]; then
tdir="$TMPDIR"
elif [ -w '/tmp' ]; then
tdir='/tmp'
elif [ -w "$HOME" ]; then
tdir="$HOME"
elif [ -w "$PWD" ]; then
tdir="$PWD"
else
echo "$me: error: can't find a writable directory for creating the temporary file" 1>&2 ; exit 1
fi

tf="$( TMPDIR="$tdir" mktemp "${tdir}/${me}.XXXX" 2>/dev/null )"
if [ -z "$tf" ]; then
echo "$me: error: can't create temporary file" 1>&2 ; exit 1
fi

checkbits ()
{
local bps abps tbps=0 n=0
bps="$( metaflac --show-bps "$1" )"
flac -ac "$1" 2>/dev/null | fgrep 'wasted_bits' | cut -d '=' -f 3 | cut -f 1 > "$tf"
while read wb; do
tbps=$(( tbps + ( bps - wb ) ))
((n++))
done < "$tf"
abps=$(( ( ( tbps * 10 / n) + 5 ) / 10 )) # (* 10 + 5) / 10 for proper rounding
printf "%2u/%2u bits\t%s\n" "$abps" "$bps" "$1"
}

for f in "$@"; do
case "$f" in
*.flac) checkbits "$f" ;;
*) continue ;;
esac
done

rm -f "$tf"

Usage:

CODE

fbits *.flac

Output with a lossyFLAC album (Daft Punk - Homework):

CODE

12/16 bits 01. Daftendirekt.lossy.flac
11/16 bits 02. WDPK 83.7 FM.lossy.flac
11/16 bits 03. Revolution 909.lossy.flac
10/16 bits 04. Da Funk.lossy.flac
11/16 bits 05. Phœnix.lossy.flac
10/16 bits 06. Fresh.lossy.flac
11/16 bits 07. Around the World.lossy.flac
10/16 bits 08. Rollin' & Scratchin'.lossy.flac
11/16 bits 09. Teachers.lossy.flac
10/16 bits 10. High Fidelity.lossy.flac
10/16 bits 11. Rock'n Roll.lossy.flac
12/16 bits 12. Oh Yeah.lossy.flac
11/16 bits 13. Burnin'.lossy.flac
10/16 bits 14. Indo Silver Club.lossy.flac
10/16 bits 15. Alive.lossy.flac
11/16 bits 16. Funk Ad.lossy.flac

Output with a "true" 24 bit album (The Beatles - Love):

CODE

23/24 bits 01. Because.flac
22/24 bits 02. Get Back.flac
22/24 bits 03. Glass Onion.flac
23/24 bits 04. Eleanor Rigby ∕ Julia (transition).flac
22/24 bits 05. I Am the Walrus.flac
21/24 bits 06. I Want to Hold Your Hand.flac
22/24 bits 07. Drive My Car ∕ The World ∕ What You're Doing.flac
23/24 bits 08. Gnik Nus.flac
22/24 bits 09. Something ∕ Blue Jay Way (transition).flac
22/24 bits 10. Being for the Benefit of Mr. Kite ∕ I Want You (She's So Heavy) ∕ Helter Skelter.flac
21/24 bits 11. Help.flac
22/24 bits 12. Blackbird ∕ Yesterday.flac
22/24 bits 13. Strawberry Fields Forever.flac
22/24 bits 14. Within You Without You ∕ Tomorrow Never Knows.flac
22/24 bits 15. Lucy in the Sky With Diamonds.flac
22/24 bits 16. Octopus's Garden.flac
22/24 bits 17. Lady Madonna.flac
22/24 bits 18. Here Comes the Sun ∕ The Inner Light (transition).flac
22/24 bits 19. Come Together ∕ Dear Prudence ∕ Cry Baby Cry (transition).flac
22/24 bits 20. Revolution.flac
22/24 bits 21. Back in the U.S.S.R..flac
23/24 bits 22. While My Guitar Gently Weeps.flac
22/24 bits 23. A Day in the Life.flac
22/24 bits 24. Hey Jude.flac
22/24 bits 25. Sgt. Pepper's Lonely Hearts Club Band (reprise).flac
22/24 bits 26. All You Need Is Love.flac

Output with a 16 bit album, upsampled to 24 bits (Daft Punk - Homework):

CODE

16/24 bits 01. Daftendirekt.flac
16/24 bits 02. WDPK 83.7 FM.flac
16/24 bits 03. Revolution 909.flac
16/24 bits 04. Da Funk.flac
16/24 bits 05. Phœnix.flac
16/24 bits 06. Fresh.flac
16/24 bits 07. Around the World.flac
16/24 bits 08. Rollin' & Scratchin'.flac
16/24 bits 09. Teachers.flac
16/24 bits 10. High Fidelity.flac
16/24 bits 11. Rock'n Roll.flac
16/24 bits 12. Oh Yeah.flac
16/24 bits 13. Burnin'.flac
16/24 bits 14. Indo Silver Club.flac
16/24 bits 15. Alive.flac
16/24 bits 16. Funk Ad.flac

[nu774]

BTW, I think just a slight gain scaling from the original would be enough to fill full 24bits.
For example, you can choose scale factor 0.9, which doesn't have finite binary representation.

[greynol]

You can add noise or simply use Audacity which can change all of the data with a mere save command without having to perform any editing, provided you didn't change the program's silly and annoying default behavior.

Just about anything you do to randomize the least significant 8 bits will likely be inaudible if the rest of the bits are being used and the playback volume is held at a sane level.

Also, spectral views are intended to view frequencies. They aren't going to be very useful in detecting minute changes in amplitude.

This post has been edited by greynol: Nov 6 2012, 12:55

[2Bdecided]

There's no way of knowing for sure. Unlike low bitrate mp3 encoding, converting to 16-bits leaves no tell-tale signature.

Once "up-converted" to 24-bits, it's trivial to fill the bottom 8 bits with noise (any operation, even a tiny gain-change, will fill them with some non-zero values anyway). Noise-shaped dither at the 16-bit level could be a tell-tale, but it might not have been used, and if it was, you can use a filter to drop the level to something benign looking. A similar rising noise spectrum (though only above 20k, and at a lower level) is also found on DSD-sourced recordings - so you don't need to be that careful - just dropping it to DSD-like levels will do the trick.

A recording with analogue silence won't give anything away, but a recording with 16-bit dithered digital silence could alert some people. In this case, it's easy to replace it with 24-bit digital fades and silence.

So, as I said, there's no way to know.


If you had both the 24-bit and 16-bit versions, you could show that the 16-bit version was probably generated from the 24-bit version (if it was), rather than the other way around - but you couldn't prove that the 24-bit version itself wasn't generated from a previous 16-bit version.

So, if version A=24-bits and version B=16-bits, you could show that B was created from A, rather than A being created from B. But you cannot show that A was not created from U, a 16-bit version that you don't have access to. That possibility always exists, and you cannot disprove it.

(I am assuming a real recording in a normal recording studio with real microphones, which will almost always result in a noise floor that's far above the 16-bit dithered noise floor. If you had an entirely synthetic recording, it could have an arbitrarily low measurable noise floor, far below 16-bit dither, and here you could very easily prove whether it was native 24-bits, or upconverted from 16-bits).

Cheers,
David.

[vincefalks]

Thanks guys (and yeah thanks greynol for pointing that out about the spectral view), I now have confirmation from the webstore that it was an encoding error when ripping from the physical CD disc to upload on the store.

I'd like to see if those ideas above show anything.

So Now that I know more of how the 24-bit files were created: well obviously it was somewhere at the FLAC encoder level? Looks like they accidentally had 24-bit (flac) output selected in whatever ripping program they used, and that's how it happened. probably first extracted as 16-bit wav, but then converted from that into 24-bit flac, you think? with that in mind, any more clarity of how one could detect a flac.exe-produced 24-bit blow-up job?

Also, get this: their response to me pointing out the error, was such that they weren't actually to take any action unless I requested them to. they merely said 'This does not detract from the files but means that they are slightly larger than they need to be.' They actually are twice the size they need to be.

They seemed content with me wasting hard drive space for absolutely no reason, just because they, a professional music downloads web store, made a mistake when ripping the files.

Lazy.

NOW, after telling them I either want a refund or re-ripped files from the disc, within 5 minutes they have come back and said they have new (fixed) 16-bit files for you on the store. they obviously didn't re-rip the files from the disc, they clearly just downconverted the 24-bit files to 16-bit, right??? and unless that's done with a really good dithering algorithm, that could be inferior to an original straight rip from the 16-bit, right?

what's the bet they used some dodgy program (like, I dunno, is dbpoweramp a good example of dodgy?) to downconvert it again? Should I continue to not be satisfied and point this out to them, and demand a proper true lossless re-rip with no mucking around, and also is there a way to detect whether something's been (with damage), downconverted from 24-bit to 16-bit (I guess this is impossible without comparing it to an original perfect CD rip from the original unprocessed 16-bit.....?)?

ah - started writing this before finishing reading 2Bdecided's reply.....well with this in mind (16->24->16), is there any way I can at least prove the 24->back to->16 bit, bit?

[greynol]

Don't blame flac.exe. It will not take 16-bit input and produce a 24-bit encode. The processing occurred before the compression.

dBpoweramp is not a dodgy program. While foobar2000 can be configured to produce a 24-bit flac from a 16-bit wave, it isn't a dodgy program either.

Hopefully they didn't use dither when converting the 24-bit files into a 16-bit files or that will ruin any possible chance that you arrive back at the original 16-bit data. With respect to the conversion from 16-bit to 24-bit, dither can only compromise quality. Dither is used to de-correlate error when processing data. When reducing bit depth is the only processing, dither is really only useful when going from a higher depth to a lower one, and only when the higher depth actually has meaningful data in the bits that will be discarded.

This post has been edited by greynol: Nov 6 2012, 17:40

[skamp]

They actually are twice the size they need to be.

Are you sure? I just tried upconverting a 16 bit album in FLAC to 24 bit, and the resulting FLACs are almost the same size at the originals.

As a sidenote, I scanned my entire FLAC collection with my script, and I found an oddity: the album "Smash" by The Offspring only has 15 effective bits (out of 16, it's a CD rip). I wonder why that is.

[greynol]

6dB of gain was applied to the master with no further processing?

This post has been edited by greynol: Nov 6 2012, 17:43

[vincefalks]

Don't blame flac.exe. It will not take 16-bit input and produce a 24-bit encode. The processing occurred before the compression.

Ah yeah that makes total sense. sorry about not thinking much. it's been a long time since I have been active in audio softwares and the like, really.

dBpoweramp is not a dodgy program. While foobar2000 can be configured to produce a 24-bit flac from a 16-bit wave, it isn't a dodgy program either.

well, I just mean more anything that's not sox/weiss etc. grade dithering.

A originally assumed dbpoweramp just was dodgy but then thought, well, how can i assume that. of course - i am nowhere near knowledgable, nor aware of any improvements made to other systems/softwares.

i'll be honest in acknowldging my purist tendancies here...

Hopefully they didn't use dither when converting the 24-bit files into a 16-bit files or that will ruin any possible chance that you arrive back at the original 16-bit data. With respect to the conversion from 16-bit to 24-bit, dither can only compromise quality.

Well I wouldn't bet they know what they're doing. They'd be I'd say most likely using some one-click solution and wouldn't have even heard of the word dither before. (I'm at least one step higher than that ;P.)

So should I demand the web store to re-rip from the CD (saying that this multi-step processing has rendered the file anything but lossless), or is it likely these 16->24->16 files are not too damaged?

This is what i want to try and find out through a bit of audio analysis, or make a judgement on.

[greynol]

You would really have to go out of your way to do 16 -> 24 -> 16 for it to result in audible degradation.

[vincefalks]

They actually are twice the size they need to be.

Are you sure? I just tried upconverting a 16 bit album in FLAC to 24 bit, and the resulting FLACs are almost the same size at the originals.

When I convert my originally-provided and bloated 24-bit files to 16-bit flac with foobar (and yes I've checked compression levels too, to make them equivalent, as much as I can guess), it's more than a ratio of 2:1. so yes it's definitely twice too much pointless stuff they were making me download and store on my hard drives.

You would really have to go out of your way to do 16 -> 24 -> 16 for it to result in audible degradation.

Ok I guess I'll leave it be then. Thanks.

[skamp]

it's more than a ratio of 2:1

Then they probably did something more than a straightforward 16 to 24 bit conversion (shifting values by one byte).

[Destroid]

Interesting discussion. I hope the market does not attempt to sell consumers 24/96 downloads of "high quality" that were converted from CD sources. Yuck. Reminds me of another discussion of shrink-wrapped CD's with track(s) from lossy sources.

As a sidenote, I scanned my entire FLAC collection with my script, and I found an oddity: the album "Smash" by The Offspring only has 15 effective bits (out of 16, it's a CD rip). I wonder why that is.

6dB of gain was applied to the master with no further processing?

Not to hijack the discussion (and not wanting to start a new thread) but my pressing has no such wasted bits in case you wanted to investigate further.*

*Note: The album gain of my backup image reported a RG value of -7.82. Other than that, the spine area of the back label of my pressing has the number 86432-2. Good luck!

[julf]

I used your idea to make a BASH script that prints out the average number of effective bits of FLAC files.

Many thanks! Ran it on my collection of 20,000 tracks, and only found 6 albums showing significant "unused" bits. One, a pure spoken voice one (language training record) had "effective" bits between 13 and 15 depending on the track, 2 CD's (Smoke & Strong Whiskey by Christy Moore and In My Memory by Dj Tiësto) had consistent 15 bits out of 16, and 2 supposedly 24 bit downloads (A Retrospective by The Unthanks and the LSO Haitink Beethoven Symphony no. 9, both from B&W Society of Sound) had consistent 16 effective bits.

The one strange one was Portico Quartet from B&W Society of Sound:

22/24 bits Portico Quartet/Portico Quartet/10 Trace.flac
23/24 bits Portico Quartet/Portico Quartet/01 Window Seat.flac
16/24 bits Portico Quartet/Portico Quartet/02 Ruins.flac
16/24 bits Portico Quartet/Portico Quartet/03 Spinner.flac
23/24 bits Portico Quartet/Portico Quartet/06 Laker Boo.flac
19/24 bits Portico Quartet/Portico Quartet/05 Export to Hot Climes.flac
16/24 bits Portico Quartet/Portico Quartet/09 City of Glass.flac
20/24 bits Portico Quartet/Portico Quartet/08 4096 Colours.flac
22/24 bits Portico Quartet/Portico Quartet/07 Steepless.flac

[2Bdecided]

An even smarter analysis can figure out which 24-bit (or even 16-bit) values are used, and how often. If there are some values which are unused (e.g. 0 is, 1,2,3,4,5,6 aren't, 7 is, 8,9,10,11,12,13 aren't, etc etc), you have a clear case of undithered conversion and scaling. If values (other than zero) are used disproportionately to those around them, you probably have some strange distortion or scaling. If values at or near full scale are over-used compared with those around them, you probably have clipping.

This technique still won't catch 16>24-bit conversions which added noise in the 8 LSBs, and will often miss problems if the occur before a final stage of (especially noise-shaped) dither. But it's surprising what it does catch.

It's hard to automate the detection of something wrong via this analysis - you mostly just have to eyeball the distribution. For 24-bit audio, it's only meaningful for long pieces of music, given the 16M different possible sample values. With 16-bit audio, you can get meaningful results with more than a minute or so of music.

Cheers,
David.

[Porcus]

it's more than a ratio of 2:1

That's using FLAC. What's the bitrate of your 16 bit files? If < 700 ...

[Porcus]

I hope the market does not attempt to sell consumers 24/96 downloads of "high quality" that were converted from CD sources. Yuck.

Well ... does it really make a difference? Had the CD standard been 48 kHz, one could maybe easier have exposed a few 24/96's as CD-sourced, and gotten the hi-rez hysteria the bad press it deserves.

[pdq]

Let's not forget that a file that was up-converted from CD will have an integral number of frames, i.e. an exact multiple of 1/75 seconds.

[2Bdecided]

Let's not forget that a file that was up-converted from CD will have an integral number of frames, i.e. an exact multiple of 1/75 seconds.

...if they ripped it properly! ...and didn't skip any silence automatically.

[julf]

Let's not forget that a file that was up-converted from CD will have an integral number of frames, i.e. an exact multiple of 1/75 seconds.

Good point. Turns out that out of the B&W Society of Sound supposedly 24 bit downloads I looked at, the Portico Quartet one had exact multiples of 588 samples in all of the tracks, and the album by The Unthanks had exact multiples in 6 out of 11 tracks... I guess we know where they came from...

[pdq]

...and it makes sense that if the tracks had been resampled from 44.1 kHz then the low byte of each 24 bit sample would not be zero. Add to this the fact that resampling technically reduces quality, and these files are not as good as the original CD.