R128Scan [obsolete], Now implemented by bundled foo_rgscan Options

[kode54]

This component has been obsoleted by the ReplayGain scanner bundled with foobar2000 version 1.1.6.

I have produced a scanner interface almost entirely from scratch, based on libebur128. Basic information and a binary are posted here, and the full source is posted here, but I warn you, it isn't pretty.

It writes -18 - LUFS to the normal ReplayGain track and album gain level tags, and optionally scans for the "true" peak by resampling the signal to 192KHz first, using the foobar2000 resampler service, but this is disabled by default because it has a pretty heavy speed cost.

No attempt is made to support reading or writing the EBU R128 "standard" of tags proposed in this topic, because that is beyond the scope of a mere scanner component. The player core must first support representing that information internally with its binary ReplayGain information structures, and then every input component that supports reading and/or writing tags must be modified to support these changes.

Change log:

• Version 1.21 - 2011-02-07 14:12 UTC:
  
  • Fixed a bug in libebur128 when handling weird sample rates.
• Version 1.20 - 2011-01-31 01:33 UTC:
  
  • Fixed album scanning results reporting.
• Version 1.19 - 2011-01-31 01:22 UTC:
  
  • Disabled apply tags button when there are no tags to apply results to.
• Version 1.18 - 2011-01-30 23:53 UTC:
  
  • Implemented full error reporting.
• Version 1.17 - 2011-01-30 06:23 UTC:
  
  • Fixed another crash issue with IO error handling in the scanner thread.
• Version 1.16 - 2011-01-30 05:28 UTC:
  
  • Fixed a potential problem with threaded album scanning where not all of the threads would result in valid scanner instances.
• Version 1.15 - 2011-01-29 02:44 UTC:
  
  • Updated to libebur128-0.1.9.
  • Changed scanner function to reconfigure libebur128 instead of resampling or failing outright.
• Version 1.14 - 2011-01-28 05:48 UTC:
  
  • Implemented proper track and album skipping.
• Version 1.13 - 2011-01-28 05:04 UTC:
  
  • Implemented multi-threaded album scanning.
• Version 1.12 - 2011-01-27 07:04 UTC:
  
  • Scanner status now only lists the files currently being scanned.
  • Scanner status window now shows the count of processed files and the total count.
  • Limited thread count by the number of jobs to execute.
• Version 1.11 - 2011-01-27 06:15 UTC:
  
  • Fixed channel mapping setup for non-stereo files.
  • Added an abort check to the scanner processing loop just in case.
  • Changed the scanner results to a modeless dialog to get rid of the Windows 7 taskbar progress indicator artifact.
  • Changed all applicable get_item() calls to [] operators.
• Version 1.10 - 2010-01-26 10:34 UTC:
  
  • Changed the "true" peak scanner to always use an upsampling factor of 4 instead of a fixed sample rate of 192KHz.
• Version 1.9 - 2011-01-26 09:32 UTC:
  
  • Fixed mixed up infinity/NaN in ebur128.cpp.
  • Fixed rg_offset to return the correct invalid gain constant when the result is invalid or out of range.
• Version 1.8 - 2011-01-26 08:47 UTC:
  
  • Implemented support for mid-file and mid-album sample rate changes.
• Version 1.7 - 2011-01-26 08:15 UTC:
  
  • Fixed albums-by-tags scanning mode to include the last album detected in the batch.
  • Disabled multi-threading in debug builds.
  • Fixed "true" peak calculation to instantiate the resampler for more than just the first track in the album.
  • Restructured "true" peak calculation resampler code to properly handle the last chunk produced by the decoder
• Version 1.6 - 2011-01-26 06:03 UTC:
  
  • Implemented thread priority control.
  • First public release.
• Version 1.5 - 2011-01-26 05:53 UTC:
  
  • Implemented finer grained progress indicating.
• Version 1.4 - 2011-01-26 05:16 UTC:
  
  • Fixed RG scanner result display dialog showing the track gain in the album peak column when scanning in track gain mode.
• Version 1.3 - 2011-01-26 05:16 UTC:
  
  • Made "true" peak scanning optional.
• Version 1.2 - 2011-01-26 04:24 UTC:
  
  • Fixed RG result info display when there are results from more than one job in the batch.
  • Fixed RG scanner to remove the current job's track names and update the display on I/O error.
• Version 1.1 - 2011-01-26 03:45 UTC:
  
  • Fixed RG info applying filter result finding function to actually return the item indices so that album set tags are written correctly.
• Version 1.0 - 2011-01-26 02:31 UTC:
  
  • Album scan mode now keeps track of the sample rate and channel information across tracks.
• Version 0.0 - 2011-01-26 02:18 UTC:
  
  • Started project.

[lvqcl]

Big thanks!

Bug report:
When I select N albums and use "Scan selection as albums (by tags)", it scans only first N-1 albums.
True peak scanning works only for the very 1st track.

Also, I think that resampling 44,1 and 88,2 to 174,6 kHz instead of 192 kHz should be a bit faster (especially for SoX resampler users )

[thuan]

Scan selection as albums (by tags) does not work for me. Please check.

EDIT: As I selected only one album, the mentioned operation does not work as also confirmed by lvqcl.

This post has been edited by thuan: Jan 26 2011, 09:01

[Canar]

The component currently has difficulty dealing with albums that contain tracks with varying sampling rate. Also, it was updated just a few minutes ago to deal with lvgcl's problem. Could you verify that all tracks in your album are of the same sampling rate, please, thuan?

[thuan]

Yes they are

[kode54]

Updated.

Also, for one thing, the specification, or at least the weirdos in that R128GAIN topic, call for resampling to 192KHz for the peak measurement. And besides which, why does it matter for SoX resampler users? Does its resampler have some sort of speed optimization for even multiple upsampling?

Currently, I just use the core resampler_entry::g_create() function, which instantiates the first resampler service it can find that accepts the given source and target sample rates. The order in which it searches for resampler DSP services is linear, but the order in which loaded components' services appear is randomized on every startup, so there's no guarantee you'll be getting either foo_dsp_std's polyphase resampler, or SoX, or any other installed resampler. Well, unless you remove all DSP components which have resamplers, except for the one you want to use.

The source and target sample rate selection was only an issue back when foobar2000 still bundled a SSRC-based resampler. Now that it bundles a polyphase resampler that supports all rates, it will be selected automatically if it is the first resampler in the global services list.

[Raiden]

Great work!

A few things that I noticed:
When adapting the return values of the library for C++, "0.0 / 0.0" should be "std::numeric_limits<double>::quiet_NaN()" and "-1.0 / 0.0" should be "-std::numeric_limits<double>::infinity()" (with a minus!).
Also, for RG tags one must limit the output values of the library when it returns -infinity (on silent tracks).

[kode54]

Fixed, thanks for the report.

[lvqcl]

Also, for one thing, the specification, or at least the weirdos in that R128GAIN topic, call for resampling to 192KHz for the peak measurement.

Here: http://www.hydrogenaudio.org/forums/index....st&p=738148

4 × over-sampling

And besides which, why does it matter for SoX resampler users? Does its resampler have some sort of speed optimization for even multiple upsampling?

My tests on Core2 2,5GHz:
True Peak = Off: WAV 44,1k scanning is 420x realtime, WAV 48k is 400x realtime;
True Peak = On, PPHS: WAV 44,1k is 29x realtime, WAV 48k is 33x realtime;
True Peak = On, SoX: WAV 44,1k is 18x realtime, WAV 48k is 41x realtime.
True Peak calculation slows scanning severely (for no apparent benefits IMHO) but nevertheless 4x upsampling is faster.

there's no guarantee you'll be getting either foo_dsp_std's polyphase resampler, or SoX, or any other installed resampler.

According to my tests, fb2k never uses built-in PPHS resampler when there's another resampler with priority > 0.

[kode54]

Fine, I changed it to 4x upsampling. And yes, that feature is disabled by default for a reason.

Also, I didn't notice the priority checking in the resampler DSP enumeration. Ah, well. I guess SoX is programmed with a higher priority.

[lvqcl]

Hi, its me again. It doesn't work correctly with mono files: calculates only peak values, not gain.

[mudlord]

True Peak = Off: WAV 44,1k scanning is 420x realtime, WAV 48k is 400x realtime;
True Peak = On, PPHS: WAV 44,1k is 29x realtime, WAV 48k is 33x realtime;
True Peak = On, SoX: WAV 44,1k is 18x realtime, WAV 48k is 41x realtime.
True Peak calculation slows scanning severely (for no apparent benefits IMHO) but nevertheless 4x upsampling is faster.

True Peak = Off: MP3 44,1k scanning is 524.17x realtime
True Peak = On: MP3 44,1k is 105.69x realtime

On AMD Phenom x6 1055T 2.8Ghz, 8GB RAM

This post has been edited by mudlord: Jan 26 2011, 18:30

[lvqcl]

Probably quadraphonic (FL, FR, BL, BR) files are affected too.

Simple patch: replace

CODE

for ( unsigned i = 0, j = 1; i < last_channels; i++, j <<= 1 )
{
    int channel = EBUR128_UNUSED;
    switch ( last_channel_config & j )
    ...

with

CODE

for (unsigned i = 0; i < last_channels; i++)
{
    int channel = EBUR128_UNUSED;
    switch ( audio_chunk::g_extract_channel_flag(last_channel_config, i) )
    ...

but this results in 3.01 dB difference between 'mono' and 'mono converted to stereo' files...

[lvqcl]

I think this means that ebur128_gated_loudness() function should be changed:

CODE

return ebur128_energy_to_loudness(gated_loudness * ( (st->channels==1 && st->channel_map[0]==EBUR128_CENTER) ? 2 : 1));

Added:
...But it's the player that maps center channel to left & right, so probably it should be done on player's side (or at least somewhere in foo_r128scan.cpp, not in ebur128.cpp) ?

This post has been edited by lvqcl: Jan 26 2011, 21:45

[Canar]

"True peak" seems designed to try catch and prevent inter-sample clipping. I'm confident that most modern DACs are designed to handle this edge case anyhow. We shouldn't need to worry about it that much.

[C.R.Helmrich]

I have produced a scanner interface almost entirely from scratch, based on libebur128. Basic information and a binary are posted here, ...

C.R.Helmrich likes this. It's great to see how quickly developers people adopt R 128. Maybe there actually is a real chance to end the loudness war.

Quick test with foobar 1.1.2 beta on two albums shows that the results are identical to those I got by hand from r128-sndfile.exe last week.

Minor beautification/question:

After scanning, the progress of which is nicely indicated in Win 7 on the taskbar icon , the icon still has a vertical green progress line, as shown below, even though scanning is finished.



During scanning, you show a semicolon-separated list of all files being processed, which is reduced one-by-one. Did you make sure that kind of handling is buffer-overflow-proof if one selects thousands of files at once? And wouldn't it be better to show just the currently processed file?

Thanks a lot for your work!

Chris

[kode54]

Updated again. I have not made any changes to the volume level adjustment for mono files. The algorithm processor itself should probably be adjusted instead. Or maybe the player's output components, or maybe the operating system itself. You know, since the DirectSound code uses WAVEFORMATEXTENSIBLE to output mono files to only the center channel, and it's up to the operating system, or the sound card's drivers, or whatever, to promote the result to the left and right speakers in a stereo setup.

And the progress indicator artifact was due to using a modal dialog from inside the threaded process callback instead of spawning a modeless dialog and returning.

While the status string should never overflow, since memory is allocated dynamically, I've changed it to only list the files currently being scanned. Of course, that will be up to four filenames at once on quad core or faster systems, since I limit the scanner to up to four threads.

[Nowings69]

Updated again.

Great work.
Id like to add just like "apply to data" as normalize.
Well.Ive using your foo_input_avs and reencode it for video file.
If this component will be stability then could you add it please

Yes I know its rectangle but those icons are 15x15(upper section is 22x22)


This post has been edited by Nowings69: Jan 27 2011, 09:17

[kode54]

foo_rgscan already has a function to apply the gain correction to MP3 files. Adding that to foo_r128scan would be redundant, not to mention I would have to code it from scratch, since the foo_rgscan code I have from 2004 did not have the mp3gain capability added yet.

[Nowings69]

foo_rgscan already has a function to apply the gain correction to MP3 files. Adding that to foo_r128scan would be redundant, not to mention I would have to code it from scratch, since the foo_rgscan code I have from 2004 did not have the mp3gain capability added yet.

Yes you are right、Of course　I understand it that only MP3(without e.g. vorbis read vorbis comment in ffdshow and mesepack ds filter)
I wish it is come true in the near future by this wonderful idea

This post has been edited by Nowings69: Jan 27 2011, 11:15

[kode54]

You're free to keep foo_rgscan installed and use its mp3gain functionality.

[Andreasvb]

Not sure if it's a bug, but:

When scanning as selection it seems to only use 1 core.
If I choose Scan per-file then it's using all my 4 cores.

Would also be good to see how fast it scans in the progress window not just in the results window, like the regular RG scanner.

Thanks.

[lvqcl]

Probably this issue is related to libebur128: http://www.hydrogenaudio.org/forums/index....st&p=740589

Select several (>=4) albums and choose "Scan as albums (by tags)". You'll see 100% CPU load.

[kode54]

Yes. Dividing a single album up onto multiple cores would require significant changes to the scanner library. I don't think foo_rgscan supports multi-threading single albums, either.

[lvqcl]

It does support.