Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Opus Audio Transparency? (Read 24322 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Opus Audio Transparency?

Before I say anything else, let me just note that I am totally new here at Hydrogenaudio and this is my first post. If a moderator feels this thread is posted in the wrong section, by all means move it. I'm honestly a bit overwhelmed, as I feel like this topic could be related to several areas, but I think I've found a good place for the time being. Also, a disclaimer: None of this post is meant to violate TOS #8. I am not attempting to claim that any one codec or bitrate is better or higher quality or more efficient in ANY way. I only mean to solicit opinions and possibly conjecture from others.
TL;DR I'm new and I don't believe I am in violation of TOS #8.

With all the buzz about Opus, the newly standardized audio codec on Hydrogenaudio, I thought I might conduct some tests of the transparency of Opus when encoding music. Specifically, I am interested in converting my existing FLAC collection to Opus for use on an MP3 player with Rockbox. Basically, I scoured the web trying to find a standard or recommended bitrate that would give me at least near-transparent audio quality. I could find none, although it appears that the Hydrogenaudio Wiki article on Transparency recommends ~144kbps for "complex audio" with Opus. I was somewhat skeptical of this number, mostly because I haven't seen it anywhere else, and I didn't think it would be possible to do much better than HE-AAC at around 150.

I figured a good way to judge transparency would be to take a generally accepted number for another format (HA Wiki says most people find ~192kbps MP3 to be transparent in most cases) and compare various Opus bitrates to that. My way of comparing is not at all scientific: just looking at spectrograms of various audio files. I do not expect to use spectrograms to prove anything one way or another, but I thought my results were interesting and worth sharing, so here they are.

NOTE: I used LAME VBR quality V2 to test against because the Wiki said it was generally considered to be transparent/near transparent.

ABBA - Take a Chance On Me | FLAC | LAME V2 | Opus 144kbps
ADELE - Rumor Has It | FLAC | LAME V2 | Opus 144kbps
DEADMAU5 FEAT. IMOGEN HEAP - Transmiscommunications | FLAC | LAME V2 | Opus 144kbps
ENYA - Orinoco Flow | FLAC | LAME V2 | Opus 144kbps
LADY GAGA - Bloody Mary | FLAC | LAME V2 | Opus 144kbps
NICKI MINAJ - Roman Holiday | FLAC | LAME V2 | Opus 144kbps
PINK MARTINI - Hey Eugene! | FLAC | LAME V2 | Opus 144kbps

I would love to hear a more experienced analysis of these results. Feel free to sound off! Do you think that Opus is transparent or *nearly transparent* at ~144kbps VBR, like the HA Wiki says? Does the speckled "noise" at the top of Opus graphs degrade the audio quality, or help hint at information that was present in the FLAC files? If an Opus file does not appear to be transparent at this bitrate, do you think it is fair to rate it higher quality than its LAME V2 counterpart?

My response: After extensive listening, I have found differences between the Opus files and their FLAC parents to be present, but difficult to distinguish. LAME performed well also, but I was able to pick out those files from the FLAC more often. I think Opus is at least *nearly transparent* at 144kbps, but I don't have very solid proof for it, other than my own ears.

Thanks so much for your comments! Again, any help is appreciated. What do you think would be the best way to measure the "rate of transparency" for an Opus file short of an organized listening test? Any other bitrates you think I should try?

~ Ethan

Opus Audio Transparency?

Reply #1
Also, a disclaimer: None of this post is meant to violate TOS #8. I am not attempting to claim that any one codec or bitrate is better or higher quality or more efficient in ANY way. I only mean to solicit opinions and possibly conjecture from others.


Conjecture and opinions aren't a TOS8 bypass. Why would they be?


Quote
TL;DR I'm new and I don't believe I am in violation of TOS #8.


Did you even read TOS8?

Quote
8. All members that put forth a statement concerning subjective sound quality, must -- to the best of their ability -- provide objective support for their claims. Acceptable means of support are double blind listening tests (ABX or ABC/HR) demonstrating that the member can discern a difference perceptually, together with a test sample to allow others to reproduce their findings. Graphs, non-blind listening tests, waveform difference comparisons, and so on, are not acceptable means of providing support.



Quote
I was somewhat skeptical of this number, mostly because I haven't seen it anywhere else, and I didn't think it would be possible to do much better than HE-AAC at around 150.


HE-AAC has no benefit above +- 80kbps, AAC encoders will turn it off above that bitrate for that reason.

Quote
My way of comparing is not at all scientific:


Then why post it?

Quote
just looking at spectrograms of various audio files.


Are you encoding images with Opus? It's not meant to do that. Are you encoding audio? Then why do you think looking at images of it will tell you anything?

Quote
My response: After extensive listening, I have found differences between the Opus files and their FLAC parents to be present, but difficult to distinguish.


How did you test this?

For what it's worth, Opus is a very new codec and is still being improved rapidly, so people haven't done that much actual listening tests with it yet, and those that were done will tend to get outdated quite quickly.

Opus Audio Transparency?

Reply #2
The claims of transparency at a given bitrate are outdated (if not meaningless at all)
http://www.hydrogenaudio.org/forums/index....st&p=815365

I think there should be a minimal revision for HA's wiki articles.
The freedom of an expression is undiscussable but such misleading corrections to HA's wiki should be corrected at time.

Opus Audio Transparency?

Reply #3
Also, a disclaimer: None of this post is meant to violate TOS #8.


Ironically, one of the main purposes of TOS #8 is to save you time by discouraging this kind of thing. 

Opus Audio Transparency?

Reply #4
I feel like people have generally misunderstood the purpose of my post. I'm not trying to show superiority of Opus to MP3 or anything like that. My purpose was to explore the rate at which "complex audio" can be considered transparent when encoded with a particular codec. The spectrograms are relevant because they help show the loss of audio data in particular frequencies. The FLAC files held much higher concentrations of data at higher frequencies than either of the two alternatives. The images are simply a means of illustration of this. After all, transparency is the ideal result of lossy data compression, and the particular frequencies of data an encoder chooses to throw away can make a great deal of difference.

The spectrograms illustrate the differences between data retention between two different encoders and different bitrates. I'm not trying to imply that one is better than the other, just show that there is a difference and discuss whether or not one sample might be similar in comparison to another.

I do not think my original question to be unreasonable. I'm just exploring transparency in a codec for which there exists little, if any, evidence regarding bitrates and transparency. I don't believe that any of my tests or samples or images constitute anything scientific or solid, but I do think they are interesting.

Opus Audio Transparency?

Reply #5
First of all, you should have taken a bit more time to delve into the previous answers given.

Second of all, spectrograms are nearly useless given the way that the so-called frequency concentrations being represented by colors of varying degrees (such as linear scale vs. log scale) and are still not representative of what is audible.

Just my 0.02.
"Something bothering you, Mister Spock?"

Opus Audio Transparency?

Reply #6
A spectrogram comparison tells us virtually NOTHING about transparency.

Opus Audio Transparency?

Reply #7
You have to learn more about how perceptual coders work.
They are based on human audio perception (psychoacoustics), and can do anything to the waveform to spare bits so long as the difference is not audible, and give more bits to more important, sensible parts to human ear.
Therefore, you cannot simply and naively compare the quality of perceptual coders by looking at waveform or spectrogram.

Opus Audio Transparency?

Reply #8
...After all, transparency is the ideal result of lossy data compression, and the particular frequencies of data an encoder chooses to throw away can make a great deal of difference...

Sorry you went to the trouble to make graphs but they are not relevant. Your test methodology is based on invalid assumptions. Lossy encoders are not trying to preserve frequencies unless they can be heard.

Throwing away frequencies that are not audible (because masked by other frequencies occurring at the same time) is mostly how lossy encoders are able to compress the data. You have to use your ears to determine if the data has been audibly changed. This is why TOS#8 specifically rules out graphs.

Opus Audio Transparency?

Reply #9
...After all, transparency is the ideal result of lossy data compression, and the particular frequencies of data an encoder chooses to throw away can make a great deal of difference...

Throwing away frequencies that are not audible (because masked by other frequencies occurring at the same time) is mostly how lossy encoders are able to compress the data. You have to use your ears to determine if the data has been audibly changed. This is why TOS#8 specifically rules out graphs.
And "throwing away frequencies" doesn't do the real complexity of designing a good lossy codec justice. It's far more than just that.
It's only audiophile if it's inconvenient.

Opus Audio Transparency?

Reply #10
There is a reason why Garf answered in the way that he did: the OP was clearly not on the right path.  That yet another round of edification was necessary, clearly shows how the attitude shown in this binned post really only serves to promote ignorance.

Opus Audio Transparency?

Reply #11
Clearly I have some reading to do! Thank you all for helping me straighten this out. I have decided that an ABX test would be more useful than comparing spectrograms. I have completed a few short tests. The Opus encoder I used is 1.0.1-rc3 (I have no idea if this is the most recent, though I installed it a few days ago, so it shouldn't be that old). Again, I am testing raw FLAC to 144kbps Opus.

I am no audiophile, so keep in mind that I may not be representing things correctly.

My ABX results log: (link)
Audio files used for ABX test: (link) ~5.74MB

I'm only one person, and I haven't tested very extensively, but I think that this may be of some help if anybody wants to pick up where I left off and do some more testing.

~ Ethan

Opus Audio Transparency?

Reply #12
My ABX results log: (link)
Audio files used for ABX test: (link) ~5.74MB

I'm only one person, and I haven't tested very extensively, but I think that this may be of some help if anybody wants to pick up where I left off and do some more testing.

~ Ethan


You are guessing the right answer half the time (within 1), which indicates that, to the accuracy of that length test, the Opus encoded sample is indistinguishable to you.

Opus Audio Transparency?

Reply #13
yep.. looks like opus at 144kbps is transparent to you.

Now, ratchet down the bit rate until you start hitting 95%+ accuracy in telling them apart.. That is the bit rate at which opus loses transparency to you and you can then reliably encode at just a bit higher than that and not worry about transparency anymore. You'll probably be shocked how low that bit rate will be.

Opus Audio Transparency?

Reply #14
My ABX results log: (link)
Audio files used for ABX test: (link) ~5.74MB

I'm only one person, and I haven't tested very extensively, but I think that this may be of some help if anybody wants to pick up where I left off and do some more testing.

~ Ethan


You are guessing the right answer half the time (within 1), which indicates that, to the accuracy of that length test, the Opus encoded sample is indistinguishable to you.


That's what I gathered. Of course, I don't have wonderful ears, and I'm not trained to hear audio differences, but I had an extremely difficult time distinguishing between the samples.

yep.. looks like opus at 144kbps is transparent to you.

Now, ratchet down the bit rate until you start hitting 95%+ accuracy in telling them apart.. That is the bit rate at which opus loses transparency to you and you can then reliably encode at just a bit higher than that and not worry about transparency anymore. You'll probably be shocked how low that bit rate will be.


I am definitely planning to try this. I should start testing on 96kbps later today.
Thanks for everybody's feedback. I'm still new to this ABX testing, and I appreciate your insights.

~Ethan

Opus Audio Transparency?

Reply #15
got any results:)?

Opus Audio Transparency?

Reply #16
I am definitely planning to try this. I should start testing on 96kbps later today.
Thanks for everybody's feedback. I'm still new to this ABX testing, and I appreciate your insights.

~Ethan


Did you ever get around to that ABX testing?

Opus Audio Transparency?

Reply #17
Perhaps he could try testing the new alphas.

Opus Audio Transparency?

Reply #18
Perhaps he could try testing the new alphas.

I'm happy to help the retests along by providing some windows binaries of the latest build from git. Is it considered ok to attach them to a post? I can't see any TOS which restrict it.

Please note that accepting random executables from strangers over the internet is usually not a good idea though - strongly recommend building it yourself if you can.

Opus Audio Transparency?

Reply #19
providing some windows binaries of the latest build from git.

make absolutely sure that your binaries are versioned accordingly.  meaning don't get the latest opus.git repository and have it report itself as libopus 1.1 alpha.  this has been raised as a very frustrating (for the developers) issue in the past.  same probably applies to opus-tools.

Opus Audio Transparency?

Reply #20
With all the buzz about Opus, the newly standardized audio codec on Hydrogenaudio, […]
Since it’s been bumped to my attention, and to dispel any possible confusion in the future: Opus was standardised by the IETF as a multi-purpose codec for internet-based uses. It is not a “standardized audio codec on Hydrogenaudio” in the sense of being recommended by the site for general musical usage and as superior to other lossy codecs; for, although the staff and most users support the initiative and view Opus as a high-quality codec with lots of potential, it is not yet as widely supported and tested as the main implementations of MP3, AAC, Vorbis, etc. In other words, please do try it, test it, and help to improve it; but it’s not the time to re-rip everything and discard your old copies.