I know it's obvious that it's nonsense to judge quality by graphical method. Anyway I just wanted to know what is the graphical of difference between original one and compressed one.

After I substracted the file I got a quite strange graph. There seemed to be more frequency loss above 1300Hz. The strangeness is that the changes is suddenly and not smooth. Is that the way human hearing? (masking effects)

I have tried in 1.14 and 1.15f and both of them show the same result even in the highest quality possible.

Our ears are progressively more insensitive at higher frequencies so the difference you see on the graph represents tones we can't hear (assuming a perfectly employed psymodel). An audio CD stores musical information at full 16-bit resolution from 20Hz-22.050KHz. No one could possibly hear a -90dB 22KHz signal, but this will be stored on the CD anyhow. Musepack and every other lossy codec (using normal settings) discard these kinds of signals.

And why 13000Hz? Because that's around the point where the ATH curve becomes very steep.

Correction: 1300 Hz not 13000 Hz

That's even more strange. Our ears become more sensitive at 1.3 kHz and above and the peek is at around 3.5kHz. If our ears sensitive to those frequencies, why there should be more differences between the original and the encoded one at those freqs?

I think if human are sensitive to some frequency, it should be sensitive too to here the difference in that frequecies.

I'm going to take a guess, but a problem that every psychoacoutics encoder has is that it is harder to get accurate information about bass sounds into the psymodel (limited window size). Because of this, they are usually more conservative when encoding bass sounds than what is really needed.