I have searched on google dozens of times trying to find out how to do this correctly. I have found a vast amount of information, however I have big problems with the channel mapping. I can use MPlayer to pipe the raw audio to nero aac, but the channels are always wrong. What I assume should be heard in both speakers (because that is what it is like when I play the raw ac3 file in VLC) is only heard in one. I have found I think at least 3 different channel remapping schemes (like -af channels=6:6:0:4:1:0:2:1:3:2:4:3:5:5) but they don't seem to correct this either.
I've also been trying different channel mappings with faac. I actually came up with one that seems to work (-I 3,6) even if I dont think it was on any site. The problem is that it is faac when I think nero aac is shown to be better in any listening tests that I have seen (but I think they were old, there might be new improvements that I am unaware of), and that while it seems to play fine in VLC, in mplayer the speaking is just from one of my two speakers. This might be fine but I think it would be better to find a way that works in most players, if it is possible. I dont think it would be good to be constrained to just one media player when there are different times when I would want to use each one.
I have also tried using foobar2000 with wine to encode it (foo_ac3.dll and neroaacenc.exe). I was expecting it to work great. It did work in mplayer, but not VLC. I suspect some of the first encodings that didn't work in VLC will actually work in mplayer, but I haven't tried them. I have also tried unsuccessfully to get BeLight or BeSweet working on linux with wine but I haven't had any luck.
Whenever I have said in this post that something seemed to work in a player, it isn't for sure because I don't have a vast knowledge of 5.1 (what comes out of what channel) and as I said I don't have 5.1 sound system. I can mostly just go by if the voices are in both of my two speakers when played or just one and how it compares to the original ac3 when being played.
Does anyone know of the holy-grail of 5.1 AAC encoding on Linux? Or does anyone have any information on why some encodings seem to work in VLC but not mplayer or vice-versa? It would be greatly appreciated. If you need any more information on how I did some encodes then don't hesitate to ask