Make yourself at home:
vorbis-dev mails '07 Januaryvorbis-dev mails '07 Februaryvorbis-dev mails '07 MarchCheck specificially the threads with "multichannel" and "5.1" in the subject.
This reflects
my view of the multichannel situation and possible solutions for Vorbis I
I guess the first things you need to do is get familar with the Vorbis I spec and the libVorbis source code.

I noticed that the 2nd paragraph of the
5.1 bounty wiki entry ist just baloney since an ambisonic channel mapping hasn't been defined in the Vorbis I spec nor a mechanism to tell the decoder how to upmix it to 5.1. I guess some Ambisonics zealot added this without knowing much about Vorbis I.
Until now I only addressed the "coupling" (in terms of redundancy removal). You should also think about how and when "point stereo" can be used when coupling L with LS, R with RS, L with R or LS with RS. This requires some listening tests. To further improve things you should include inter channel masking. I'm not sure whether inter channel masking is already done anywhere in the source or not. But it makes sense in a 5.1 environment for closely spaced speakers (like L, C, R).
So, to recap:
- get familar with the source and the spec on coupling/psy model related matters
- do some experiments about sound localization in 5.1 and how pairwise intensity coding can (if) be applied without messing too much with the spatial image
- introduce pairwise coupling for 5.1, possibly adaptively like I described in the mail
- introduce inter channel masking
- introduce new floor / residue modes for the .1 channel
Good luck!
SG