It sounds like you've resigned yourself to not participating in events with all-talk.
That's fine, but unfortunate. I still suggest/recommend pulling the audio card out of the PC and running just on-board audio for problem isolation. If you don't have overflow issues with the card gone (and driver disabled) then you know the problem isn't with Vista or your motherboard, right?
On the other hand, if you still have problems, then you can feel better about the card and begin to suspect some type of Vista and/or other driver issue.
I assume (danger!) this (all talk) has never worked for you. If true, then the next thing to troubleshoot, after pulling your sound card out and checking/using on-board audio, would be to ensure audio has been disabled on your motherboard.
I'd also be curious, to also doublecheck, that only one audio source exists within Device Manager. You'd only want the Creative X-Fi to be listed and not any other audio drivers/devices.
Minimally, I'd check Device manager and your BIOS. Easy things to check and ensure. I'd still recommend pulling the card, but I realize that's more of a pain to do. Good luck!