Well, I dont know anything about the xbox adapter you are using since I dont own an xbox.
What I can tell you is that even the slower Wireless B connection runs at 12Mbps (which is 3 times faster than most people's cable internet). With Fios, there are many levels and I dont know which level you are running at, however, 12Mbps is still plenty fast, and the only thing you might notice is a bandwidth crunch when you donwload all your pron at 5 in the morning.
I am wondering exactly how you are using an xbox adapter to somehow get wireless to your computer, perhaps the slow down has to do with the fact that the drivers were meant for the xbox or somehting? I guess what I am saying is that it shoudlnt be the wireless signal itself slowing you down, but the adapter (or less likely the router).