If you truly have feedback, unplug your external speakers and remove any other sources of EM interference near your PC (speakers, stereo, iPod, cell phone, cordless phone, TV, radio, etc).
If you still have "feedback", it's probably not real feedback. It's a problem with the headphone and/or PC itself. If you don't have any feedback, then you have real "feedback" and it's due to EM interference.
I'm going to guess you don't have real feedback, and you're having a problem with the headset or your PC. The easiest thing to do would be to get another mic. Borrow one from a friend or just get a cheap one. Even if the quality of sound is bad, see if the feedback is still present.
If it is, then it's a problem with your PC not with the headset and mic. In that case, you should probably check your drivers and update them (I'm guessing Realtek, but could be anything).
On the other hand, if you try a different mic and things sound better, it's a busted headset/mic. Plantronics is pretty good, and you can approach them for repair if you can't return at Circuit City.