I have to agree with Volcom, get both. I'm an avid GH player, though I don't play as much as I did when the first one came out. At one point, I could play "More Than a Feeling" on Expert without looking at the screen. As far as playing the "guitar", GH wins easily. Rock Band may have a couple of songs that GH3 doesn't, but GH3 has songs that are more interesting for the guitar, a better guitar controller, and some crazy songs near the end.
Having said that, Rock Band is the better game. It may not beat GH3 when it comes to the guitar, but that's it. It has both drums and vocals, giving you three instruments you can play when going solo. It has some great songs, also, some of which are better for drums or vocals, so yeah, they're not as guitar intensive as GH3 songs. And once you do get the band going, with a full set of four people, it is awesome. My friends and I used the character creator to make ourselves up, so we get to see ourselves on all the album covers and playing at the shows. It's a blast.
As a party game, Rock Band wins. As a drum game or karoake game, Rock Band wins. As a guitar game, Guitar Hero 3 wins. Overall, the edge goes to Rock Band.