My brother is a manly guy, and he taught himself to knit several years ago. He makes the most awesome sweaters. Be warned though, nice yarn is not cheap yarn...
I would say, if it's something you want to learn, don't worry about being "manly" or not - just do it and ignore the scoffers.
Like you, im wanting to get into cross-stitching in the near future, mainly to keep myself busy & outta trouble. Im gonna get my next door neighbor to teach me (she did some place mats during x-mas 1 time I think). Let me just tell ya 1 thing; there will be some ladies out there that will start liking you more for that skill (depending upon how well you knit things together).
Knitting is a positive thing. I know that there was a lady that made a knitting program in some prison and it created a good and nurturing environment for the inmates.