Are you worried that your Japanese is starting to sound feminine or masculine? Are you a guy who watches a lot of Japanese idols (ex. AKB48), is dating a Japanese girl, or likes anime/manga with female leads? Are you a girl who watches a lot of Johnnys, is dating a Japanese guy, or likes Shonen anime/manga?
Are you afraid that because you spend all your time listening to the opposite sex you will start to sound like them? Put your fears to rest. Unless you purposefully want to sound like the opposite sex, you only need to employ one technique: Variety.
As long as you have some variety with your listening materials, you will sound the way you want to sound. Regardless of your main focus on one specific area of listening, if you have variety, your brain will subconsciously pick up the way you should talk. You will naturally follow the speaking patterns of people who are your gender and closest to your age.
There was a time when I used to watch a lot of children’s TV shows, Yakuza shows, and 時代劇 (Japanese samurai dramas). I may have occasionally inserted some phrases I learned to add some humor to my spoken Japanese, but I never subconsciously started speaking like them, no matter how much I listened to them.
Now go relax, and watch as much AKB48/Johnnys as you want.