aaf Voices don't match gender during animations

[PlayingOneHanded]

I'm having the problem that the voices during animations don't necessarily match the actors gender. Actors in the "top" position always have male voices while the bottom actor always uses female voices. 

Is this problem related to the animation packs or is this a problem with AAF itself and can I fix it somehow?