I utilised an old mod called fight or flight which would actually completely stun them when they got lower than a certain health threshold. I think it imposed some sort of surrender script on them.
The SexiS mod has a module for getting raped, called SexiS defeated.
http://skyrim.nexusmods.com/mods/15223/? - Fight or Fly, should fix your problem. I was using it and it worked fine, just that they had the same problem of being hostile again if the battle went on, or after a set time period. They would also totally regain their health like F***ing zombies. That bothered me. Maybe using the rape mod, You can implement some sort of kill or submit script? i.e; the rape would either do damage or turn them to a different faction? Like, why not rape people into becoming stormcloaks or imperials, then they would just like, go and live in peace or something after getting raped into enlisting.
I understand the problem with turning them to the PC's faction, so that's just my minor workaround, if it can be implemented at all.
http://www.loverslab.com/topic/15205-sexis-sex-in-skyrim-alpha-updated-apr-1/ - The Sexis defeated model works like a charm. It does the same thing as fight or fly, but to the player. You get stunned for a while, then a strip option activates, then you get raped. It even works with lovers hook and obviously probably can be configured to using Sexlab's anim list. However, you would usually die after the rape, because you would have no health. Like, the health parameter would be at 0.15, but health would be allowed to drop to zero and you would instantly die, there wasn't any sort of safety net stop. Also, turning enemies and followers neutral so as to not continue the fight, or maybe not, then the follower would continue fighting and rape/get raped during the time...seems more realistic?
One other thing, followers need this script too, since I just mentioned it. If my follower beats the **** out of someone and then MY character rapes her...not fair, ya know?
Dunno, just had some suggestions for you, take them as you will.
Edit- Ahh, also, does this mod work for custom races? I haven't been able to do it, my assumptions being I'm using the Temptress race and/or I'm using archery on them and the script won't apply?