Surely it depends entirely on who its with? Amongst other things yes, but whether or not sex is a privilege depends an awful lot on the other participant(s) and what is being done and to who.
I've *shudder* ended up with a gf at one point who I quickly found out I'd probably chew my own arm off if she fell asleep on it to avoid waking her whilst I ran like hell. And quite a few really hot ones that turned out to be complete fucking psychos. For some unknown reason, I seem to have attracted more than what could possibly be my fair share of girls who were (and doubtless still are) bona-fide fruit-fucking-well-bins.
Think I WANT to have to do a 10-15 mile plus run without stopping to catch my breath whilst making a phonecall to the bloody ER and have somebody I care about dragged against their will out of their workplace in public? who then fucking pinches some bloody speed off me, tells me they are pregnant with my kid, after a suicide attempt, shows me an ultrasound scan and then fucks off never to be seen again. With a bunch of amphetamine sulfate paste? sorry fucker. But if that kid turns out to be a right bastard, you've earned it. Made your bed (or at least, she made lying up the stairs, on a sofa, a leather couch-chair and over the lounge table and on the carpet one night) she can lie in it/on it/over/around/up it.