The logical thing is to of course immediately shut the door on the 50, because that would kill the fewest people on the average. The expected outcome of leaving it open long enough to let them all out is you kill 10% of 1000 people or 100 people versus the 50 you would kill by immediately closing the door because the 10% risk is that all 1000 of you die if you leave the door open long enough to save the 50.
That would definitely be my husband's answer, but I would find it difficult to be so coldly logical if I were the captain of a submarine actually faced with the situation of directly killing some of my crew. I would probably try some creative solution like Duncvis', even though I know logically that the expected outcome of that would probably be you immediately kill 25 people plus 5% of 975 or approximately 74 people instead of 50 or 100, assuming the risk is linear with time.
The answer that kills the fewest people on the average is you immediately close the door and trap and kill the 50 sailors.