I couldn't give up dairy totally.
Totally addicted to stilton (the blue kind, not really fond of white stilton much, I like mine blue and stinky.)
Another thing to try is lactase, lactaid is one brand I know of in the US, its simply a packaged form of the otherwise missing enzyme required to digest lactose.