After I eliminated dairy I felt so much better, and I'm not lacking in any nutrients either. When I cook I now use nut milks which are healthier and I find tastier, as well as coconut meat and butters for substitution. That also works the same for cheeses, nut cheeses are truly amazing, and there are some great recipes out there and it's so easy to make at home.
I don't miss it at all, I think part of it is changing the way I look at it too, take away the marketing aspect about how amazing it is and start to look at the scientific research behind the effects and that makes it easier. Added to that the trials and errors of eliminating and having it in my diet, I now simply categorise milk as one of those items that do more harm than good and what's the point of putting toxins into my body? I only have one!
So whether you or not you have an intolerance for dairy, you've got nothing to lose by eliminating it from the system, even if it's for a trial run. See if it does make a different to the way you feel!