I definitely noticed a difference when I eliminated wheat from my diet, and it was a process, coming off something that I had eaten since I was a child. When I was a child I remember making our own breads and all the ingredients were fresh and as close to the original form as possible, now everything on the shelves are heavily processed and changed so much that it's hard to figure out what actually is in it. I think our society sometimes tries to be 'too smart' and end up forgetting that getting back to basics and simplifying things are actually the best way to go about things.
Just because you don't suffer from allergies doesn't mean what you put into the body still has no effect, it might be to a lesser degree but there's no harm in testing it out. The great thing now is that there are so many wheat free products, and raw products that we do have other options. Give yourself a week, eliminate wheat and see if it does make a change to your overall health!