Agreed, when talking to an Imperial you might be using imperial units, but in your head you just WILL keep doing the math, even though an inch might be just as easily to visualize and think of as a cm, your brain gets hardwired that way in early life.
Same for us that learned the Imperial system (though I think the US standards vary from the English system slightly). When I'm given a metric measurement my brain does a rough calculation to figure out what it really means.