Ball caps are a item of formal attire where I live. I do remove mine when dining in but even in fine dining spots you will see some guy or gal wearing theirs. My dad would have beat my butt if I had ever came to the kitchen table with mine on but then that was in Iowa and 65 years ago. I do have two Polaris ball caps that match my ride but don't like how they fit.
I totally get it, here in the desert ball caps are very common attire as well - - heck, I work in a financial office and my normal attire is Levi shorts and one of my grunt style T shirts - - - if I liked them a ball cap would fit right in