If I was wrecking peoples lives, and upsetting many more like HS does, I don't think I would feel safe to travel anywhere without at least one full time bodyguard.
Not to do so, would be very naive, exceptionally brave - or just simply foolish!
I know that people with a varying degree of particular disorders (including types of autism and savant syndrome), can find it impossible to empathize with others - so it's just possible that HS is genuinely unaware of the hurt he's causing so many people - and that he's taking a big gamble with his own personal safety.
If I read in tomorrows news papers that HS had been murdered in a frenzied attack - my first thoughts would be something along the line.. 'well I'm not that surprised. I wonder who it was - a distraught manager or other member (or former member) of staff? Or maybe a tenant of one of his many properties? Or who knows - a family member, trying to save the empire, before it's too late?'.
OK, maybe I watch too many Midsomer Murders