Fame as a disequilibrium

In The Curtain: An Essay in Seven Parts, Milan Kundera says something very interesting about fame:

A man becomes famous when the number of people who know him is markedly greater than the number he knows. The recognition enjoyed by a great surgeon is not fame: he is admired not by a public but by his patients, by his colleagues. He lives in equilibrium. Fame is a disequilibrium. There are professions that drag it along behind them necessarily, unavoidably: politicians, supermodels, athletes, artists.

Kundera seems to imply that fame as a disequilibrium is a bad thing, that is, bad for the person who is famous. But why is it bad for a person that the number of people who know him greatly exceeds the number of people he knows?

I think in order for fame to be bad for the person who is famous, there has to be another disequilibrium: the number of people who know him is markedly greater than the number of people who have good reasons to know him. Such disequilibrium is bad for the person who is famous because what he gets is no more than others' attention. Such attention is not accompanied by respect or admiration, but may still give the person who receives the attention a false sense of accomplishment and an inflated ego.