You'd certainly get that impression if you read the chat rooms inhabited by refs, who take a singular delight in reminding young trainee refs about how firm and severe they must be with players.
The referees on these sites, usually describing a match they have reffed in, take delight in pointing out how they got one over the player or got the last word by sending him off.
Hopefully they are in a very small minority.
I did watch a match recently in which it was clear the very articulate and loud ref had little but contempt for the players, booking willy nilly and generally talking down his nose to players. One player subbed himself because he was so frustrated and the other was sent off for two yellow cards for dissent.
Both knew I was a ref and sought my advice. I told them not to say another word to the referee, to get changed and accept the one match ban for two yellows. I was a little surprised the ref made it to his car in one piece.
If this is what happens to experienced referees, perhaps it's time to recognise a good time to give it up and take up another sport when bitterness takes over from enjoyment.