As a site administrator, you certainly can approach anyone and remove them from the premises (coach, refs, parents, teachers, media). The administrator is in charge of the facility. Just like your own home. So even though an administrator cannot kick a person out of the game, they can remove them from the premises which, in essence, is removing them from the game.
I have never seen it done to that extreme, but it can happen.
Generally, administrators let refs ref, coaches coach, and they supervise. But I could see an instance where a coach was abusive to an athlete and an administrator would remove them.