Shutting down hate speech events in a public forum – say, at your university – is not restricting freedom of speech.
Freedom of speech grants everyone the right to express their opinion without fear of prosecution, but it does not grant people the privilege of a podium, especially not within an institutional context that legitimizes their views.
Granting, say, even fascists freedom of speech does not mean you have to grant them an institutional platform so they can recruit and organize.