But, say you reply to Alice (on a different server) and Bob (on a third server) follows Alice but not you. Then, if Bob clicked on Alice's toot and expanded it to see all replies, Bob's server would request the replies from Alice's server, and get your toot—even if Bob's server wasn't federated with #fosstodon.
There are probably a few other cases like that (hashtag searches, maybe? Not actually sure how that works)