Why do I have forbidden sockets?

I was tinkering with some C# code that uses TcpListeners recently, and hit on a strange issue where my code would run fine on on machine, and fail on another. It took me a while to find an answer in Google, so here’s a reminder to my future self: