If you're sending too many packets, you should see warnings in your console log such as "[TNet] Packets sent in the last second -- sent XX, received: XX".
I haven't ran into anything similar personally, even when I had over 300 clients connected when Windward first launched. Most recently I've been running a Sightseer server where I was unintentionally sending way too many packets per client and again no issues aside from the warnings in the console log. As cmifwdll suggested, I'd try running a profiler on it -- run your server from within Unity, should be easy to find the problem.