Stresstesting - EndSend Exception

[enzi]

Hello!

It's the following setup:
- client and server on the same machine.
- client focused that it has 100% performance (best with vsync off)
- server in idle mode

What happens is that the server is stuttering but the client still sends around 1kb/sec.

As a result I'm getting random disconnects that I could pin down to an EndSend exception:

1. NullReferenceException: Object reference not set to an instance of an object
2. TNet.GameServer.EndSend (Boolean reliable) (at Assets/Scripts/TNet/Server/TNGameServer.cs:718)
3. UserCommandProcessor.FixedUpdate () (at Assets/Scripts/ArenaGank/UserCommandProcessor.cs:82)
4.  

I added some stuff so if your lines don't match:
718: mBuffer.EndTcpPacket(); // in public void EndSend(bool reliable)

and after a little while:

1. 30.915 Process player connections nextObject reference not set to an instance of an object   at TNet.GameServer.EndSend (Boolean reliable, TNet.TcpPlayer player)
2. [0x0005d] in C:\Users\Enzi\Documents\Unity3d\AG_Server\Assets\Scripts\TNet\Server\TNGameServer.cs:681
3.   at TNet.GameServer.ProcessPlayerPacket (TNet.Buffer buffer, TNet.TcpPlayer player, Boolean reliable)
4. [0x00199] in C:\Users\Enzi\Documents\Unity3d\AG_Server\Assets\Scripts\TNet\Server\TNGameServer.cs:923
5.   at TNet.GameServer.ThreadFunction ()
6. [0x00125] in C:\Users\Enzi\Documents\Unity3d\AG_Server\Assets\Scripts\TNet\Server\TNGameServer.cs:283
7. UnityEngine.Debug:Log(Object)
8. StartServer:Update() (at Assets/Scripts/AG/StartServer.cs:45)
9.  

where:
681:mBuffer.Recycle(); // in public void EndSend(bool reliable, TcpPlayer player)
923: EndSend(true, player); // in case Packet.RequestPing:
283: if (ProcessPlayerPacket(buffer, player, true))

Are you familiar with this problem? Does the buffer run out or something? Threading problem? *shiver*

As a workaround I'm setting the priority for the unity server to high. It eliminates the problem but the server will still crash under heavy load. (I guess)

Thanks in advance!

[ArenMook]

Judging by your callstack, EndSend is being called from FixedUpdate? Wuh?

I suggest looking closer at your own modifications, and keep in mind that there is a limit on the send buffer (hardware side), and if you exceed it, further attempts will fail. If you get a null ref on "mBuffer", then "mBuffer" is null, meaning you've likely called BeginSend twice in a row like so:

BeginSend()
BeginSend()
EndSend()
EndSend() <-- crash

As I said, check your code.

[enzi]

Hm, doesn't happen. Are you ruling out a threading problem?

To clarify, every networkPlayer on my server has a UserCommandProcessor script that checks if there was a new userCommand recieved, if yes process them. After this simulation tick I send position/rotation/animation states, etc.. to the client. This happens every 0.015ms. Maybe it's too much. I use some timings that Valve uses as a reference.

[ArenMook]

Right, but unless you thread-safe your own send logic, it won't be thread safe. TNet is thread safe, but Begin/Send stuff is not. In my case I always use it from the main thread only (as Unity's updates run on a single thread). You can easily check for this by adding an error message to BeginSend that checks if mBuffer is already not null.

[enzi]

Ok, 2 good news and 1 bad news. You're right in your assumption that the buffer is not null when I call BeginSend.
The bad news is, it's thrown from a ResponsePing packet and not my packets.

The 2nd good one is that I can fix the problem when I comment out

1. //BeginSend(Packet.RequestPing);
2. //EndSend();
3.  

in TNGameClient.cs

Any ideas?

edit: I'll just ignore this behavior and rewrite the current individual sending to a more centered approach. Just wanted to let you know about this behavior. A system can never be solid enough. I ran into threading problems myself last week and that was NOT fun, especially in a network environment. Testing and debugging is soooo time consuming.

edit2: Sending is rewritten and I can't reproduce the problem anymore. Thanks!

[enzi]

Damn, the problem still persists.

Ping is still sending in between my sends even though it runs from a single gameobject with a PacketSender script that sends at a given interval.

Back to Square1 I guess. ^^

[ArenMook]

As I said, you need to make sure that everything you send happens from the same thread -- meaning from inside an Update() function in one of Unity's scripts -- which are by nature single-threaded.

[enzi]

My unity server setup is the root cause. There's the unity thread running and parallel the server thread.

I know it sounds weird, initially I thought it would be a good idea.

edit: I got it under control with a SendLock object and Monitor.Enter(sendLock); in BeginSend and Monitor.Exit(sendLock); at the end of every EndSend. Not perfect but it's thread safe now.