Author Topic: Stresstesting - EndSend Exception  (Read 3129 times)

enzi

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 21
    • View Profile
Stresstesting - EndSend Exception
« on: February 02, 2013, 08:51:43 PM »
Hello!

It's the following setup:
- client and server on the same machine.
- client focused that it has 100% performance (best with vsync off)
- server in idle mode

What happens is that the server is stuttering but the client still sends around 1kb/sec.

As a result I'm getting random disconnects that I could pin down to an EndSend exception:
  1. NullReferenceException: Object reference not set to an instance of an object
  2. TNet.GameServer.EndSend (Boolean reliable) (at Assets/Scripts/TNet/Server/TNGameServer.cs:718)
  3. UserCommandProcessor.FixedUpdate () (at Assets/Scripts/ArenaGank/UserCommandProcessor.cs:82)
  4.  
I added some stuff so if your lines don't match:
718: mBuffer.EndTcpPacket(); // in public void EndSend(bool reliable)

and after a little while:
  1. 30.915 Process player connections nextObject reference not set to an instance of an object   at TNet.GameServer.EndSend (Boolean reliable, TNet.TcpPlayer player)
  2. [0x0005d] in C:\Users\Enzi\Documents\Unity3d\AG_Server\Assets\Scripts\TNet\Server\TNGameServer.cs:681
  3.   at TNet.GameServer.ProcessPlayerPacket (TNet.Buffer buffer, TNet.TcpPlayer player, Boolean reliable)
  4. [0x00199] in C:\Users\Enzi\Documents\Unity3d\AG_Server\Assets\Scripts\TNet\Server\TNGameServer.cs:923
  5.   at TNet.GameServer.ThreadFunction ()
  6. [0x00125] in C:\Users\Enzi\Documents\Unity3d\AG_Server\Assets\Scripts\TNet\Server\TNGameServer.cs:283
  7. UnityEngine.Debug:Log(Object)
  8. StartServer:Update() (at Assets/Scripts/AG/StartServer.cs:45)
  9.  
where:
681:mBuffer.Recycle(); // in public void EndSend(bool reliable, TcpPlayer player)
923: EndSend(true, player); // in case Packet.RequestPing:
283: if (ProcessPlayerPacket(buffer, player, true))

Are you familiar with this problem? Does the buffer run out or something? Threading problem? *shiver*

As a workaround I'm setting the priority for the unity server to high. It eliminates the problem but the server will still crash under heavy load. (I guess)

Thanks in advance!
« Last Edit: February 02, 2013, 08:53:25 PM by enzi »

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,152
  • Toronto, Canada
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #1 on: February 02, 2013, 09:24:13 PM »
Judging by your callstack, EndSend is being called from FixedUpdate? Wuh?

I suggest looking closer at your own modifications, and keep in mind that there is a limit on the send buffer (hardware side), and if you exceed it, further attempts will fail. If you get a null ref on "mBuffer", then "mBuffer" is null, meaning you've likely called BeginSend twice in a row like so:

BeginSend()
BeginSend()
EndSend()
EndSend() <-- crash

As I said, check your code.

enzi

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 21
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #2 on: February 02, 2013, 09:58:27 PM »
Hm, doesn't happen. Are you ruling out a threading problem?

To clarify, every networkPlayer on my server has a UserCommandProcessor script that checks if there was a new userCommand recieved, if yes process them. After this simulation tick I send position/rotation/animation states, etc.. to the client. This happens every 0.015ms. Maybe it's too much. I use some timings that Valve uses as a reference.

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,152
  • Toronto, Canada
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #3 on: February 03, 2013, 08:25:29 AM »
Right, but unless you thread-safe your own send logic, it won't be thread safe. TNet is thread safe, but Begin/Send stuff is not. In my case I always use it from the main thread only (as Unity's updates run on a single thread). You can easily check for this by adding an error message to BeginSend that checks if mBuffer is already not null.

enzi

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 21
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #4 on: February 03, 2013, 09:40:40 AM »
Ok, 2 good news and 1 bad news. You're right in your assumption that the buffer is not null when I call BeginSend.
The bad news is, it's thrown from a ResponsePing packet and not my packets.

The 2nd good one is that I can fix the problem when I comment out

  1. //BeginSend(Packet.RequestPing);
  2. //EndSend();
  3.  

in TNGameClient.cs

Any ideas? :)

edit: I'll just ignore this behavior and rewrite the current individual sending to a more centered approach. Just wanted to let you know about this behavior. A system can never be solid enough. ;) I ran into threading problems myself last week and that was NOT fun, especially in a network environment. Testing and debugging is soooo time consuming.

edit2: Sending is rewritten and I can't reproduce the problem anymore. Thanks!
« Last Edit: February 03, 2013, 11:24:01 AM by enzi »

enzi

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 21
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #5 on: February 03, 2013, 07:08:10 PM »
Damn, the problem still persists. :(

Ping is still sending in between my sends even though it runs from a single gameobject with a PacketSender script that sends at a given interval.

Back to Square1 I guess. ^^

ArenMook

  • Administrator
  • Hero Member
  • *****
  • Thank You
  • -Given: 337
  • -Receive: 1171
  • Posts: 22,152
  • Toronto, Canada
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #6 on: February 04, 2013, 09:03:14 AM »
As I said, you need to make sure that everything you send happens from the same thread -- meaning from inside an Update() function in one of Unity's scripts -- which are by nature single-threaded.

enzi

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 21
    • View Profile
Re: Stresstesting - EndSend Exception
« Reply #7 on: February 04, 2013, 01:27:33 PM »
My unity server setup is the root cause. There's the unity thread running and parallel the server thread.

I know it sounds weird, initially I thought it would be a good idea. :D

edit: I got it under control with a SendLock object and Monitor.Enter(sendLock); in BeginSend and Monitor.Exit(sendLock); at the end of every EndSend. Not perfect but it's thread safe now.
« Last Edit: February 04, 2013, 04:28:35 PM by enzi »