TNet/RFC Debugging

[gg67]

I'm running into some hard to reproduce networking bugs, mostly things like people getting booted from the channel (not timing out according to the server) that don't give me any debug output. My hunch is has things to do with people joining rooms, waiting for the room to load (we have some environments being downloaded from the web), RFC calls be fired and received, but then they leave and try again b/c the screen is frozen, etc.

What's the best way to watch all the RFC calls going around? Idk if anybody has experience debugging these issues that don't spit out errors and are hard to reproduce. Any advice would be appreciated!!

[ArenMook]

If it takes a while to load the level, then the player may get booted from the server due to inactivity. The server boots players after 10 seconds of no messages. You can change this by having players call TNManager.SetTimeout prior to loading the level.

[gg67]

I've had our timeout at 60 seconds since our game is okay with inactivity. And if the player were to be booted bc of that, would the server log that as a "timeout"? Instead its just a disconnect. And it's one players are inside the room and everything is loaded.

Players join the channel, load the room, everything is fine. Then poof! They are disconnected from server.

[ArenMook]

Yeah that sounds unusual. I would suggest adding some extra debugging to the server side where it sends the Disconnect message. TNet never boots anyone without a cause, there is always a packet that gets sent first. It can happen if an error occurs.

[gg67]

So like when the server receives the Disconnect packet, I print out what debug data? I haven't spend a ton of time playing and tweaking the server side of TNet so I'm not totally sure how to go about doing this.

Thanks!

[gg67]

I'm seeing a lot of this error. I'm about to pull your new changes from github.

EndOfStreamException: Failed to read past end of stream.
  at System.IO.BinaryReader.ReadByte () [0x00000] in <filename unknown>:0
  at TNManager.CreateGameObject (UnityEngine.GameObject prefab, System.IO.BinaryReader reader) [0x00000] in <filename unknown>:0
  at TNManager.OnCreateObject (Int32 creator, Int32 index, UInt32 objectID, System.IO.BinaryReader reader) [0x00000] in <filename unknown>:0
  at TNet.GameClient.ProcessPacket (TNet.Buffer buffer, System.Net.IPEndPoint ip) [0x00000] in <filename unknown>:0
  at TNet.GameClient.ProcessPackets () [0x00000] in <filename unknown>:0
  at TNManager.Update () [0x00000] in <filename unknown>:0


Update 1 - Updating TNet on server and client seemed to fix things at the moment. In fact, restarting the executable on the server seems to fix things for a bit, then they get hectic again. The longer the server is up, the more channel open, closed, joined, left, things start getting bad. How are RFCs stored when they are saved? Do they go away when a channel is closed? Is there some way they could get crossed? And what's the best way to monitor what RFCs are saved?

[ArenMook]

RFCs disappear when the last player leaves the channel.

Judging by the stack trace, it received an object creation packet, but doesn't have a byte ID for what type of an object this is?

The structure of a Create packet is always rigid at the beginning:
- int: ID of the player that requested this object to be created.
- ushort: Index of the object being created (within a static list of prefabs on the client).
- uint32: Unique Identifier (aka Object ID) if requested, 0 otherwise. 0-16777215 range.
...followed by arbitrary amount of data.

This arbitrary data is expected to contain 1 byte for the type identifier, followed by function parameters. It looks like in your case you are getting a Create call with no identifier somehow. I'd love to know how that happens.

P.S. Delete the server.dat file.

P.P.S. Try to catch when this happens on the server. TNGameServer.cs, ProcessChannelPacket function around line 1258 is where the Packet.ResponseCreate gets sent. The "if (buffer.size > 0)" check below is what you're looking for. Debug break if it's zero.

[gg67]

I've never deleted the server.dat file because I never wrote or read to it. Does TNet automatically do this behind the scenes?

[ArenMook]

Yup. All persistent data is saved there.

[gg67]

Does persistent data include any data in a non persistent room? I don't create persistent rooms either.

Although deleting it may actually be helping. I noticed when I pulled you repo to the server last night everything seemed to work smoothly, even when restarting the server script didn't help.

[ArenMook]

It depends on when the save gets initiated. The server can be saved while it's running, for example -- which will just fully save its current state. It's very important to delete it every time the server/client code gets updated.

[gg67]

1. Will deleting the server.dat while the server is running cause issues? Or will I need to restart it?

2. This is bizarre. I've had this issue since I first got TNet and have finally found out what's going on. Let's say I'm syncing player's positions 15 times a second via UDP.

• Player A is in channel A.
• Player B is in channel B.
• Player A moves 10m to left of where he spawned in channel A and joins channel B.
• Player B is moved 10m to the left in channel B when Player A joins channel B.
  

What in the world is happening? It seems like the initial packets are over flowing from channel to channel.
Note: This only happens to the first player to join a channel and when other people join (usually only the second player to join).

UPDATE: After hours of debugging, I decided to include the channelID in the RFC call to update player position. Turns out, the very first packet in the new room is from the old channel. Do I need to call LeaveChannel before I call JoinChannel? I'm using the same packet ordering methods you use in your TNet Space Fighter Game kit.

Quick Fix: I kept the channelID in the RFC call and just ignored any packets where the packet's channel ID and TNMangaer.channelID didn't match.

3. Do saved RFCs from a player get removed when that player leaves the channel?

[ArenMook]

1. You need to restart it. Stop the server, delete the dat file, and start it back up.

2. That's a good work-around, and that's the downside of using UDP. Packets may arrive in the wrong order as they are not consecutive like with TCP.

3. That depends. Will the player's objects be removed? If it was created with a persistent flag, then no. RFCs are only removed if their TNObject is removed.

[gg67]

2. I only need to worry about this when using UDP packets, yes? Will TCP be okay?

And thanks!

[gg67]

Where would i want to look if everybody is disconnected from the server a few at a time and then a whole bunch of people get kicked? Does the server ever tell the client to disconnect? What about vice versa? When does this happen?

Ran into this issue again tonight w/ around 20 people in a channel. Happened after 30 minutes and then people couldn't even connect to server. Restarted server and all was well for about 30 minutes and then happened again. What could kick someone from server and then not let them reconnect?

And the worst part is I can't reproduce reliably...so i can't debug properly. So frustrating