Questions and Answers about Scaling (Horizontally and Vertically)

[kamilion]

So, I bought tnet3 the night before last, tried running the server on my odroid c1 quadcore ARM SoC with 3.0.1 and it crashed; got the 3.0.2b update and it's working fine now.
Unzipped the server source and built it, watched the binary run on my odroid with htop, and started going through the example scenes.
I was playing with the multiple channels example and spawned massive amounts of cubes in each channel; and had fifteen clients connected. Things worked pretty much as I expected.

I've been poking through the classes, and notice there's http/websocket support, new ipv6 support, and SendQuickly to try udp if possible.

If possible, I'd like to provide webgl builds as well as win/lin/osx builds of my game.
One of the things that wowed me in the demo video was loading the assault helicopter asset.
Can serverside assets be streamed to webgl? Am I better off giving up on webgl if I want to deal with loading player-exported assets?

How do I provide my players with a unity3d scene suitable for exporting content?

I have access to extremely large servers, a quad socket opteron with 32 cores and 128GB of memory, or 4 dell C6100 blades with dual xeon E5520s and 48GB ram each for 192GB total.

I intend to build a first/third person shooter on a ~9.8x9.8km area of terrain, with all objects being droppable and persistent.
This means things like: Weapons, Ammo, bullet casings (brass), first aid kits, briefcases containing multiple items...
I have a working pickup truck with an ik-based steering wheel and my mechanim driven player characters using a third person chasecam.

I have not yet added a first person camera or attempted to resolve the problem of syncing mouselook to an avatar's head rotation and spinal curve.
In particular; the player characters I'm using have an odd rotation to the spine in a few of the various 3D models (Ruffian, police Swat, civilian, worker, clerk) so I'm thinking of improving and adding a second avatar with no mesh, and only the hip, spine, and head joints and writing a short script to override the animation controller on the meshed avatar to follow the 'shadow' avatar via IK. The shadow avatar only needs to exist for player-controllable avatars. Preferably, it would be a prefab that would instantiate the meshed avatar of the player's choice. I think this would allow me to deal with AI-driven avatars as well, but my question is, where do I put that logic, as tnetserver does not seem to simulate a world?
Do I run a unity client in a VM with some kind of account with special permissions on tnet? (admin?) How does Admin work, anyway? For that matter; how does the lobby work? Can it redirect to another IP? Can multiple simultaneous sockets be open from the client if the lobby server and channel host are on different systems? Can the lobby server instruct the client whom to connect to next already, or do I have to write additional classes for tnet server to deal with routing?

To keep with the idea of server authority, I'll be writing the input scripts separately from the scripts actually pulling the strings inside the spawned avatar, syncing input frequently -- but what is the best practice to sync the avatar's state? Do I sync the character controller somehow? The animator? The 'shadow spine' should be enough for player controlled characters in first person along with syncing some animation booleans of which state they're in (such as reloading a weapon); but there's also a dead state where the player is a disembodied camera that can wander for a while before respawning, leaving the avatar behind as a ragdoll.

At a high level, what are the tnet functions I'd be doing for handling player characters like this?

What about entering a vehicle? What's the best practice to reparent a player avatar as a child of the vehicle? With unet this proved extremely hard because initially I had it set up so players and vehicles both had networkidentities; reparenting a player to a ladder worked (no networkidentity), but reparenting a player to a vehicle did not. What I ended up doing in an earlier demo was adding an IK point on the car seat for the hip bone and having IK drag the player along with the vehicle without reparenting after handing off input control.
Over UNET this sort of mostly worked and even had the hands yankin' the steering wheel around through the IK targets... but eventually he'd start to drift. I have a screenshot around somewhere of his hip somewhere in the hood with his arms pinned behind him still desperately trying to fondle the steering wheel to control the pickup. I still think that given a server-authoritative architecture, I should be doing more things like using IK pods on the avatar's hands and setting world items to follow the pod, while exposing the magazine's grasp point as an IK target for the detach/reload animations lerping to work without having to make a whole bunch of custom animations and figure out how to sync them. I figure just let the IK solver deal with it on the clients and hand-wave away draggy-laggy items. Reparenting things when networking is involved just seems like a recipe for things getting out of hand, I don't know why, I just have that feeling after dealing with UNET. Hoping someone will tell me "nah, reparenting works fine in tnet, just do it like this:", I guess.

I haven't yet started the conversion over to TNET3 so I'm trying to figure out the recommended way to do things before diving in and wasting a whole bunch of time spinning my wheels.

From the tests I did on the odroid; it seems like TNet is a store-and-forward setup. However, it stores state to disk in a format that doesn't look like it supports being updated concurrently.
(This means I probably shouldn't run it more than once from a central SMB/NFS share.)

Do I write new code to store lots of little files and pray I don't run into a multiple writer situation? (seems to work for sqlite in practice)

That brings up the question, given that I have a choice between a single large multithreaded server or four independent blades, how best do I scale my game to meet a higher player demand?

The channel system seems like it will work pretty well for my large terrain, splitting areas of concern up and allowing the use of multiple scenes in unity (since punching a hole in a terrain's so hard, I made a crummy model of an railway entrance with a barred gate, jiggling the lock will trigger the scene+channel change, I imagine) but I'm not exactly understanding a good strategy towards growing even bigger. As players do their thing, resources will spawn, things will be crafted, and lots and lots of data has to be stored.

So, I probably want to back this with some kind of persistence in a database. One that's clustered, scales easily, accepts a common interchange format, and has a .net driver.
If those are my requirements; rethinkdb seems like a pretty good choice, given it stores json documents, has a .net driver and the most recent version adds access control.
There's two .NET drivers:
https://github.com/bchavez/RethinkDb.Driver
and
https://github.com/mfenniak/rethinkdb-net

Both of them have a minimum requirement of .net 4.5, although rethinkdb-net suggests it is "100% compatible with Mono (3.0+)", I still couldn't get it to work within unity itself and UNET.

But tnetserver doesn't rely on the unityengine namespace, and can be compiled against the 4.6 target according to this topic: http://www.tasharen.com/forum/index.php?topic=14259.0

Given that's the case; what kind of plan of action should I put together if I wanted to store data in rethinkdb? One of it's better features is being able to register a 'changefeed' on a document or document value and get an event delivered when it changes on the server.

With that information in mind; is it going to be worth spending my time implementing support code in tnetserver for accessing rethinkdb/paying someone else to, or should I start thinking of a different scaling strategy to deal with hundreds to thousands of concurrent players in a semi-seamless large region?

Since I've outlined pretty much everything about my requirements and plans; anyone wanna PM me a quote in USD after looking over rethinkdb and the drivers?
If it's useful; I've got no problems with the end result showing up in tnet itself for everyone to use, if that has any discount for sharing, I'm all for it.

I've tried various photon setups on previous projects, bolt and a trial of PUN, various other stuff in the asset store like this mysql client thing and trying to port one of the rethinkdb .net drivers to unity's outdated flavor of mono, and Forge has to run inside a unity process...
I've got my own server hw that I only have to pay a power bill and hurricane electric 100mbit uplink, so I really don't want any of the cloudy-stuff that's so popular.

(Sorry for the huge post; but I've got a lot of complex questions, and I suspect other people'd like to know the answer to at least one or two of these themselves, but are too busy doing other stuff to ask)

[Bill.Smock]

Responding to this ridiculous post to pay it forward for the people who have helped me (CW and Aren).  Can't answer everything though...

I think this would allow me to deal with AI-driven avatars as well, but my question is, where do I put that logic, as tnetserver does not seem to simulate a world?
Do I run a unity client in a VM with some kind of account with special permissions on tnet?

Run AI logic on the channel host's client.  This includes spawning things that you only want spawned once.

1. if(TNManager.isHosting) // AI logic

How does the lobby work?

The server's lobby keeps a list of available server instances.  You open a server instance and connect to the lobby server simultaneously.

1.         /// <summary>
2.         /// Start a local game server and connect to a remote lobby server.
3.         /// </summary>
4.  
5.         static public bool Start (int tcpPort, int udpPort, string fileName, Type type, IPEndPoint remoteLobby, bool openPort = true)

To make a list available to clients, you will have to build a lobby UI that keeps track of the available server instances.  Based on this post, you may as well buy the Starlink package.  It has everything you need to get a lobby up and running relatively quickly.  I purchased it and it's working out well for me.

Can it redirect to another IP?

The lobby can contain multiple server 'buttons' that allow you to connect to the server of your choice.

Can the lobby server instruct the client whom to connect to next already, or do I have to write additional classes for tnet server to deal with routing?

AFAIK you will have to keep track of whatever metrics determine which server is next to allow players, and then you would have to send a function to the clients that tells them which IP to connect to when they're ready to connect. (assuming you don't have a server buttons lobby).

To keep with the idea of server authority, I'll be writing the input scripts separately from the scripts actually pulling the strings inside the spawned avatar, syncing input frequently -- but what is the best practice to sync the avatar's state? Do I sync the character controller somehow? The animator? The 'shadow spine' should be enough for player controlled characters in first person along with syncing some animation booleans of which state they're in (such as reloading a weapon); but there's also a dead state where the player is a disembodied camera that can wander for a while before respawning, leaving the avatar behind as a ragdoll.  At a high level, what are the tnet functions I'd be doing for handling player characters like this?

Go through all of the source code of the included tutorials.  Multiple times if you have to.  Everything you need to know for these questions is available in the examples.

What about entering a vehicle? What's the best practice to reparent a player avatar as a child of the vehicle? With unet this proved extremely hard because initially I had it set up so players and vehicles both had networkidentities.

Add TNObject script to any objects you want to play with over the network.  Remote function calls and remote creation calls will probably be enough to get this done.

I haven't yet started the conversion over to TNET3 so I'm trying to figure out the recommended way to do things before diving in and wasting a whole bunch of time spinning my wheels.

Connect to a LAN server.  Connect to a remove server.  Create a server instance with a PC.  Now that you know how to do those... Each server instance can contain any number of channels.  Channels can be created within the instance on the fly.  In order to create a new channel, the client must already be connected to a server instance.  Each channel has a host (TNManager.isHosting) and that's how you decide who runs AI.  Go through the examples 100x so you completely understand RFCs and RCCs in this system.  They are amazingly simple once you understand them. 

The channel system seems like it will work pretty well for my large terrain, splitting areas of concern up and allowing the use of multiple scenes in unity (since punching a hole in a terrain's so hard, I made a crummy model of an railway entrance with a barred gate, jiggling the lock will trigger the scene+channel change, I imagine) but I'm not exactly understanding a good strategy towards growing even bigger. As players do their thing, resources will spawn, things will be crafted, and lots and lots of data has to be stored.

Two questions to ask yourself:  "Do I want the data stored locally on their PC or the server?" and "Why not try looking into datanodes?"


This post really looks like you're getting baked out of your mind, coming up with a thousand ideas at once, and then praying that there's a solution out there that will take care of all this shit at once.  You need to write a design document and stick to it.  This is insanity. 

[kamilion]

Run AI logic on the channel host's client.  This includes spawning things that you only want spawned once.

1. if(TNManager.isHosting) // AI logic

Where's this go? in tnetserver's codebase or the unity client? That would imply I'd need to run a VM with a headless unity client running the AI code.

The server's lobby keeps a list of available server instances.  You open a server instance and connect to the lobby server simultaneously.

1.         /// <summary>
2.         /// Start a local game server and connect to a remote lobby server.
3.         /// </summary>
4.  
5.         static public bool Start (int tcpPort, int udpPort, string fileName, Type type, IPEndPoint remoteLobby, bool openPort = true)

To make a list available to clients, you will have to build a lobby UI that keeps track of the available server instances.  Based on this post, you may as well buy the Starlink package.  It has everything you need to get a lobby up and running relatively quickly.  I purchased it and it's working out well for me.
The lobby can contain multiple server 'buttons' that allow you to connect to the server of your choice.
AFAIK you will have to keep track of whatever metrics determine which server is next to allow players, and then you would have to send a function to the clients that tells them which IP to connect to when they're ready to connect. (assuming you don't have a server buttons lobby).

Great, so it already can act as a classic master-server and route a player inbound to another server.
Does it retain a connection to the lobby server during gameplay? I didn't notice anything like that in the autojoin example.

Go through all of the source code of the included tutorials.  Multiple times if you have to.  Everything you need to know for these questions is available in the examples.

Add TNObject script to any objects you want to play with over the network.  Remote function calls and remote creation calls will probably be enough to get this done.
Connect to a LAN server.  Connect to a remove server.  Create a server instance with a PC.

Yes, I've read large chunks of the source. You're incorrect that everything is in the examples; such as demonstration logic for transferring a mechanim player avatar to a vehicle.
It's nice that there's a demonstration vehicle, and even nicer that the network-specific code inherits from the singleplayer class. But it's still unclear what kind of best practice for the handoff should entail.

I've already gotten the standalone server running on an ARM system on a chip, as I've already had lots of experience in C# as part of the TShock server team for Terraria. The issue I'm having is that it's unclear where I should be extending; additional methods and packets available in tnetserver, or getting the server to store datanodes in a more cluster-friendly way.

  Now that you know how to do those... Each server instance can contain any number of channels.  Channels can be created within the instance on the fly.  In order to create a new channel, the client must already be connected to a server instance.  Each channel has a host (TNManager.isHosting) and that's how you decide who runs AI.

Non-player AI has to be serverside. They'll be doing things like vending weapons to players, so that has to run in some sort of trusted context. And the number of channels is finite, 32bit int giving a bit over 2bil channels; but it IS finite. But if I'm understanding correctly; at least one unity client must be connected for tnet to have a host in a channel. No clients, no host, everything goes idle and 'the world pauses' until another client comes along to 'be a tnet host'.

So that tells me that I shouldn't do something like track wallclock time in tnetserver to trigger events occuring; INSTEAD I should have one or many headless unity clients running in VMs dealing with timed events, AI, market transfers, and such, or just talk over HTTPS to a REST microservice directly for some things instead of involving the tnet server.

Go through the examples 100x so you completely understand RFCs and RCCs in this system.  They are amazingly simple once you understand them. 
Two questions to ask yourself:  "Do I want the data stored locally on their PC or the server?" and "Why not try looking into datanodes?"

It's an MMO; players can't be trusted. I'll allow players to build content locally and submit to a preview grid for inspection (so they're not making OP guns, ETC) but one of the biggest reasons I bought tnet was the datanode export to file and upload to dedicated server. The workflow I build around that isn't going to be terribly complex.
Remote procedure calls have been around since ages; and DBUS on linux is a lot more complex. TNET's much simpler but has some odd quirks that I'm trying to generate an internal explanation of.

This post really looks like you're getting baked out of your mind, coming up with a thousand ideas at once, and then praying that there's a solution out there that will take care of all this shit at once.  You need to write a design document and stick to it.  This is insanity.

There IS a solution out there that takes care of everything at once: Unity. 

Everything I spoke of is mostly built into unity or slightly extended through another asset package.
Large terrain is handled by terrain composer and instantiated by seed; but it's still using unity terrains and unity texture splatmapping under the hood.
Characters are all mechanim driven using poses and lerping for smooth animation tweening, again, built into unity.
Vehicles interact with player characters through IK, already built in and I have working code that won't take me long to port to TNET RPCs.

Everything else really comes down to inventory control, and I have working unet prototypes for the weapons, tracking individual bullets in clips, picking brass up to increment a counter on the player's inventory, decrementing that counter at a reload station to craft new ammo, mixing ammo types in clips...

The problem is scale; while the terraincomposer stuff can load and unload terrain chunks on the fly when channel handover's done; datanode presents some concurrency limits and I need to know the lifecycle when I have hundreds of thousands of channels and millions of tracked objects.

I appreciate your comments; but just because a project is full of simple components that interact at a higher level does not mean it has to be complex, nor do I have to be on drugs to make extended use of unity's built in tools.

Also, I have a design document; but I'm not going to subject y'all to 56 pages of flowcharts detailing economic pressures and item crafting progressions for a bird's eye Q&A post on a forum, Not even if you asked nicely. There's enough information here already for someone to clone my project if they wanted to.
I'm used to large projects: https://github.com/kamilion/kamikazi-core
I grew up and live in silicon valley, used to work at NASA Ames doing robotics, now I work at a company that recycles servers at the end of their lease time; so I've been slowly building up a VPS provider package using Xen and ubuntu. If I need to spawn a hundred 2 core 1GB VMs to host this all; I can easily do it.

Capacity is not a problem for me:
https://goo.gl/photos/8USkuspAGL2d2HBm9

I have a quarter rack with coresite santaclara with a 100mbit commit on a 10000mbit pipe from hurricane electric. Latency from the US isn't an issue.
http://puu.sh/prT2v/96db274a93.png
I might have to figure something else out for europe.

Not trying to boast here; just pointing out that I've got a lot of technical experience in systems administration and a glut of inexpensive equipment available.
(Willing to trade server parts for code writing too, fire me a PM, peoples.)

[ArenMook]

Where's this go? in tnetserver's codebase or the unity client? That would imply I'd need to run a VM with a headless unity client running the AI code.

In your game code. In TNet, the server merely echoes packets to their intended destination. Clients run everything. So in short, code your game like you normally would if it was single player. When you need to sync something, don't just call function A -- instead, send the function call through TNet. For example instead of this:

1. DoStuff();

Do this:

1. tno.Send("DoStuff", Target.All);

1. [RFC] void DoStuff () { }

All of this is explained in the tutorials. You can read all the tutorials without buying TNet.

Does it retain a connection to the lobby server during gameplay? I didn't notice anything like that in the autojoin example.

That's up to you. You can, it's just a matter of keeping the game object with the lobby script attached around.

I've already gotten the standalone server running on an ARM system on a chip, as I've already had lots of experience in C# as part of the TShock server team for Terraria. The issue I'm having is that it's unclear where I should be extending; additional methods and packets available in tnetserver, or getting the server to store datanodes in a more cluster-friendly way.

Don't touch the TNServer unless you absolutely have to. Everything should remain on your game side.

Non-player AI has to be serverside.

No, the server is just there to echo packets. It knows nothing about your game or the game's logic. There's also no point in running AI in an area if there's no one there. Let players run the AI for you. As Bill mentioned, you can use TNManager.isHosting, or better still tno.isMine. For example, if before you had this:

1. void Update ()
2. {
3.    // AI code
4. }

You will now have this:

1. void Update ()
2. {
3.     if (tno.isMine)
4.     {
5.         // AI code
6.     }
7. }

The idea is that only the player that owns the object will run its AI. The "tno.isMine" will evaluate only on one client. When that client leaves, another player automatically inherits the ownership (the transition is seamless and invisible from your point of view).

No clients, no host, everything goes idle and 'the world pauses' until another client comes along to 'be a tnet host'.

That is correct. If you really want there to be something going on with no one around, build a headless Linux client with Unity and have it connect to your game (and join appropriate channels). Remember: server will echo the packets for you. Clients will run the code. If you want an authoritative client of your own, a separate headless build is the way to go. It's still going to be a client, so the server remains untouched. To other players it will seem that the AI is server-side, when in fact it would be a separate client, controlled by you. As a nice advantage of this, is you can migrate your headless client from one point to another without shutting down the server just by launching the headless client it in point B, then shutting down point A.

It's an MMO; players can't be trusted.

If you need that kind of trust, either go with the dedicated headless client approach I just suggested, or add verification to the server side (have the server process packets, not just forward them). This will complicate the server side greatly, but it's pretty simple to do. Just open up the TNGameServer and look for the ProcessForwardPacket() / ProcessChannelPacket() functions.

large terrain

This is easy to do in TNet 3. TNet 3 supports multiple channels that you can join/leave at will as you traverse your world. All channel data can remain persistent, and objects can travel from one channel to another. Simply subdivide your huge world into smaller logical chunks, with each chunk having a channel ID. I recommend simply basing them on the world position personally, but it's up to you. As players traverse the world, have them join appropriate channels around them. When they create world objects (for example dropping a piece of equipment), create them in the closest channel to wherever they are. This will make them appear to all others nearby, and will vanish when those players get out of range.

Consult the multiple channels example that comes with TNet. It shows all of this.

[ArenMook]

Unrelated, but I am actually looking for a host that would be cheaper than Amazon. Hosting a Windward server on Amazon would run me around $200 per month, even though all I care about is the I/O performance. TNet's server uses virtually no memory (<300 MB at peak) and no CPU (5% with 300+ players).