uLink Server Operations

The main focus for this guide is recommendations for game server operations when hosting Unity / uLink servers. At the end of this chapter there are guides for operations of the uLink policy server and the uLink master server on both Windows and Linux.

It is important to understand that a uLink server is not something separate or unique, it is a normal Unity server built and published as a standalone Unity application usually running with the graphics and interactivity turned off, on a Windows server. It is powerful, fast and very stable. In addition to that, it is easy to operate and this chapter will give you a good start.

One of the many things that makes the Untiy server very powerful compared to other game server platforms dedicated for web games is that the Unity server includes a physics engine for doing complete authoritative physics simulations for real-time games on the server side, and a lot more.

Please not that detailed uLink hosting instuctions are availble for Amazon EC2, but the same guide can be used as a check list for hosting uLink game servers with any other hosting provider.

Building a dedicated Unity server

This text is written for a game with one or more authoritative Unity servers hosted in some datacenter on Windows. uLink supports the same platforms as Unity. For practical purposes, we will focus on Windows for the server side.

In the Unity editor, the server should have a special scene, to make it easy to build and publish the server resources and scripts into a dedicated exe file. Press ctrl-shift-B in the editor, select the server scene and select PC and Mac Standalone, set Target Platform to Windows and press Build. Save the exe file locally. Note that in addition to the exe file there is a data folder. The exe file and data folder are what you need to transfer to the Windows server in the datacenter.

Transfer files to the datacenter

Transfer the exe file and the data folder to any folder at the Windows host. For example, use FTP or RemoteDesktop with a mapped local drive to make the transfer.

The server can not be started on Windows Datacenter Edition OS just by just double-clicking the exe file. This because Windows datacenter editions lacks some core libraries for running Unity at all. This is the only Windows OS that is allowed, and offered, by hosting companies, so be prepared for running your Unity server in batch mode.

When the server is supposed to be running without graphics, it must be started with a -batchmode command-line parameter. For example:

server1.exe -batchmode

But if you set up a Windows server at your office, it is possible to run the game servers with graphics. This is only for testing purposes and debugging since graphics will use a lot of resources on the server. We recommend having a camera in the server scene for this kind of debugging. Having that camera is very convenient, to be able to move around and get a visual of what is happening in the server when testing the game.

You can read more about command line arguments in Unity at Unity Manual - Command Line Arguments

Ports and firewalls

The developer working with uLink can choose any port number for running the game server. Just make sure this port is open in the datacenter's firewall (if the datacenter has a firewall) and make sure it forwards UDP traffic to the windows host.

Besides the datacenter's firewall there is usually the Windows firewall that is blocking incoming traffic by default. Make sure the Windows firewall allows incoming traffic to the UDP port the game server is listening on.

If you have fundamental connectivity issues, please try and turn the windows firewall completely off momentarily to eliminate it as a possible reason. If connections then work turn the firewall back on and reconfigure it. The windows firewall is the most common reason for connectivity issues and it's configurations can sometimes be changed with Windows updates.

Now, when the firewall is open, it should be easy to connect clients to the running game server. Make sure the game client is configured to connect to the IP-address/domain name of the windows host and the port number is correct.

There is no need to configure anything in the firewall for the clients. All home routers and firewalls are capable of handling this kind of UDP communication between the client and a server, when the server has a public IP address.

The Unity web player demands a running policy server in addition to the Unity server. Read about the policy server in the Policy Server chapter. Server operations for the policy server is described further down in this chapter.

Logging

By default, the Unity server writes all log output to a file named output_log.txt located in the data folder for the server executable. The uLink logging feature, through the uLink.NetworkLog class, also writes output to this log file by default. If you want to provide a different log writer method, perhaps for performance reasons or because you're using some other storage method, you can set the static writer delegate fields in uLink.NetworkLog. You can set up a different log writer method for each of the logging levels Info, Warning, Error and Debug.

With uLink.NetworkLog you can add a category to each log message. This should be common practice, as it opens up for dynamic filtering of log messages. All log messages from the uLink API follows this practice. When you build your server from the unity editor, the log level settings in the uLink settings GUI is also built with it. This way you can set the global minimum log level, or a specific log level for each of the logging categories. You can also turn off logging categories by the same method. If you want to use different log settings for the client and server, it may be easier to set the different logging settings from a startup script on both builds. Log settings by category is also very handy when debugging both your client and server, as you can turn on the Debug level for one or a few categories, while still keeping the log output reasonably small.

There is a number of user defined categories in the uLink.NetworkLogFlags enumeration, that can be freely used for your own purposes. Just define a constant with you own category name and assign it one of the UserDefinedX values. Then you can use you constant as the category when calling one of the uLink.NetworkLog log methods.

All code that uses Debug.Log() will write to the output_log.txt file.

We recommend that you add a timestamp to all your log outputs. This makes it a lot easier to debug problems by reading the logfile after a bug is detected. For example, add System.DateTime.Now.ToString() before the string messages in all event logging.

When running multiple game servers on the same machine, it is convenient to direct all log files into one single directory. This makes it easy as an operator to scan all log files from one single place. You can specify the log file path of a Unity exe with the command line argument -logFile.

For example:

test1.exe -batchmode -logFile c:/logfiles/test1_12345_log.txt

Starting several Unity servers

As long as every game server has a unique port it is possible to host a lot of uLink game servers on the same Windows host. If you are building a zoned MMO world it is natural to build one exe file for every zone and host each zone as a dedicated server with a unique port number.

When building an FPS game with several identical game servers running in parallel, make some kind of programming or configuration for these servers to pick one unique UDP port each. A good practice is sending the port number to the server when it starts, like this:

test1.exe -batchmode -port=12345

The game programmer can write code in a script in the server scene that reads the command line arguments and uses the given port number when the server starts. Put the code in an Awake() method. The programmer can use the following call to get the command line arguments:

System.Environment.GetCommandLineArgs()

If there is a need to start several identical Unity servers, for example when hosting several identical game servers for an FPS game, just make sure their log output gets directed to separate files, like this:

FPS_server_instance.exe -batchmode -port=7001 -logFile c:/logfiles/server_7001_log.txt
FPS_server_instance.exe -batchmode -port=7002 -logFile c:/logfiles/server_7002_log.txt

A smart way of naming your log files is to use the port number like the example does. If the logfile is named server_7001_log.txt, it will be fast and easy to find the log file for a game server if you detect that the server listening on UDP port 7001 has a problem. When you have hundreds of game servers running in parallel on a powerful dedicated Windows server, this detail is important.

And the other way around, when you find a fatal exception in a log file and you want to kill the game server process that is the owner of this log file, just look at the command line argument for all processes in the Windows task manager.

We recommend using the free application Baretail for reading several log files live on the server: http://www.baremetalsoft.com/baretail/index.php

Windows restart

Make a batch file for starting all the Unity servers. If all Unity servers are supposed to restart if the Windows server restarts, then use any of the many available Windows features for running this bacth file at startup.

CPU, memory and bandwidth

The amount of CPU, memory and bandwidth each Unity server needs is very much depending on the game and how it is programmed.

The minimum memory requirement for running a Unity server, including the uLink plugin, in batchmode with no connected players is 7.1 MB.

The following table shows the result from running our own Snowbox demo game server with different number of players on a Windows machine with an Intel E5430, 2.67 GHz processor:

PlayersMemory (in MB)CPU (of 1 core)Downstream bandwidth (in KB/s)
024.00%0.00
424.51%10.0
825.02%46.5
1625.55%195
3228.013%790
6431.039%3,230

The Snowbox game is not optimized for low-bandwidth or low-memory usage. Animations have been removed from the server to reduce the CPU usage, but the server is doing collision detection for all characters and all snowballs. This load test was made with 10 movement RPCs per second from each client and 1 snowball throw-RPC every two seconds.

Monitoring CPU, memory and bandwidth

The windows task manager works fine for monitoring CPU and memory usage. When running multiple identical Unity servers on the same Windows host it can be hard to se what port number and what log file they are using. In the Windows task manager it is possible to se the command line arguments used when starting the server. Look at the command line args to find the correct log file for a game server behaving bad (for example using too much CPU). To monitor upstream and downstream bandwidth for each individual Unity server, open the Windows tool Resource Monitor and click the Network tab. Note that all uLink traffic is based on UDP, except the policy server which uses TCP.

Trimming CPU usage versus latency

The Unity server will always go for 100% CPU to maximize the frame rate. This is OK behavior for a client, but not for a server. Therefore it is very important to always set this property in the Unity server scene:

Application.targetFrameRate = 60;

If this property hasn't been set, the Unity server will always show 100% CPU usage and the server operator will have no clue what so ever if the server is running smooth or if there is some kind of bug eating all the CPU resources.

The number 60 is a good starting point. This will make the server tick at least 60 times per second (also when running in batchmode and there is no graphics) and thus respond to incoming traffic within an interval of 16.6 ms (the mean expected value is 8.3 ms). Tick rate and frame rate is the same thing for a Unity server.

If you want to lower the CPU usage on the server, for example if the server is using lots of CPU doing physics simulations, then one optimization is lowering the target frame rate. But be aware that the server's addition to the round trip latency for you network traffic will increase when you do this.

Note that if you are running on an EC2 virtual instance you'll need to use the target frame rate fix to avoid having your server instance taking a large portion of the CPU resources as the frame rate will try to reach maximum values.

Partitioning servers to CPU cores

Unity servers are single-threaded. That means a single Unity server can not use the full power of a multicore processor. A Unity game server will only be able to use one core up to 100% CPU and then the frame rate will decrease.

It is normal and OK to run several Unity servers on one core. How much CPU every game server will use is very much depending on gameplay and the number of connected players. It is possible to host hundreds of Unity game servers on a modern 8 core Windows machine, at least if the number of players per server is low (2-4) players. If you do this, make sure the game servers are evenly distributed, resulting in a CPU load on each core at around 80 will be minimal. If one of the cores reaches 100the game servers using this core. Keep the CPU below 100 drops will result in higher server latency addition since it will not be able to handle network input and output at the normal frame rate.

This kind of partition among cores can be done very easily with a Windows tool called start. This is a script that starts eight Unity servers, and two of them are dedicated to each core on an 4-core machine.

start /affinity 1 test1A.exe -batchmode -port=12001
start /affinity 1 test1B.exe -batchmode -port=12002
start /affinity 2 test1C.exe -batchmode -port=12003
start /affinity 2 test1D.exe -batchmode -port=12004
start /affinity 4 test1E.exe -batchmode -port=12005
start /affinity 4 test1F.exe -batchmode -port=12006 
start /affinity 8 test1G.exe -batchmode -port=12007
start /affinity 8 test1H.exe -batchmode -port=12008 

The affinity option controls which cores the server will be allowed to execute on and can also be changed manually at runtime using the Windows task manager. Just right-click the server process and choose Set Affinity...

Hosting the policy server

If clients are Unity web players, then a Policy server needs to be hosted on the same external IP/domain name as to the actual Unity server. The motivation for a policy server is documented in the Policy Server chapter.

When using a single host machine, we recommend starting a policy server on the same host machine as the Unity server. When using multiple host machines we recommend having one policy server started on each machine. This is seldom a problem since it is small and easy to start and demands almost no resources at all. The only time it gets incoming network traffic is once per web player just before player connects to the Unity server.

It is possible to have just one single policy server for multiple host machines. All that is needed is to configure the datacenter's router to redirect all UDP traffic on port 843 to the one machine where the single policy server is running.

The policy server is a standalone exe file that is installed on your local machine when uLink is installed. Install uLink, select the exe file from the folder C:\Users\Public\Documents\uLink Tools\PolicyServer. Transfer the file to the host machine and start it. There is no need to insert the uLink license key.

The policy server can be hosted on Linux. Linux can be used when hosting the ulink Master server or Pikko Server, otherwise Windows is the required operating system for hosting Unity servers. The only requirement for hosting the policy server on Linux is that mono must be installed and fortunately mono is included in most Linux distributions by default. Transfer the exe file for the policy server to the host machine and start it with this simple command:

mono uLinkPolicyServer.exe

When hosting the uLink master server (including the uLink proxy server) on Linux there is also a need to host the policy server on the same Linux host machine.

We recommend configuring Windows/Linux to automatically start the policy server when the operation system is started.

Hosting the master server

Some game types need a lobby system where players can search and find running game servers that need more players to join. The built-in lobby system in uLink is called the master server. Since the release of uLobby, the recommended lobby system is uLobby, since it is more advanced and easier to use. But the uLink master server can still be used (if you want to), and especially if you want to use the proxy feature included in the uLink master server.

The master server is a standalone exe file that is installed on your local machine when uLink is installed. Install uLink, select the exe file from the folder C:\Users\Public\Documents\uLink Tools\MasterServer. Transfer the file to the host machine and start it. There is no need to insert the uLink license key.

The master server default UDP port number is 23466. Make sure this port is open in the firewall.

When the master server is running, clients can connect to it and search for available Unity game servers and the Unity game servers can connect to it to register new running game sessions. The network traffic to the master server is not very heavy because the actual game traffic is not going through the master server. At least when the proxy server feature turned off.

The policy server can be hosted on Linux. The only requirement is that mono is installed and fortunately mono is included in most Linux distributions by default. Transfer the exe file to the host machine and start it with this simple command:

mono uLinkMasterServer.exe

Hosting the proxy server

The proxy server is a feature in the master server that is not turned on by default. The proxy server is a man-in-the-middle traffic relay server for games that have game servers hosted on players' machines. When the network traffic of a game is going via the proxy server it is possible for all kinds of Unity clients (web players, standalone clients, even iPhone clients) to act as a Unity/uLink game server. The proxy server makes it possible for clients without a public IP/domain name to act as a Unity server. Read more about using the master server and a proxy server in the Master Server chapter.

The proxy server must be enabled when the master server starts. The port range for UDP sockets that the proxy server can use has to be assigned:

uLinkMasterServer.exe -proxy 41000:42000

Make sure the firewall is open for traffic to the assigned ports. Every player not acting as a game session server will get a UDP socket in the proxy server. For example: If you have 10 players starting 10 game sessions and then 5 additional players connecting to each game, the proxy server will open 5 * 10 = 50 sockets using UDP ports 41000-41049.

Proxy server performance

The proxy server will send and receive all network traffic for each game server needing a proxy. It is multithreaded and will use all available CPU cores. We recommend hosting the master server including the proxy server on a dedicated Linux server with many CPU cores. Linux is a bit cheaper and the network tweaking possibilities and the monitoring tools are more powerful, especially for detecting packet loss in the OS buffers for incoming and outgoing UDP packets (Linux command: netstat -su).

We have load tested the proxy server on an 8 core Linux server equipped with two Intel E5430, 2.67 GHz (4 cores each). We tested with an FPS game and tried to reach as many concurrent players as possible. The traffic from each player was simulated with 10.5 packet/s with around 300 Byte/packet. The total bandwidth from each client was 3.2 kB/s. Normal bandwidth for FPS games, from client to server, is between 1.5 and 5 kB/s.

The result depends on how many players there are in each game server. The test was done without network compression techniques such as culling and LOD-ing. Network culling decrease bandwidth usage when players are unable to see each other, and LOD (Level Of Detail) can reduce the network traffic for moving objects that are far away. In these tests all actions from each player were broadcast to all other players, without the usage of mentioned compression techniques. This way we are able to present some worst case numbers below.

When adding more players per game server, the bandwidth needed from game server to clients will increase heavily because each player's activities must be sent to all the other players in the same game.

The proxy server can handle:

  1. 1800 players in 450 game servers with 4 players per game server. The downstream bandwidth from the proxy server is 19.5 MB/s.
  2. 1600 players in 200 game servers with 8 players per game server. The downstream bandwidth from the proxy server is 28 MB/s.
  3. 1200 players in 75 game servers with 16 players per game server. The downstream bandwidth from the proxy server is 55 MB/s.

When there are many players per game server in an FPS game, we recommend hosting the game servers for this game in the datacenter instead. The bandwidth out from a game server with, for example 32 players connected, will be high and therefore it may become a bottleneck to host this kind of game server on one of the player's machines. If the bandwidth for this player is limited, then the complete game with all 32 players will suffer from packet loss and other related network problems.

The latency for the proxy server architecture can be optimized a lot since all network traffic between players has to pass the proxy server twice. Please consider hosting authoritative game servers in a datacenter instead of using the proxy server if you want to offer your players minimal latency.

The configuration details for this Linux setup is available for all uLink customers on request.