PikkoServer

uLink is by itself capable of being used for developing massively multiplayer online games (MMO games). Using the tools described in the uLink manual it is possible to build a zoned, large-scale game with many simultaneous players. However, uLink also has support for using PikkoServer, an advanced load-balancing networking backend that connects several uLink servers into a seamless world, and dynamically load balances network objects based on their geographical distribution as to minimize network communication.

The main difference between an MMO and a regular networked game is the amount of players able to interact together in the virtual world. In a regular client-server game there is only a single server for each game region, and the amount of players able to play in the region is determined by how many players the server can handle. Using PikkoServer to develop an MMO, on the other hand, there can be several servers working together to distribute the load in a single region, allowing for much greater numbers of players. This means you can have large scale games with very heavy server side computation.

PikkoServer is not just a backend for MMO games, but is also capable of load balancing civilian and military high fidelity real time simulations, as well as constructive simulations. There is ongoing evaluation work to try it out in these contexts.

PikkoServer is in closed beta with Behaviour Interactive's Warhammer 40 000: Eternal Crusade. This manual is work in progress, as we want to add results that illustrate our description of the system.

Overview

PikkoServer is a load-balancer and distributor for online games, used for building MMOs capable of handling thousands of simultaneous players in the same area. PikkoServer makes it possible to use several servers for governing a single region in a game. It dynamically distributes the servers over the region so that they each cover their own section of it along with the objects in that section. These servers are just normal servers written using uLink, but with some extensions to jack into the PikkoServer framework. PikkoServer coordinates them to make for an object distribution over the region optimized for certain metrics depending on the load balancer configuration in PikkoServer. It is aware of the positions of the network objects in each server and of what region each server covers. When an object crosses a server's boundary, PikkoServer moves the object from the old to the new server.

Before delving into the details of how to develop MMOs with uLink, let's take a look at the classic approach of structuring an MMO game and how PikkoServer can improve on this.

The Classic MMO Approach

The structure of an MMO game is quite different from that of an ordinary, single-server based game. Since the world in an MMO is usually very large, it has to be spread out over many servers in some way. And since the number of simultaneous players in the world is usually also large, the load generated from them needs to be handled in some way.

The usual approach for dealing with these problems is to divide the world into a number of zones, where each zone covers a static portion of the game world. Each zone is then handled by its own game server that takes care of all the players and actions in that region. The zones are typically isolated from each other, in that they only rarely communicate with other zones' servers (typically this is done when they need to move objects to or receive objects from nearby zones) and, more importantly, in that players cannot see the actions taking place in a nearby zone. When players walk into a new zone, they need to be switched from the old server to the new server. This normally induces a noticeable delay for the player as the new zone is loaded and the old one unloaded.

An example of zoning would be in an MMORPG where a town is one zone and a region outside the town is a separate zone. When walking from the town to the outside area, a loading bar appears to notify the player of the zone switch, and after that, all the players in the town disappear and the players outside the town appear.

In addition to zoning, another important concept often used in MMO games is that of instances. Usually associated with MMORPGs, instances are akin to zones in that they are isolated portions of the world and can be handled by a separate server each. However, instances are used for creating many separate views of the same region of the world. Using the MMORPG example again, a certain dungeon in the world could be made instanced and thus spread over many game servers. Each instance would contain its own copy of the dungeon and act separately from the other instances. Upon entry to the dungeon, a random server could be chosen for the player. Many players could then visit the dungeon at the same time, but only players on the same server would be able to see and interact with each other.

The PikkoServer Architecture

uLink using PikkoServer supports developing MMOs using the same techniques and overall architecture described above. The Peer-To-Peer chapter shows how zoning can be implemented by having several servers and handing objects over between them. Instances can be made in a similar way.

While zoning and instancing work well for increasing the size of the world and the total number of simultaneous players in a game world, they have some major drawbacks. The maximum number of players that can interact with each other in a zone is still limited by how many players the server handling that zone can cope with. No matter how many servers are added, the limit for each zone will stay the same. And since you can't see the activities happening in other zones, this sets the limit for how many players you can see and interact with at the same time.

PikkoServer offers a sophisticated approach for solving this problem by allowing a zone to be distributed over many servers. Each server handles a part of the players in the zone, and PikkoServer coordinates the servers to ensure a seamless distribution. The client doesn't notice this server-side distribution - from the outside it will still look like a single, contiguous area with players spread out in it. The difference is that there can be a lot more players.

This is accomplished by PikkoServer working as a global distributor of networking data for the servers in a zone, deciding what part of the zone each server should handle along with the objects and clients in it. From the perspective of PikkoServer, each server in a zone is called a cell server. A cell server is basically just a normal game server with some extra features to make it fit into the PikkoServer architecture. Pikko Server is aware of the location of all network game objects and of the regions handled by cell servers. As objects and clients move around in the world, PikkoServer moves them between cell servers according to what regions are currently handled by which server. Moving an object or client from one cell server to another is called a handover. The handover functionality must be provided by the cell server according to a standard interface.

PikkoServer's Operation

PikkoServer acts essentially as an internal coordinator of a mesh network of cell servers that serve external clients. Cell servers begin by connecting to PikkoServer. PikoServer continuously adjusts the object distribution over connected cells in order to optimize criteria specified by the configured load balancer; primarily geographic locality. It also calculates through a culling algorithm what proxy objects there need to be on cell servers in order to enable seamless gameplay logic (such as raycasts crossing cell boundaries) to be written by the cell programmer. In turn, through proxy objects PikkoServer coordinates what cell servers should connect to each other to form the mesh network. Connected neighboring cell servers reduce load on PikkoServer by allowing many types of traffic to bypass it, greatly increasing scalability and minimizing latency for typical game worlds where there is a geographical separation between objects.

A client can connect to any cell server, at which point the cell will register the client for load balancing and assign it a camera object that determines visibility for the client. That client will get information from its current cell server, as well as other neighboring cell servers in its vicinity. Clients are seamlessly handed over along with their camera object to another cell during load balancing and there is no special client logic required to deal with this. The fact clients connect to cell servers rather than PikkoServer is different from the early prototype version of PikkoServer used for the world record in 2012. This change was made to allow driving huge outgoing bandwidth from cell servers running on ordinary gigabit Ethernet cards, while making it easier for the cell server programmer to write logic that limits what traffic clients receive and don't receive.

Above shows the Pikko architecture as in 2014.

To keep PikkoServer informed of the state of the world, the cell servers periodically send updates on the positions of the objects in the cell server. This is taken care of behind the scenes by uLink and needs not be implemented by the programmer, although some parameters, such as the sending interval, can be configured to fit the needs of game.

Using the position information and its local information on the regions handled by each cell server, PikkoServer notices when objects cross cell server boundaries and a handover is needed. When this happens, a handover message is sent to the cell server that the object left, requesting the state of the object. The cell server needs to handle this message and respond to it by serializing the state of the object. This state will then be sent by PikkoServer to the cell server that was entered, making sure it starts out with the exact same object. Implementing the serialization of objects and their proxies is the responsibility of the cell server programmer. If the handed over object is a camera for a client player, that client will be transparently handed over without any additional logic either in the cell server or the client being necessary.

The Mast Model

PikkoServer uses a model similar to how cellular networks work for determining the regions handled by each cell server. In a cellular network, a mobile phone is always connected to the closest physical mast, and is automatically switched between masts when moving around. The same principle is used for cell servers. Each cell server is assigned an associated mast in the virtual world by PikkoServer, and players and other game objects are then handled by the server with the closest mast. When a player moves away from its current mast toward another, a handover is initiated by PikkoServer and the player is moved to the new server.

The regions handled by cell servers in each zone are not static, they are continuously changed to adapt to the current situation in the zone. PikkoServer moves the masts around in the zone in order to balance the amount of players handled by each cell server. This means that handovers can happen between cell servers even when no player is moving, until equilibrium is achieved. This dynamic nature of the masts makes for a very flexible and scalable architecture that always adapts to the current gameplay.

For masts to be able to move around quickly, the cell servers governing a zone need to have the entire zone loaded into memory for the duration of the game. In this way, they will be able to handle any part of the zone equally well as any other and will be able to accept players entering from anywhere. Although it would be conceivable for cell servers to only load the part of the zone that currently contains players, and dynamically load in new parts as players move around, this is not currently supported since cell servers are not aware of their locations in the virtual world.