UDN
Search public documentation:

NetworkingOverview
日本語訳
中国翻译
한국어

Interested in the Unreal Engine?
Visit the Unreal Technology site.

Looking for jobs and company info?
Check out the Epic games site.

Questions about support via UDN?
Contact the UDN Staff

UE3 Home > Networking & Replication > Unreal Networking Architecture

Unreal Networking Architecture


Overview


Multiplayer gaming is about shared reality: that all of the players feel they are in the same world, seeing from differing viewpoints the same events transpiring within that world. As multiplayer gaming has evolved from the little 2-player modem games that characterized Doom, into the large, persistent, more free-form interactions of games like Quake 2, Unreal, and Ultima Online, the technologies behind the shared reality have evolved tremendously.

Implement Networking from the beginning!

One important thing to realize is that if you plan to support networked multiplayer in your game, build it in and test it as you develop your game! Building an efficient networking implementation will have a significant impact on your game object design decisions. Retrofitting a solution is hard, and design decisions that make a lot of sense when networking is not considered such as splitting functionality across many objects may cause significant issues with multiplayer.

Peer-to-Peer model

In the beginning, there were peer-to-peer games like Doom and Duke Nukem. In these games, each machine in the game was an equal. Each exactly synchronized its input and timing with the others, and each machine carried out the same exact game logic on exactly the same inputs. In conjunction with completely deterministic (i.e. fixed-rate, non-random) game logic, all players in the machine perceived the same reality.

The advantage of this approach was simplicity. The disadvantages were:

  • Lack of persistence. All of the players had to start the game together, and new players couldn't come and go as they pleased.
  • Lack of player scalability. Because of the lock-step nature of the networking architecture, the coordination overhead and chance of network-induced failure increases linearly with the number of players.
  • Lack of frame rate scalability. All of the players had to run at the same internal frame rate, making it difficult to support a wide variety of machine speeds.

Client-Server model

Next came the monolithic client-server architecture, pioneered by Quake, and later used by Ultima Online. Here, one machine was designated "server", and was responsible for making all of the game play decisions. The other machines were "clients", and they were regarded as dumb rendering terminals, which sent their keystrokes to the server, and received a list of objects to render. This advancement enabled large-scale Internet gaming, as game servers started springing up all over the Internet. The client-server architecture was later extended by QuakeWorld and Quake 2, which moved additional simulation and prediction logic to the client side, in order to increase visible detail while lowering bandwidth usage. Here, the client receives not only a list of objects to render, but also information about their trajectories, so the client can make rudimentary predictions about object motion. In addition, a lock-step prediction protocol was introduced in order to eliminate perceived latency in client movement.

Still, there were some disadvantages to this approach:

  • Lack of open-endedness - When users and licensees create new types of objects (weapons, player controls, etc), glue logic must be authored to specify the simulation and prediction aspects of those new objects.
  • Difficulties of the prediction model - In this model, the network code and the game code are separate modules, yet each must both be fully aware of the implementation of the other, in order to keep the game state reasonably synchronized. A strong coupling between (ideally) separate modules is undesirable because it makes extensibility difficult.

Unreal networking architecture

Unreal introduces into multiplayer gaming a new approach termed the generalized client-server model. In this model, the server is still authoritative over the evolution of the game state. However, the client actually maintains an accurate subset of the game state locally, and can predict the game flow by executing the same game code as the server, on approximately the same data, thus minimizing the amount of data that must be exchanged between the two machines. Servers send information about the world to clients by replicating relevant actors and their replicated properties. Clients and servers also communicate through replicated functions, which are replicated only between the server and the client which owns the Actor on which the function is called.

Further, the game state is self-described by an extensible, object-oriented scripting language - UnrealScript - which fully decouples the game logic from the network code. The network code is generalized in such a way that it can coordinate any game which can be described by the language. This achieves a goal of object-orientation which increases extensibility, the concept that the behavior of an object should be fully described by that object, without introducing dependencies on other pieces of code which are hard-wired to know about the internal implementation of that object.

Basic concepts


Goal

The goal here is to define Unreal's networking architecture in a fairly rigorous manner, because there is a fair amount of complexity involved that is easy to misinterpret if not defined exactly.

Basic terminology

We define our basic terminology precisely:

  • A variable is an association between a fixed name and a modifiable value. Examples of variables include integers such as X=123, floating point numbers such as Y=3.14, strings such as Team="Rangers", and vectors such as V=(1.5,2.5,-0.5).
  • An object is a self-contained data structure consisting of a fixed set of variables.
  • An actor is an object capable of independently moving around in a level and interacting with other actors in that level.
  • A level is an object which contains a set of actors.
  • A tick is an operation that updates the entire game state given that a variable amount of time called DeltaTime has passed.
  • The game state of a level refers to the complete set of all actors that exist in that level and the current values of all of their variables at a time when a tick operation is not currently in progress.
  • A client is a running instance of Unreal Engine which maintains an approximate subset of the game state suitable for approximately simulating the events that occur in the world, and for rendering an approximate view of the world for a player.
  • A server is a running instance of Unreal Engine which is responsible for ticking a single level and communicating the game state authoritatively to all of the clients.

The update loop

All of the above concepts are well-understood with the possible exception of the tick and game state. So, these will be described in more detail. First of all, here is a simple description of Unreal's update loop:

  • If I am a server, communicate the current game state to all of my clients.
  • If I am a client, send my requested movement to the server, receive new game state information from the server, render my current approximate view of the world to the screen.
  • Perform a tick operation to update the game state, given that a variable amount of time DeltaTime has passed since the previous tick.

A tick operation involves updating all of the Actors in the level, carrying out their physics, informing them of interesting game events that have occurred, and executing any necessary script code. All of the physics and update code in Unreal is designed to handle a variable amount of time passing.

For example, Unreal's movement physics looks like:

 Position += Velocity * DeltaTime

This enables greater frame rate scalability.

While a tick operation is in progress, the game state is being continually modified by the code which executes. The game state can change in exactly three ways:

  • A variable in an Actor can be modified.
  • An Actor can be created.
  • An Actor can be destroyed.

The server is the man

From the above, the server's game state is completely and concisely defined by the set of all variables of all actors within a level. Because the server is authoritative about the game play flow, the server's game state can always be regarded as the one true game state. The version of the game state on client machines should always be regarded as an approximation subject to many different kinds of deviations from the server's game state. Actors that exist on the client machine should be considered proxies because they are a temporary, approximate representation of an object rather than the object itself.

When a client loads a Level to use in a networked multiplayer game, it deletes all Actors in the Level except those that have either bNoDelete set to true or bStatic set to true. Other Actors relevant to that client (as determined by the server) will be replicated from the server to the client. Some Actors (such as the GameInfo Actor) are never replicated to a client.

Bandwidth Limitation

If network bandwidth were unlimited, the network code would be very simple: at the end of each tick, the server could just send each client the complete and exact game state so that the client always renders the exact view of the game as is occurring on the server. However, the Internet reality is that 28.8K modems have only perhaps 1% of the bandwidth necessary to communicate complete and exact updates. While consumers' Internet connections will become faster in the future, bandwidth is growing at a rate far lower than Moore's law which defines the rate of improvement in games and graphics. Therefore, there is not now and never will be sufficient bandwidth for complete game-state updates.

So, the main goal of the network code is to enable the server to communicate a reasonable approximation of the game state to the clients so that the clients can render an interactive view of the world which is as close to shared reality as is reasonable given bandwidth limitations.

Replication

Unreal views the general problem of "coordinating a reasonable approximation of a shared reality between the server and clients" as a problem of "replication". That is, a problem of determining a set of data and commands that flow between the client and server in order to achieve that approximate shared reality.

Actors


Roles

In general, every Actor has a Role and a RemoteRole property, which have different values on the server and the client. Every Actor on a server has a Role set to ROLE_Authority.

Actors on the server may have as their RemoteRole:

  • ROLE_AutonomousProxy - PlayerControllers and the Pawn they control, when replicated to the owning client
  • ROLE_SimulatedProxy - All other replicated Actors
  • ROLE_None - Actors which are never replicated to any client

The RemoteRole of an Actor on the server is the Role of that Actor on the client. All Actors replicated to a client have RemoteRole set to ROLE_Authority.

Definition

The Actor class defines the ENetRole enumeration and two variables, Role and RemoteRole, as follows:

Actor.uc
// Net variables.
enum ENetRole
{
   ROLE_None,              // No role at all.
   ROLE_SimulatedProxy,    // Locally simulated proxy of this actor.
   ROLE_AutonomousProxy,   // Locally autonomous proxy of this actor.
   ROLE_Authority,         // Authoritative control over the actor.
};
var ENetRole RemoteRole, Role;

The Role and RemoteRole variables describes how much control the local and remote machines, respectively, have over the actor:

  • Role == ROLE_SimulatedProxy - means the actor is a temporary, approximate proxy which should simulate physics and animation. On the client, simulated proxies carry out their basic physics (linear or gravitationally-influenced movement and collision), but they don't make any high-level movement decisions. They just go. They can only execute script functions with the simulated keyword; and they can only enter states marked as simulated. This situation is only seen in network clients, never for network servers or single-player games.
  • Role == ROLE_AutonomousProxy - means the actor is the local player. Autonomous proxies have special logic built in for client-side prediction (rather than simulation) of movement. They can execute any script functions on the client; and they can enter any state. This situation is only seen in network clients, never for network servers or single-player games.
  • Role == ROLE_Authority - means this machine has absolute, authoritative control over the Actor.
This is the case for all actors in single-player games. They can execute any script functions; and they can enter any state.

This is the case for all Actors on a server.

On a client, this is the case for Actors that were locally spawned by the client, such as gratuitous special effects which are done client-side in order to reduce bandwidth usage.

On the server side, all Actors have Role == ROLE_Authority, and RemoteRole set to one of the proxy types. On the client side, the Role and RemoteRole are always the exactly reversed relative to the server's value. This is as expected from the meaning of Role and RemoteRole.

Most of the meaning of the ENetRole values is defined by the replication statements in the UnrealScript classes such as Actor and PlayerPawn. Here are several examples of how the replication statements define the meanings of the various role values:

ALERT! Note: These examples are strictly relevant for Unreal Engine 1 and Unreal Engine 2; but the core concepts stay the same for Unreal Engine 3.

  • The Actor.AmbientSound variable is sent from the server to clients because of this replication definition in the Actor class:
    • if(Role == ROLE_Authority) AmbientSound;
  • The Actor.AnimSequence variable is sent from the server to clients, but only for actors rendered as meshes, because of this replication definition in the Actor class:
    • if(DrawType == DT_Mesh && RemoteRole <= ROLE_SimulatedProxy) AnimSequence;
  • The server sends clients the Velocity of all simulated proxies when they are initially spawned and all moving brushes because of this replication definition in the Actor class:
    • if((RemoteRole == ROLE_SimulatedProxy && (bNetInitial || bSimulatedPawn)) || bIsMover) Velocity;

By studying the replication statements in all of the UnrealScript classes, you can understand the inner workings of all of the roles. There is really very little "behind-the-scenes _magic_" going on with respect to replication: At a low C++ level, the engine provides a basic mechanism for replicating actors, function calls, and variables. At the high UnrealScript level, the meanings of the various network roles are defined by specifying what variables and functions should be replicated based on the various roles. So, the meaning of the roles is almost self-defining in UnrealScript, with the exception of a small amount of behind-the-scenes C++ logic which conditionally updates physics and animation for simulated proxies.

Relevancy

Definition

An Unreal Level can be huge, and at any time a player can only see a small fraction of the Actors in that level. Most of the other Actors in the level aren't visible, aren't audible, and have no significant effect on the player. The set of Actors that a server deems are visible to or capable of affecting a client are deemed the relevant set of Actors for that client. A significant bandwidth optimization in Unreal's network code is that the server only tells clients about Actors in that client's relevant set.

Unreal applies the following rules (in order) in determining the relevant set of Actors for a player:

  • If the Actor's RemoteRole is ROLE_None, then it is not relevant.
  • If the Actor is attached to the skeleton of another actor, then its relevancy is determined by the relevancy of its base.
  • If the Actor has bAlwaysRelevant, then it is relevant.
  • If the Actor has bOnlyRelevantToOwner set to true (used for Inventory), it is only potentially relevant to the client whose player owns that Actor.
  • If the Actor is owned by the player (Owner == Player), then it is relevant.
  • If the Actor is hidden (bHidden = true) and it doesn't collide (bBlockPlayers = false) and it doesn't have an ambient sound (AmbientSound == None) then the actor is not relevant.
  • If the Actor is visible according to a line-of-sight check between the actor's Location and the player's Location, then it is relevant.
  • If the Actor was visible less than 2 to 10 seconds ago (the exact number varies because of some performance optimizations), then it is relevant.

Note that bStatic and bNoDelete Actors (which remain on the client) can also be replicated.

These rules are designed to give a good approximation of the set of Actors which really can affect a player. Of course, it is imperfect: the line-of-sight check can sometimes give a false negative with large Actors (though we use some heuristics to help it out), it doesn't account for sound occlusion of ambient sounds, and so on. However, the approximation is such that its error is overwhelmed by the error inherent in a network environment with such latency and packet loss characteristics as the Internet.

Prioritization

In deathmatch games on modem-based Internet connections, there is almost never enough bandwidth available for the server to tell each client everything it desires to know about the game state, Unreal uses a load-balancing technique that prioritizes all actors, and gives each one a fair share of the bandwidth based on how important it is to gameplay.

Each Actor has a floating point variable called NetPriority. The higher the number, the more bandwidth that Actor receives relative to others. An Actor with a priority of 2.0 will be updated exactly twice as frequently as an Actor with priority 1.0. The only thing that matters with priorities is their ratio; so obviously you can't improve Unreal's network performance by increasing all of the priorities. Some of the values of NetPriority we have assigned in our performance-tuning are:

  • Actor - 1.0
  • Pawns - 2.0
  • PlayerController - 3.0
  • Projectiles - 2.5
  • Inventory - 1.4
  • Vehicule - 3.0

Replication


Replication Overview

The network code is based on three primitive, low-level replication operations for communicating information about the game state between the the server and clients:

Actor replication

The server identifies the set of "relevant" Actors for each client (Actors which are either visible to the client or are likely to somehow affect the client's view or movement instantaneously), and tells the client to create and maintain a "replicated" copy of that Actor. While the server always has the authoritative version of that Actor, at any time many clients might have approximate, replicated versions of that Actor.

When a replicated Actor is spawned on a client, only Location and Rotation (valid if bNetInitialRotation is set to true) are valid during PreBeginPlay() and PostBeginPlay(). Replicated Actors can only be destroyed because the server closes their replication channel, with the exception being if the Actor properties bNetTemporary and bTearOff have been set to true.

Actor property replication is reliable. This means that the property of the client version of the Actor will eventually reflect the value on the server, not that all property value changes will be replicated. In any case, Actor properties are only replicated from the server to the client; and such properties are replicated only if they are included in the replication definition of the Actor class which defines that property.

The replication definition specifies replication conditions, which describe when and if to replicate a given property to the client currently being considered. Even if an Actor is relevant, not all of its properties are replicated. Careful specification of replication conditions can substantially reduce bandwidth use.

There are three Actor properties which are only valid during replication, and change values depending on the client for which the server is determining replication:

  1. bNetDirty is true if any replicated properties have been changed by UnrealScript. This is used as an optimization (no need to check UnrealScript replication conditions, or to check whether properties which are only modified in script have been changed if bNetDirty is false). Don’t use bNetDirty to manage replication of frequently updated properties!
  2. bNetInitial remains true until initial replication of all replicated Actor properties is complete.
  3. bNetOwner is true if the top owner of the Actor is the PlayerController owned by the current client.

Variable replication

Actor variables that describe aspects of the game state which are important to clients can be "replicated". That is, whenever the value of the variable changes on the server side, the server sends the client the updated value. The variable may have changed on client side too - in which case the new value will overwrite it. Variable replication conditions are specified in the Replication{} block of an UnrealScript class.

Function call replication

A function that is called on the server in a network game can be routed to the remote client rather than executed locally. Alternatively, a function called on the client side may be routed to the server, rather than called locally. Function replication is specified in the function definition by the server, client, reliable and unreliable keywords.

Example

To give a concrete example, consider the case where you are a client in a network game. You see two opponents running towards you, shooting at you, and you hear their shots. Since all of the game state is being maintained on the server rather than on your machine, why can you see and hear those things happening?

  • You can see the opponents because the server has recognized that the opponents are relevant to you (i.e. they are visible) and the server is currently replicating those Actors to you. Thus, you (the client) have a local copy of those two player Actors who are running after you.
  • You can see that the opponents are running toward you because the server is replicating their Location and Velocity properties to you. Between Location updates from the server, your client simulates the movement of the opponents locally.
  • You can hear their gunshots because the server is replicating the ClientHearSound function to you. The ClientHearSound function is called for a PlayerPawn whenever the server determines that the PlayerPawn hears a sound.

So, by this point, the low-level mechanisms by which Unreal multiplayer games operate should be clear. The server is updating the game state and making all of the big game decisions. The server is replicating some Actors to clients. The server is replicating some variables to clients. And, the server is replicating some function calls to clients.

It should also be clear that not all Actors need to be replicated. For example, if an Actor is halfway across the level and way out of your sight, you don't need to waste bandwidth sending updates about it. Also, all variables don't need to be updated. For example, the variables that the server uses to make AI decisions don't need to be sent to clients; the clients only need to know about their display variables, animation variables and physics variables. Also, most functions executed on the server shouldn't be replicated. Only the function calls that result in the client seeing or hearing something need to be replicated. So, in all, the server contains a huge amount of data, and only a small fraction of it matters to the client - the ones which affect things the player sees, hears or feels.

Thus, the next logical question is, "How does the Unreal engine know what Actors, variables, and function calls need to replicated?"

The answer is, the programmer who writes a script for an actor is responsible for determining what variables and functions, in that script, need to be replicated. And, he is responsible for writing a little piece of code called a "replication statement", in that script, to tell the Unreal Engine what needs to be replicated under what conditions. For a real-world example, consider some of the things defined in the Actor class.

ALERT! Note: Not all of these variables are in Actor.uc in Unreal Engine 3 (they were present in Unreal Engine 1 and 2); but the core concepts are still valid.

  • The Location variable (a vector) contains the Actor's location. The server is responsible for maintaining the location, so the server needs to send that to clients. So the replication condition basically says "Replicate this if I am the server".
  • The Mesh variable (an object reference) references the mesh that should be rendered for the actor. The server needs to send that to clients, but it only needs to be sent if the Actor is being rendered as a mesh, i.e. if the Actor's DrawType is DT_Mesh. So the replication condition basically says "Replicate this if I am the server and the DrawType is DT_Mesh.
  • In the PlayerPawn class, there are a bunch of boolean variables that define key presses and button presses, such as bFire and bJump. These are generated on the client side (where the input happens), and the server needs to know about them. So the replication condition basically says "Replicate this if I am a client".
  • In the PlayerController class, there is a ClientHearSound function that tells the player that he or she hears a sound. It's called on the server but, of course the sound needs be heard by the actual person playing the game, who is on the client side. So the replication condition for this function might be "Replicate this if I am the server".

From the above examples, several things should be apparent. First of all, every variable and function that might be replicated needs to have a "replication condition" attached to it, that is, an expression which evaluates to true or false, depending on whether the thing needs to be replicated. Second, these replication conditions need to be two-way: the server needs to be able to replicate variables and functions to the client, and the client needs to be able to replicate them to the server. Third, these "replication conditions" can be complex, such as "Replicate this if I'm the server and this is the first time this Actor is being replicated across the network."

Therefore, we need a general-purpose way of expressing (complex) conditions under which variables and functions should be replicated. What is the best way to express these conditions? We looked at all of the options, and concluded that UnrealScript, which is already a very powerful language for authoring classes, variables, and code, would be a perfect tool for writing replication conditions.

UnrealScript: The replication statement

In UnrealScript, every class can have one replication statement. The replication statement contains one or more replication definitions. Each replication definition consist of a replication condition (a statement that evaluates to true or false), and a list of one or more variables to which the condition applies.

The replication statement in a class may only refer to variables defined in that class. This way, if the Actor class contains a variable DrawType, then you know where to look for its replication condition: it can only reside there in the Actor class.

It's valid for a class to not contain a replication statement; this simply means that the class doesn't define any new variables or functions that need to be replicated. In fact, most classes do not need replication statements because most of the "interesting" variables that affect display are defined in the Actor class, and are only modified by subclasses.

If you define a new variable in a class, but you don't list it in a replication definition, that means that your variable is absolutely never replicated. This is the norm; most variables don't need to be replicated.

Here is an example of the UnrealScript syntax for the replication statement, enclosed by a replication {} block. This is taken from the PlayerReplicationInfo class:

PlayerReplicationInfo.uc
replication
{
   // Things the server should send to the client.
   if ( bNetDirty && (Role == Role_Authority) )
      Score, Deaths, bHasFlag, PlayerLocationHint,
      PlayerName, Team, TeamID, bIsFemale, bAdmin,
      bIsSpectator, bOnlySpectator, bWaitingPlayer, bReadyToPlay,
      StartTime, bOutOfLives, UniqueId;
   if ( bNetDirty && (Role == Role_Authority) && !bNetOwner )
      PacketLoss, Ping;
   if ( bNetInitial && (Role == Role_Authority) )
      PlayerID, bBot;
}

Reliable vs unreliable

Functions replicated with the unreliable keyword are not guaranteed to reach the other party and, if they do reach the other party, they may be received out-of-order. The only things which can prevent an unreliable function from being received are network packet-loss, and bandwidth saturation. So, you need to understand the odds, which we will grossly approximate here. The results vary wildly among different types of network, so we can make no guarantees:

  • LAN - In a LAN game, we guesstimate that unreliable data is received successfully approximately 99% of the time. However, in the course of a game, hundreds of thousands of things are replicated, so you can be sure that some unreliable data will be lost. Therefore, even if you're aiming for LAN performance only, your code needs to handle your unreliably replicated functions gracefully in the case that they are lost over the wire.
  • Internet - In a typical low quality 28.8K ISP connection, unreliable functions is generally received 90%-95% of the time. In other words, it is very frequently lost.

To get a better feeling for the tradeoffs between reliable and unreliable functions, check out the replication statements in the Unreal scripts, and gauge their importance vs. the reliability decision we made. Be cautious and use reliable functions only when absolutely necessary.

Variables are always reliable

Variables are always guaranteed to reach the other party eventually, even under packet-loss and bandwidth saturation conditions. Changes in such variables are not guaranteed to reach the other party in the same order in which they were sent. Also, while the value of a variable will eventually be synchronized, not every change in its value may be replicated.

Replication Conditions

Here is a simple example of a replication condition within a class's script:

Pawn.uc
replication
{
   if( Role==ROLE_Authority )
      Weapon;
}

This replication condition, translated into English, is "If the value of this Actor's Role variable is equal to ROLE_Authority, then this Actor's Weapon variable should be replicated to all clients for whom this Actor is relevant".

A replication condition may be any expression that evaluates to a value of true or false (that is, a boolean expression). So, any expression you can write in UnrealScript will do, including comparing variables; calling functions; and combining conditions using the boolean !, &&, ||, and ^^ operators.

An Actor's Role variable generally describes how much control the local machine has over the Actor. ROLE_Authority means "this machine is the server, so it is completely authoritative over the proxy Actor". ROLE_SimulatedProxy means "this machine is a client, and it should simulate (predict) the physics of the Actor". Role is described in more detail in a later section, but the quick summary is this:

  • if (Role == ROLE_Authority) - means "If I am the server, I should replicate this to clients".
  • if (Role < ROLE_Authority) - means "If I am a client, I should replicate this to the server".

The following variables are very often used in replication statements because of their high utility:

  • bIsPlayer - Whether this Actor is a player. True for players, false for all other Actors.
  • bNetOwner - Whether this Actor is owned by the client for whom the replication condition is being evaluated. For example, say "Fred" is holding a DispersionPistol, and "Bob" isn't holding any weapon. When the DispersionPistol is being replicated to "Fred", its bNetOwner variable will be True (because "Fred" owns the weapon). When it is being replicated to "Bob", its bNetOwner variable will be false (because "Bob" does not own the weapon).
  • bNetInitial - Valid only on the server side (i.e. if Role = ROLE_Authority). Indicates whether this Actor is being replicated to the client for the first time. This is useful for clients with Role = ROLE_SimulatedProxy, because it enables the server to send their location and velocity just once, with the client subsequently predicting it.

Replication condition guidelines

Since variables are typically replicated one-way (either from the client to the server, or from the server to the client, but never both), all replication conditions generally start with a comparison of Role or RemoteRole: for example, if(Role == ROLE_Authority) or if(RemoteRole < ROLE_SimulatedProxy). If a replication condition doesn't contain a comparison of Role or RemoteRole, there is probably something wrong with it.

Replication conditions are evaluated very, very frequently on the server during network play. Keep them as simple as possible.

While replication conditions are allowed to call functions, try to avoid that because it could be a big slowdown.

Replication conditions shouldn't have any side-effects, because the network code may choose to call them at any time including times when you don't expect. For example if you do something like if(Counter++ > 10) ..., good luck trying to figure out what is going to happen!

Variable Replication

Update mechanism

After every tick, the server checks out all Actors in its relevant set. All of their replicated variables are examined to see if they have changed since the previous update, and the variables' replication conditions are evaluated to see if the variables need to be sent. As long as there is bandwidth available in the connection, those variables are sent across the network to the other machine.

Thus, the client receives updates of the important events that are happening in the world, which are visible or audible to that client. The key points to remember about variable replication are:

  • Variable replication occurs only after a tick completes. Therefore, if in the duration of a tick, a variable changes to a new value, and then it changes back to its original value, then that variable will not be replicated. Thus, clients only hear about the state of the server's Actor's variables after its tick completes; the state of the variables during the tick is invisible to the client.
  • Variables are only replicated when they change, relative to their previously known value.
  • Variables for an Actor are only replicated to a client when they are in the client's relevant set. Thus, the client does not have accurate variables for Actors that are not in its relevant set.

UnrealScript has no concept of global variables; so the only variables that can be replicated are instance variables belonging to an Actor.

Variable type notes

  • Vectors and Rotators - To improve bandwidth efficiency, Unreal quantizes the values of vectors and rotators. The X, Y, Z components of vectors are converted to 16-bit signed integers before being sent, thus any fractional values or values out of the range -32768...32767 are lost. The Pitch, Yaw, Roll components of rotators are converted to bytes, of the form (Pitch >> 8) & 255. So, you need to be careful with vectors and rotators. If you absolutely must have full precision, then use int or float variables for the individual components; all other data types are sent with their complete precision.
  • General structs - These are replicated by sending all of their components. A struct is sent as an "all or nothing" type of thing.
  • Arrays of variables can be replicated, but only of the size of the array (in bytes) is less than 448 bytes. Dynamic arrays cannot be replicated.
    • Arrays are replicated efficiently; if a single element of a large array changes, only that element is sent.

ALERT! Note: Replication rules can change, and some have priority over others. For example, a static array inside a struct is always sent in full as it is inside a struct!

Actor Properties

Actor property replication is reliable. This means that the property of the client version of the Actor will eventually reflect the value on the server, not that all property value changes will be replicated.

  • Properties are only replicated from the server to the client.
  • Properties are replicated only if they are included in the replication definition of the class which defines that property.

Function Call Replication

Remote Routing Mechanism

When an UnrealScript function is called during a network game, and that function has a replication keyword, the keyword is evaluated and execution progresses as follows:

The function call is sent to the machine on the other side of the network connection for execution. In other words, the function's name, and all of its parameters, are crammed together into a packet of data, and transmitted to the other machine for later execution. When this occurs, the function returns immediately and execution continues on. If the function were declared to return a value, then its return value is set to zero (or the equivalent of zero in some other type, i.e. 0,0,0 for vectors, None for objects, etc). Any out parameters are left unaffected. In other words, UnrealScript never sits around waiting for a replicated function call to complete, so it can never deadlock. Rather, replicated function calls are sent off for the remote machine to execute, and the local code continues executing.

Unlike replicated variables, a function call on an Actor can only be replicated from the server to the client (player) who owns that Actor. So, replicated functions are only useful in subclasses of PlayerController (i.e. players, who own themselves), Pawn (i.e. player avatars that are owned by a Controller that is controlling them), and subclasses of Inventory (i.e. weapons and pickup items, who are owned by the player who is currently carrying them). That is to say, a function call can only be replicated to one Actor (the player who owns it); they do not multicast.

If a function marked with the server keyword is called on a client, it will be replicated to the server. Conversely, when a function marked with the client keyword is called on the server, it will be replicated to the client which owns that Actor.

Unlike replicated variables, replicated function calls are sent to the remote machine immediately when they are called, and they are always replicated regardless of bandwidth. Thus, it is possible to flood the available bandwidth if you make too many replicated function calls. Replicated functions suck away however much bandwidth is available, and then whatever bandwidth is left over is used for replicating variables. Therefore, if you flood the connection with replicated functions, you can starve the replication of variables, which visually results in not seeing other Actors update, or seeing them update in an extremely choppy motion.

In UnrealScript, there are no global functions, so there is no concept of "replicated global functions". A function is always called in the context of a particular Actor.

Replicated function calls vs replicated variables

Too many replicated functions can flood the available bandwidth (because the are always replicated, regardless of available bandwidth), where as replicated variables automatically are throttled and parceled out according to bandwidth available.

Function calls are replicated only during UnrealScript execution when they are actually called, whereas variables are replicated only at the end of the current tick when no script code is executing.

Function calls on an actor are only replicated to the client who owns that actor, whereas an Actor's variables are replicated to all clients for whom that actor is relevant.

Simulated functions and states


On the client side, many Actors exist in the form of "proxies", meaning approximate copies of Actors created by the server, and sent to the client to provide a visually and aurally reasonable approximation of what the client sees and hears during game play.

On the client, these proxy Actors are often moving around using client-side physics and affecting the environment, so at any time their functions can potentially be called. For example, a simulated proxy TarydiumShard projectile might run into an autonomous proxy Tree Actor. When Actors collide, Unreal Engine attempts to call their Touch() functions to notify them of the collision. Depending on context, the client desires to execute some of these functions calls, but ignore others. For example, a Skaarj's Bump() function should not be called on the client side, because his Bump() function attempts to carry out game play logic, and game play logic should only occur on the server. So, the Skaarj's Bump() function should not be called. However, a TarydiumShard projectile's Touch() function should be called, because it stops the physics and spawns a client-side special effect Actor.

UnrealScript functions can optionally be declared with the simulated keyword to give programmers fine-grained control over which functions should be executed on proxy actors. For proxy actors (that is, actors with Role == ROLE_SimulatedProxy), only functions declared with the simulated keyword are called. All other functions are skipped.

Here is an example of a typical simulated function:

simulated function HitWall( vector HitNormal, actor Wall )
{
  SetPhysics(PHYS_None);
  MakeNoise(0.3);
  PlaySound(ImpactSound);
  PlayAnim('Hit');
}

So, simulated means "this function should always be executed for proxy Actors".

ALERT! Note: Make sure that subclass implementations of simulated functions also have the simulated keyword in their definition! The Unrealscript Compiler will warn you when this happens.

Simulated states are similar to simulated functions.

Replication Patterns


The goals for common replication patterns in the Unreal Engine and games shipped by Epic are:

Minimizing Server CPU Utilization

  • Minimize cost of replicating actors.
    • Minimize number of potentially replicated Actors (those with RemoteRole != ROLE_None).
    • Minimize number of Actors that need to be checked for relevancy per client any given tick.
    • Minimize number of actually relevant Actors per client any given tick.
    • Minimize number of replicated properties that need to be checked per replicated actor per client any given tick.
    • Avoid unnecessarily setting bNetDirty.
  • Minimize Actor tick cost
    • Avoid spawning Actors not needed on server (particle effects, etc.).
    • Avoid executing code if it has no game play relevance.
  • Minimize cost of processing received replicated functions
    • Minimize number of functions received and processing required.

As the number of players increases, the cost of replicating Actors becomes the dominant part of server execution time, since it tends to increase geometrically rather than linearly with the number of players (since the number of potentially replicated Actors tends to be proportional to the number of players).

  • Minimizing Bandwidth Use
    • The number of relevant Actors per client
    • The frequency of property updates
    • The number of packets sent

Unsuppress DevNetTraffic to see logging of all replicated Actors and properties. The console command Stat Net is also useful. It is also useful to use the Network Profiler to examine packets that Unreal Engine is sending and receiving.

Minimizing Perceived Latency

Have clients predict behavior of client owned actors based on player inputs; simulate this behavior before receiving confirmation from the server (and correcting if necessary). We use this model for Pawn movement and Weapon handling, but not for Vehicles, as the complexity of saving and replaying the physics simulation outweighs the benefit of reduced latency for vehicles handling, where typical internet response latencies aren’t that different from typical real world vehicle control response latencies.

ReplicationInfo classes

ReplicationInfo classes have bAlwaysRelevant set to true. Server performance can be improved by setting a low NetUpdateFrequency. Whenever a replicated property changes, explicitly change NetUpdateTime to force replication. Server performance can also be improved by setting bSkipActorPropertyReplication and bOnlyDirtyReplication to true.

Using ReplicatedEvent()

When properties marked with the Repnotify keyword are replicated, the ReplicatedEvent() event is called with the name of the modified property as a parameter. This system provides an efficient way to initialize multiple properties or components based on a single replicated property update. For example, when Vehicle.bDriving changes this is picked up by ReplicatedEvent() which then calls DrivingStatusChanged() to be called. We use this in UT to turn the engine sound and other client-side effects on or off. Similarly, when the team property is received by a UTCarriedObject, it updates client-side effects, including changing component properties like material applied to mesh, or dynamic light color.

When the PlayerReplicationInfo owned by the local PlayerController on the client has its team property or owner property updated, it calls NotifyLocalPlayerTeamReceived() on all actors.

It can also be used to delay the execution of initialization code until all required properties have been replicated. Note that there is no replicated event if property doesn’t change from default value, so you'll need to make sure actor gets properly initialized in this case.

The WorldInfo class

Every game world instance in a networked game has a NetMode. The WorldInfo class defines the ENetMode enumeration and an associated NetMode variable as follows:

var enum ENetMode
{
  NM_Standalone,        // Standalone game.
  NM_DedicatedServer,   // Dedicated server, no local client.
  NM_ListenServer,      // Listen server.
  NM_Client             // Client only, no local server.
} NetMode;

The NetMode property is often used to control what code will execute on different game instance types.

The GameInfo class

The GameInfo class implements the game rules. A server (both dedicated and single-player) has one GameInfo subclass, accessible in UnrealScript as WorldInfo.Game. For each game type in Unreal, there is a special GameInfo subclass. For example, some existing classes are UTGame, UTDeathmatch, UTTeamGame.

A client in a network game does not have a GameInfo. That is, WorldInfo.Game == None on the client side. Clients should not be expected to have a GameInfo because the server implements all of the game play rules, and the generality of the code calls for the client not knowing what the game rules are.

GameInfo implements a broad set of functionality, such as recognizing players coming and going, assigning credit for kills, determining whether weapons should respawn, and so on. Here, we will only look at the GameInfo functions which are directly related to network programming.

InitGame

event InitGame(string Options, out string ErrorMessage);

Called when the server (either in network play or single-player) is first started up. This gives the server the opportunity to parse the startup URL options. For example, if the server was started with "Unreal.exe MyLevel.unr?game=unreali.teamgame", the Options string is "?game=unreali.teamgame". If Error is set to a non-empty string, the game fails with a critical error.

PreLogin

event PreLogin(string Options, string Address, out string ErrorMessage, out string FailCode);

Called immediately before a network client is logged in. This gives the server an opportunity to reject the player. This is where the server should validate the player's password (if any), enforce the player limit, and so on.

Login

event PlayerController Login(string Portal, string Options, out string ErrorMessage);

The Login() function is always called after a call to PreLogin() which does not return an error string. It is responsible for spawning the player, using the parameters in the Options string. If successful, it should return the PlayerController Actor it spawned. Login() and PostLogin() are also used to create the PlayerController Actor in a standalone game.

If the Login() function returns None indicating that the login failed, then it should set Error to a string describing the error. Failing a Login() should be used only as a last resort. If you are going to fail a login, it is more efficient to fail it in PreLogin() rather than Login().

PostLogin

event PostLogin(PlayerController NewPlayer);

The PostLogin() function is called after a successful login. This is the first point at which replicated functions can be called.

Player Movement and Prediction

Overview

If a pure client-server model were used in Unreal, player movement would be very laggy. On a connection with 300 msec ping, when you push the forward key, you wouldn't see yourself move for 300 msec. When you pushed the mouse left, you wouldn't see yourself turn for 300 msec. This would be really frustrating.

To eliminate client-movement lag, Unreal uses a prediction scheme similar to that pioneered by QuakeWorld. It must be mentioned that the player prediction scheme is implemented entirely in UnrealScript. It is a high-level feature implemented in the PlayerController class, rather than a feature of the network code: Unreal's client movement prediction is layered entirely on the general-purpose replication features of the network code.

Inner workings

You can see exactly how Unreal's player prediction works by examining the PlayerController script. Since the code is somewhat complex, its workings are briefly described here.

The approach can best be described as a lock-step predictor/corrector algorithm. The client takes his input (joystick, mouse, keyboard) and physical forces (gravity, buoyancy, zone velocity) into account and describes his movement as a 3D acceleration vector. The client sends this acceleration along with various input related information and his current time stamp (the current value of WorldInfo.TimeSeconds on the client side) to the server in a replicated ServerMove function call:

server function ServerMove(float TimeStamp, vector InAccel, vector ClientLoc, byte MoveFlags, byte ClientRoll, int View)

Then the client calls his MoveAutonomous() to perform this same identical movement locally, and he stores this movement in a linked list of remembered movements using the SavedMove class. As you can see, if the client never heard anything back from the server, the client would be able to move around with zero lag just as in a single-player game.

When the server receives a ServerMove() function call (replicated across the network), the server carries out the exact same movement on the server immediately. It deduces the movement's DeltaTime from the current ServerMove's TimeStamp and the previous ones. In this way, the server is carrying out the same basic movement logic as the client. However, the server might see things slightly different than the client. For example, if there's a monster running around, the client might have thought it was in a different position than the server (because the client is only in rough approximate sync with the server). Thus, the client and the server might disagree about how far the client actually moved as a result of the ServerMove() call. At any rate, the server is authoritative, and he is completely responsible for determining the client's position. Once the server has processed the client's ServerMove() call, it calls the client's ClientAdjustPosition() function which is replicated across the network to the client:

client function ClientAdjustPosition(float TimeStamp, name newState, EPhysics newPhysics, float NewLocX, float NewLocY, float NewLocZ, float NewVelX, float NewVelY, float NewVelZ, Actor NewBase)

Now, when the client receives a ClientAdjustPosition() call, he must respect the server's authority over his position. So, the client sets his exact location and velocity to that specified by the ClientAdjustPosition() call. However, the position the server specifies in ClientAdjustPosition() reflects the client's actual position at some time in the past. But, the client wants to predict where he is supposed to be at the present moment. So, now the client goes through all of the SavedMove's in his linked list. All moves earlier than the ClientAdjustPosition() call's TimeStamp are discarded. All moves that occurred after TimeStamp are then re-run by looping through them and calling MoveAutonomous() for each one.

This way, at any point in time, the client is always predicting ahead of what the server has told him by an amount of time equal to half his ping time. And, his local movement is not at all lagged.

Advantages

This approach is purely predictive, and it gives one the best of both worlds: In all cases, the server remains completely authoritative. Nearly all the time, the client movement simulation exactly mirrors the client movement carried out by the server, so the client's position is seldom corrected. Only in the rare case, such as a player getting hit by a rocket, or bumping into an enemy, will the client's location need to be corrected.

Movement Pattern

The following diagrams help illustrate the movement pattern on the server and client, including error adjustment.

ServerClient
ReplicateMove()
Called instead of ProcessMove(). Performs pawn physics update based on player inputs, saves (in PlayerController SavedMoves) and replicates results. SavedMove can be subclasses to save game specific movement inputs and results. ReplicateMove() also attempts to combine replicated moves to save upstream bandwidth and improve server performance.
ServerMove()<-CallServerMove()
Performs pawn physics update based on received inputs, and compares results to results sent by the client. Note that movement update is done based on the client's clock. If client has accumulated a significant position error, request correction. Else, request good move ack.Sends one or two current moves (depending on frame rate and available bandwidth) with client clock timestamps. Sending two moves at a time saves bandwidth, but increases latency for corrections. Also possibly calls OldServerMove() to re-send most recent "important" move in case of packet loss.
SendClientAdjustment()->ClientAckGoodMove()
Client response deferred to end of PlayerController tick to avoid sending multiple responses if multiple ServerMoves() were received that tick. If no error, ack good move.Update ping, based on round trip time of timestamp, and clear out savedmoves with earlier timestamp.

ServerClient
SendClientAdjustment()->ClientAdjustPosition()
Client response deferred to end of PlayerController tick to avoid sending multiple responses if multiple ServerMoves() were received that tick. If Error, call ClientAdjustPosition(). ack good move.Clear SavedMoves with timestamps prior to the correction timestamp. Move Pawn to the position specified by the server, and set bUpdatePosition.
ClientUpdatePosition()
Called from PlayerTick() when bUpdatePosition is true. Replays all outstanding SavedMoves to bring the Pawn back up to present client time.

Player State Synchronization

The PlayerController code assumes that the client and server always try to run the exact same state; ClientAdjustPosition() includes the state so that the client can be updated if it has entered a different state. In those cases where the server needs to change states but the client cannot simulate that state change itself, ClientGotoState() is used to force it into that state immediately. There is no support for handling/synchronizing UnrealScript's state stack functionality (PushState() / PopState()) and we recommend not using it for PlayerControllers.

Player Animation (Client-Side)

If animation has no gameplay relevance, it doesn’t need to be executed on the server at all. This can be controlled by SkeletalMeshComponent's bUpdateSkelWhenNotRendered and IgnoreControllersWhenNotRendered properties, as well as on a per-skeletal controller basis with SkelControlBase::bIgnoreWhenNotRendered. Client-side animation is driven by the inspection of Pawn state (physics, Pawn properties).

For animation-driven movement, the root bone motion is converted into acceleration/velocity, and that is what is being replicated. So the animation remains locked in place (relative to the Actor), and the root bone motion is transferred into acceleration/velocity moving the actor instead.

This is not heavier for the server/client than non root motion movement.

Dead Bodies

If bTearoff is true, this Actor is no longer replicated to new clients, and is torn off (becomes a ROLE_Authority) on clients to which it was being replicated. The TornOff() event is called when bTearOff is received. The default implementation calls PlayDying() on the dying Pawn.

Weapon Firing

Weapon firing is similar to the pattern for Player movement:
  • Client plays firing effect (sound, animation, muzzleflash) immediately on player input requesting fire, and calls ServerStartFire() and ServerStopFire() to request server firing.
    • Client has enough state information (ammo count, weapon timing state etc.) to correctly predict whether weapon can be fired, except in rare cases where client and server version of the relevant properties aren’t synchronized.
  • The Server spawns projectiles/damage the weapon (spawn the projectile). The projectile is replicated to the client.

Projectiles

This example is useful for simple, predictable projectiles:
  • bNetTemporary is set to true.
    • After initial replication, Actor channel is closed and Actor is never updated again. Actor will be destroyed by client.
    • Saves bandwidth, as well as server property replication tests.
  • bReplicateInstigator is set true.
    • So projectiles can interact properly with instigator.
  • Client-side effect spawning
    • Note that Actors spawned on client have ROLE_Authority on the client, do not exist on server or any other client.
    • These effects don’t need to be spawned on the server at all; nor replicated.

ALERT! Drawback: May erroneously hit or miss target if client simulation of target and/or projectile is off. Don’t use for one shot kill type projectiles for this reason.

Weapon Attachments

Avoid having interrelated groups of Actors all replicated, both because of performance and to minimize synchronization issues.

In Unreal Tournament, Weapons are only replicated to the owning client. Weapon Attachments are not replicated, but spawned client side and controlled through some replicated Pawn properties. Pawns replicate FlashCount and FiringMode, and UT Pawns replicate CurrentWeaponAttachmentClass. The ViewPitch property in Pawn is another example of this pattern in use.

Sounds

The function ClientHearSound() is called on every PlayerController for which the sound is audible. The function ClientCreateAudioComponent() is called on the Actor responsible for the sound. If the Actor doesn’t exist on the client, the sound will be played at the replicated position, with the audio component created by WorldInfo. The function ClientPlaySound() in PlayerController plays non-positional sound on client.

Try to play simulated sounds on the client whenever possible!

Physics

Replication

Physics simulation is run on both the client and the server. Updates are sent to the client from the server. The following struct is used to describe the physical state of a rigid body, and is replicated (as defined in Actor):

struct RigidBodyState
{
  var vector Position;
  var Quat Quaternion;
  var vector LinVel; // RBSTATE_LINVELSCALE times actual (precision reasons)
  var vector AngVel; // RBSTATE_ANGVELSCALE times actual (precision reasons)
  var int bNewData;
};

A struct used so that all properties change at the same time. The vectors are compressed to integer resolution, so that they are scaled before sending. Quats are compressed to only send 3 values; the 4th value is inferred from the other 3.

For physics replication, there are two types of correction:

  • Small corrections and object moving: 20% position adjust, 80% additional velocity to target
  • Large correction or object stopped: 100% position adjust

Simulation

The following scenarios describe physics simulation:
  • ROLE_SimulatedProxy Actor simulation
    • The client continuously updates the simulated actor position based on the received position and velocity.
    • If bUpdateSimulatedPosition is true, authoritative position updates are continuously sent from the server to the client (otherwise, no position updates are sent after the initial replication of the Actor).
  • Pawns on other clients
    • Unlike other Actors, simulated Pawns do not execute normal physics functions on the client. This means that physics events, like the Landed() event, are never called for pawns on non-owning clients.
    • The physics mode of the Pawn is inferred from its position, and the bSimulateGravity flag, and its predicted position is updated based on the replicated velocity.
      • The bSimGravityDisabled flag is set, temporarily turning off gravity simulation, if Pawn didn’t fit at the replicated position, and is in danger of falling through the world on the client.
  • PHYS_RigidBody Actors (Vehicles, KActors, etc.)
    • Both client and server simulate the objects, but the server sends authoritative updates to the client periodically (when the object is awake). The client then moves the object to match the server version
      • Attempts to do so smoothly, by altering velocity to bring about convergence in positions rather than snapping the position if the error is below an acceptable threshold
    • Use RigidBodyState struct for atomic replication, when all properties must be received in synch.

For Ragdoll physics, only the hip location is replicated. It is often possible to tear off completely and not replicate at all.

For Vehicles (PHYS_RigidBody Actors), there is the following network flow:

  1. Press key on client
  2. Send inputs (throttle, sterring, rise) to server - replicated function ServerDrive called
  3. Generate ouptut (OutputBrake, OutputGas, etc.); pack into replicated structs that can be sent to the client - ProcessCarInput() called on server
  4. Update vehicle on server and client; use outputs (OutputBrake, OutputGas, etc.) to apply forces/torques to wheels/vehicle - UpdateVehicle() called on client and server

Performance Tips


Optimization goal

The goal here is to maximize the amount of visibly important detail that is sent with a given bandwidth limit. With the bandwidth limit determined at runtime, your goal in writing scripts for Actors that are used in multiplayer games is to keep bandwidth use to a minimum. The techniques we use in our scripts include:

Use ROLE_SimulatedProxy and simulated movement whenever possible. For example, nearly all of the Unreal projectiles use ROLE_SimulatedProxy. The one exception is the Razorjack alt-fire blades, which the player can steer during gameplay, thus the server must continually update the position to clients.

For quick special effects, spawn the special effects Actors purely on the client side. For example, many of our projectiles use a simulated HitWall() function to spawn their effects on the client-side. Since these special effects are just decorative rather than affecting gameplay, there is no drawback to doing them completely on the client side.

When properties marked with the Repnotify keyword are replicated, the ReplicatedEvent() event is called with the name of the modified property as a parameter. See the Replication Patterns section for how this can be used to save network bandwidth.

Fine tune the default NetPriority for each class. Projectiles and players need to have high priorities, purely decorative effects can have lower priorities. The defaults that Unreal provides are good first-pass guesses, but you can always gain some improvement by fine-tuning them.

When an actor is first replicated to a client, all of its variables are initialized to their class default values. Subsequently, only variables that differ from their most recent known values are replicated. Thus, you should design your classes so that as many variables as possible are automatically set to their class defaults. For example, if an Actor always should have a LightBrightness value of 123, there are two ways you can make that happen: (1) set the class default's value of LightBrightness to 123, or (2) in the Actor's BeginPlay() function, initialize LightBrightness to 123. The first approach is more efficient, because the LightBrightness value will never need to be replicated. With the second approach, the LightBrightness needs to be replicated each time an Actor first becomes relevant to the client.

Also be aware of the following situations:

  • bNetInitial and bNetDirty don’t get cleared if the Actor reference can’t be serialized (because it isn't relevant to the client). This means that the server will continue to attempt the replicate the property, costing CPU cycles.

    Cheat Detection and Prevention

    We have encountered the following types of network-related cheats in Unreal Tournament:
    • Speedhack
      • Takes advantage of the fact we use the client’s clock for movement updates.
      • Built-in detection by verifying client and server clock don’t move at excessively different rates.
      • False positives with substantial packet loss
    • Aimbots - UnrealScript and external versions
    • Wall hacks and radars - UnrealScript and external versions

    Traffic Monitoring