Seeking advice on Server->Client world update algorithm

Discussion about everything. New games, 3d math, development tips...
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Seeking advice on Server->Client world update algorithm

Post by keless »

Hey, looking for someone who's already done an effective client-server game centered around world updates. I'm at the point of decision and I've come up with three ideas so far:

(forward note: im planning on having levels large enough that i only want to send updates to players within a leashed radius. WU = World Update)

1) while doing physics/logic updates on objects in the world, when a new item that needs to be sent to the client arises, seek a list of nearby players and send that event immediately to those players. PRO: searching for the list of players will probably already be spacially sorted at this point in the simulation so sending the WU data from this point wont require redunant processing to sort what goes to who. CON: this results in sending a series of partial-WU packets with different time stamps so that the client is recieving a constant stream of updates rather than one full WU frame (bad because its harder to keep in check, but good because it possibly uses bandwidth more efficiently)

2) reconize when a WU event is happening and serialize it to a spacially sorted place in memory (such as in an OctTree). after completing the current world update frame, for each player search the serialized WU's for WUs that apply to that player compiling them in a list, then sending that list as that player's own World Update. PRO: keeps the idea of a single WU frame which is easier to keep track of and synchronize CON: requires creating WU serialization and spacial-sort storage objects, requires more memory use for the stored updates, requires a 2nd pass (going over the stored WUs after the simulation is done), results in a bandwidth usage spike when all of the users' WU packets get sent at the same time

3) this one is kind of like (1) but tries to keep the PRO of (2): when a WU event is recognized, seek a list of players nearby and give them the event; but dont send their WU packets right away. Instead, each player has a BitStream opened at the start of a WU-frame, and as we find WU events that apply to them, write those events into that player's BitStream. When the whole WU frame simulation is finished, THEN send each player's WU packets. PRO: keeps the pros of both (1) and (2)-- ie: fast, but still sends everything in WU frames CON: redundant processing and memory use in that if player1 and player2 have similar WU's their bitstreams will be identical, also suffers from same bandwidth spike that (2) does

Right now I'm leaning towards the last one. Anyone got any insights?
a screen cap is worth 0x100000 DWORDS
Guest

Post by Guest »

3 looks fine to me.

Sending fewer (larger) packets means fewer packets that can go wrong.

The bit of optimisation you can achieve in the rare case that two players who are not in the exact same position have the same updates, is lost if you can't do a real multicast.

Do keep in mind that some packets _will_ be lost, so you need redundant packets.
It may be a good idea to send corrections/absolute positions for "nearby" objects every once in a while. Better than having errors accumulate.
Spintz
Posts: 1688
Joined: Thu Nov 04, 2004 3:25 pm

Post by Spintz »

If I understand this correctly, I would think you want to send the updates to the client immediately ( that way the server doesn't have to queue them up ) and let the client process them when it's able to.

Where you can, let the clients do the queueing, otherwise, the server's gonna run out of memory.
Image
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Post by keless »

@guest:
my idea of sending world updates was:
1) abs position/velocity/actionstate of moveable entities in radius
2) any special game-logic events

while having a seperate low-priority channel for chat


@Spintz

hrm, if I make sure to put (2) in the packet first, perhaps I can recognize when a user's WU packet is finished and send it at that point (for instance, if when running the simulation I move out of the visible OctTree node and thus im sure there's no more objects in the player's view). But this isnt reliable so worst-to-average case I wont see much benefit. However, I still think this is the way to go as speed is at a premium over memory
a screen cap is worth 0x100000 DWORDS
Spintz
Posts: 1688
Joined: Thu Nov 04, 2004 3:25 pm

Post by Spintz »

I think that's bad, not just memory, but you're doing a lot of processing to check WU's for a frame or whatever that should go to the client. IMO, dump it to the client. If it misses 1 update, it shouldn't kill it, as updates should come in frequently.

I would set up something like :

- Server sends WU's every 0.25 seconds or so.
- Server only sends a client WU's for a region he is in ( I wouldn't use radius, because then you need to worry about distance calculations, I would use sectors, like -

Code: Select all

a b c d
e f g h
i j k l
- if a client is in sector g, send him WU's for objects in b, c, d, f, g, h, j, k and l.
- if a client moves sectors, let's say from g to f, then he'll drop WU's for sector's d, h and l and will eventually get the WU's for a, e and i from the server.
- if a client happens to miss an update, don't worry about it, he'll get it with the next update ( this shouldn't happen often, if it is then there are other problems to worry about ).

There's many more "rules" that can be added to this list. I think the biggest advantage is sectors, because the server could process all WU's for an update, organize them by sectors and then send the appropriate WU's to whomever is in reach of the sectors. I would batch the data by sector though, not by client.
Image
mohaps
Posts: 248
Joined: Tue Jun 08, 2004 1:54 pm
Location: Shrewsbury MA
Contact:

Post by mohaps »

---
Saurav Mohapatra
author, artist and bona fide geek

web: http://www.mohaps.com
email: mohaps AT gmail DOT com
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Post by keless »

@spintz:
its pretty much the same system -- your 'sectors is my 'radius/spacial sort'

however, my 'spacial sort' will be based on whatever spacial partitioning system is used by the physics simulation (i leave it general because my system is supposed to have swappable components. ie: a BSP-level or an OctTree-level or what have you)

I'm planning on sending WUs in an interval of 200ms (about 1/5 instead of 2/4 a second) and im sending them RELIABLE_SEQUENCED so it can skip updates with out a problem (it'll just look jumpy cause of lag)

@mohaps:
I'm trying to avoid doing multiple passes. Rather than doing physics simulation and AI in runServerGame() and then after that making a second pass and generating WUs in sendUpdatesToClient(), I want to generate the updates while running the simulation, and sendUpdatesToClients() should simply fire off the already generated WUs.


@ myself:
I noted recently that while (3) from before has duplicated memory when clients share WU data, (2) has unused memory for all of the WUs generated that no one will hear about. If NPC/WU data out numbers the amount of players (3) is DEFINATELY better than (2).
a screen cap is worth 0x100000 DWORDS
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Post by keless »

Had an interesting AIM session with a friend of mine. After some discussion he tentatively agrees with my position on Method (3). We also got into some more technical specifics:
I'm still trying to justify 3 over 1-- here's my thought:
firstly, its simpler. each client recieves a world update frame that contains all of the entities' positions and velocities with a timestamp. It saves bandwidth because i have to send less redundant data (since any redundancy between individual WU-event packets is eliminated if I send them all in one big WU packet) And I could possibly do some security based on WU times (a lot easier than if i were doing streams)

how big is a WU packet on average?
only as big as the number of WU events a single player can see. lets assume it encompasses about 50 actors on average, with a maximum of 500.. 50 is probably even high, but lets say i've got all sorts of little un-important actors like birds or something just for the f*** of it. 'actors' includes stuff like doors/triggers/elevators/NPCs/magic effects. but by looking around my office right now and pretending people and things are actors (and ignoring all the books and crap on the desks) I count about 25 actors. so lets say we're considering multiple rooms and say average is more like 75 but max is still 500-ish

how big is an individual WU event?
im thinking something like pos(x,y,z)vel(x,y,z)anim(state1,state2,state3) so if i store location vectors as uint32 and anim states as uint8-- 216 bytes plus packet wrapper. however, since im using bit streams i get some compression in the following way: i send 1-bit infront of the velocity and another one infront of the anim states that acts as a boolean--- if its set to 0, then I assume a default value, so vel(0,0,0) since a lot of things arent moving, it saves bandwidth. so the same WU can actually be more like 100 - 220 bits

thats still a little high-- i wonder if i could compress the 3*32bit vector? perhaps I can send the player's current position (or some other 'local origin' ) at the front of the packet and have all of the WU vectors be relative to that origin and use int16s instead of int32s
a screen cap is worth 0x100000 DWORDS
mm765
Posts: 172
Joined: Fri May 28, 2004 10:12 am

Post by mm765 »

hmmm...
1. why send uint32 for pos if float has the same size (4 bytes) ?
2. dont!!! make the chat channel low prio...make it highest prio - if you encounter lag in a game and cant even chat with people that is the worst that can happen!
3. the number 1 on your list is not a valid option (ie it wouldnt work)
the most important thing is to prioritize the updates that happenend and only send those with the highest priorities. example:
(we are talking rpg here, right?) imagine a community meeting..500 players in a room...you just cant send all updates to all players (linespeed and traffic (cost)) so you just send the updates for X players that are nearest to you and interpolate the rest (X depending on the amount of bandwith you want to use).
but now if a player that doesnt belong to the nearest X attacks you you surely want to know...and you want to know the attackers position...so a current attacker is always priority 1 (after chat :) ) .
and to save bandwith you could even go so far and not send events of a certain low priority if they are outside a certain range even if they are the only events that happened.
so the focus should be on having a fixed maximum bandwith to use and then dertmine which events you send or not (and of course for that the fewer data an update needs the more events you can send with the same amount of bandwith used)
4. spintzs sectors arent the same as your radial sort at all because those sectors are fixed where your center (around which you want the radius) isnt which means that determining events that happened in a fixed sector is much easier than determining the events that happened within a certain radius around some object/player. but then here comes the bandwith limit again..even if you use sectors for fast detection of "valid" events you still need to sort them by distance. (its like frustum culling..use a coarse method first and then go into detail).
but theres more to it. regardless if you use sectors or radius - using the example from before- you need to make sure that an attack can only happen from within the sectors or the radius youre sending events from..otherwise the attack event wouldnt be sent to the attacked..
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Post by keless »

Firstly, thanks for the interest in this thread everyone, its helping me realize things before I figure them out the hard way!

@mm765:
about 4: forget about radial sort, i meant it as an example of spacial sorting in general-- it doesnt matter if its radial/octtree/bsp/something else, as long as its sorted in the most efficient way to occlude unneccesary events.

Now, you mention that I should be sorting events by priority and currently number 3 ignores this completely. 2 might be able to do it, but 1 can do it the easiest since you tell RakNet what priority (high/med/low) to give to a packet when you call send()! So 1 isnt invalid based on priority packet sorting.

I still prefer 3 the most-- write to a player's outgoing stream when nearby WU data is discovered, but dont send until the end of the frame. I think to adjust for the priority of updates I will simply have 3 bitstreams per player and write WU data to the appropriate stream. At the end of the frame update I will send each bitstream (if not empty) thru High/Med/Low priority channels.

As for high-priority chat: I slightly disagree-- certainly in a fast-paced action game you wouldnt want some moron(s)' chat flood to cause the server to lag and make you miss a kill. In an RPG this is perhaps slightly less the case. However I was already planning on sending chat and quest updates (things that dont relate to a world entity in the game) on a separate channel. So I can tweak what reliability/priority I give them.
a screen cap is worth 0x100000 DWORDS
mm765
Posts: 172
Joined: Fri May 28, 2004 10:12 am

Post by mm765 »

hmm...theres been a slight misunderstanding here im sorry i probably didnt explain it very well.
keless wrote: Now, you mention that I should be sorting events by priority and currently number 3 ignores this completely. 2 might be able to do it, but 1 can do it the easiest since you tell RakNet what priority (high/med/low) to give to a packet when you call send()! So 1 isnt invalid based on priority packet sorting.
I wasnt "voting" on any of your solutions but was saying that you probably need a different viewing angle on the topic :)
Even if you tell raknet the priority with which to send events, if you have 500 people in a room it will still try to send stuff and end up saturating your network-connection. So you have to take into account the possibility that you cant send everything that happens which is why you need to collect the events and then only send those with the highest priority and this is independent of the network-library you use. and this also invalidates 1 because if you dont collect events before you send them you cant prioritize them (this priority im talking about is not static but depends on whats happening as i thought the example above explains...let me try it again :))
500 people in one room...you send the information of the x nearest to the player (priority being distance). NOW someone starts to attack the player - priority shifts from distance to attacker, then distance as second criteria.
Another example would be some people moving around and another one opening a door. now the world-changing-event (door) has priority over the movement.
As for high-priority chat: I slightly disagree-- certainly in a fast-paced action game you wouldnt want some moron(s)' chat flood to cause the server to lag and make you miss a kill. In an RPG this is perhaps slightly less the case. However I was already planning on sending chat and quest updates (things that dont relate to a world entity in the game) on a separate channel. So I can tweak what reliability/priority I give them.
Also not what i meant :)
First: flooding has to be stopped at the client. no way you can let every player send what he wants. of course if someone figures out your network-protocol you also need to have a stop-gate on the server but first prio here is client side.
Furthermore i was still talking about the room with 500 people...your server-network.load will be much too high, lag starts. now if you cant even chat anymore you will be totally out of the game, thats why chat (which is low bandwith, can be compressed well) is important - if you can still talk to people you know that your connection isnt dead and youre only lagging. and yes, this even takes precedence over attacks/kills/whatever in this situation because the player cant do anything about it anyways (because of the lag).
of course this all is only valid for rpg-type of games. for a fast action game with a max number of 32 players id prolly go with 1 because its easiest to implement and needs the least interpolation.
there simply is no "one-size-fits-all" with networking (at least not if youre not talking lan-play only which is totally different because for lan-only you could for example go with tcp/ip instead of udp because of the reliable connections).
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Post by keless »

yeah, a little explaination of my goals right now: I'm really looking for a max of 100 players at a time on the server. Shooting for more like 4-8 or 16-32; I'm allowing clients to set up their own worlds and update them with their own quests and content. (this still brings up the issue of how to certify a server has quality content before you connect to it, but thats beyond the focus of this thread)

You're right: I didnt come to the realization that the server might actually just not send something at all. I can still fit that into #1 as easily as all the others. Assuming I have some sort of heuristic for deciding how important an individual WU-event is I simply ask the RakNet statistics how laggy the server currently is and drop low (or even med) priority events if its bad. Its up to the heuristic to realize who an attacker is and if thats imporant to this particular player, etc. etc. Personally, however, I'm aiming for a more dynamic environment where there isnt game logic about who's PvP flag is up and who's Aggro'd on who (to use MMO terms). I realize that this means I cant take advantage of some tricks so I'll end up using more bandwidth (which is why I'm not shooting for MMO status, I'm simply talking about a small MO game-- tinyMO if you like).

My goal is not to service millions of players in a giant virtual world, but to allow homegrown worlds filled with communities of people who probably all know each other outside of the game. There will be no factions hardcoded into the game, area of affect damage spells damage EVERYONE (including yourself and your allies), etc. etc. But again, I'm going beyond the scope of this thread. I just mean to fill you in a little on why I've made some of these technical decisions.

IrrlichtRPG is open source. I fully expect some hackers. My major defense against that is the fact that I want this game to be in grass roots communities, so hacking your friends is not a big interest. However, I still have to take a much more defensive position in the server realm. While putting flood checks and other things in the client is a must because it saves processing time on the server, I wont assume that all the clients actually do so-- so I have to give all authority to the server.

You say chat can be compressed well. How well? I'll definately need to do some tests, but looking at those figures I put in the last post do you think compressed text would be larger or smaller than a WU packet of say 100 entities, most of which arent moving (therefore 100bits rather than 200). Oh, and actually I said I could cut down on the vector sizes to use int16 rather than 32 by using relative coordinates, so make the WU-event size 50-150 bits range instead. So maybe 100 * ~100bits = 10,000bits (~125bytes) in an average world update for 100 entities.
a screen cap is worth 0x100000 DWORDS
keless
Posts: 805
Joined: Mon Dec 15, 2003 10:37 pm
Location: Los Angeles, California, USA

Post by keless »

I just realized something else. As I am sending only WU-events that are in the client's radius new entities will move in and out of that radius. The client will be loading these entities into and (after a timeout) out of memory. But how will he know what these things are if we arent sending enough information about them to be created on the fly in every single WU-event?

But I think I know the solution: I will already be sending entityIDs with each WU-event in a WU-packet. When a client realizes a new entity within its area, it will create a place-holder entity for it and request more specific data (what it needs to actually create a proper representation of that entity) from the server in a channel seperate from world updates. As an example, when you load into a heavily populated area in WoW, you notice shadows moving around with nothing above them before something actually pops in-- in my system that would be the client recieving WU-events for entities that it does not yet have the data on. I think in this current testing phase it would be fun to have the placeholder be a 'will-o-wisp'!

This means resource loading will have to be on a separate thread-- I think that could probably be built into the engine though.. ooh NI-KO!
a screen cap is worth 0x100000 DWORDS
mm765
Posts: 172
Joined: Fri May 28, 2004 10:12 am

Post by mm765 »

first of all you should really decide on how many players you want to support. there is a big difference between 4-8 and 100 because you have to always take the worst case into account which for 8 players means you have to send the position/velocity of 7 players to each client and with 100 players you have to send position/velocity of 99 players to each client (56 updates versus 9900 updates).
second (regarding your last post), how to handle new visible entities depends on how you want to handle your world. you said somewhere that you want the server to be in control (which is the a good idea imo). now you have to decide how much in control the server will be.if the server has 100% control and the clients are only "graphic terminals" you need to handle this on the server, send a "new player has entered visibility" message to the client and add the parameters of that new visible entity to the message (ie new player in visible radius, wears armor x, sword y). the other way would be the way you described BUT you have to be careful with that. you have to check every request from the client for reasonability for example if the client asks for information about some entity you should check if the entity is visible to the client (and then it might be easier to let the server handle that from the beginning) because otherwise its an invitation for exploits which, for example, counterstrike suffers from..wallhacks wouldnt even be possible if the client didnt know where players are as long as they are not visible (although its not that easy because if you only send entities that are visible you would see them later but thats a bit off-topic).
third, from what youre writing i am not sure if you were planning on sending the status of the whole world each time. if so - DONT - only send updates. you only need to send the state of the world for players that are entering the game and even then you only need to send the stuff going on around his initial position.
fourth, to see how good text-compression works, zip a textfile and take into account that zip adds a header so actual compression is even a bit better. otoh compression for shorter texts is not as good as for long texts.
BUT dont do it until youre absolutely certain you need to do it. if you want to know why not see http://c2.com/cgi/wiki?PrematureOptimization . same with using relative coordinates.
fifth, i havent seen any timestamps you want to send with your events - dont forget them :)

all in all..dont think of it as a world update but of synchronizing the world-state over different computers by using remote events.
Guest

Post by Guest »

With western charsets, there aren't a lot more than about 70 characters needed. (26 lowercase, 26 uppercase, space, some interpunction, 10 digits)
And they are not nearly used in equal proportions. So the simplest efficient text compression methods use the lowest number of bits for the most common character, and the highest number for the least common character.

But text compression for chat is ridiculous if you ask me. A fast typist can probably type something like 100 words per minute. So if you have 20 people typing 100 words a minute at the same time, at an average word length of, say 7 letters per word you get 20*8*100 = 16000 characters per minute, or 267 characters per second, which doesn't sound like much, even on 56k dialup.

Of course, if you have to pump back all that data to 100 separate clients it becomes a lot more, if you can't multicast. So no dialup connection for the server.

But I seriously doubt you're going to get more than one fast typer typing at the same time.
Post Reply