Question about *WorldUpdateThread*

[Leodagan]

I wonder if I'll need a second collection for Object still existing for "Client" but that server removed from "Player" Vincinity or if I could use some specific values like setting Object "Update" Time to long.MaxValue...

All this discussion lead me to think that the GameObjectUpdateArray is not enough to keep track of what Server AND Client do about Object Update...

[Stephen]

I wonder if I'll need a second collection for Object still existing for "Client" but that server removed from "Player" Vincinity or if I could use some specific values like setting Object "Update" Time to long.MaxValue...

All this discussion lead me to think that the GameObjectUpdateArray is not enough to keep track of what Server AND Client do about Object Update...

Well using the current code yes, a second collection to handle the objects data cached in client for the object went outofview is working and seems to be the only way because the obj in removed from GameObjectUpdateArray and you lose the data about this obj for the client.

I just tried to create a second specular GameObjectUpdateArray then adding the values tuple<ushort,ushort> where the worldupdatethread remove the obj because is outofview.

So i have for all clients a collection of objects that are potentially fake if deleted.

In the .RemoveObject method in region.cs , before the obj become NULL , i call a static method getting all playing clients and checking the new collection . If they have the object, i'll send to the client a SendObjectRemove and remove the value in the collection.

This is working fine ,i tested it with some players and in many situations with pets, players and every kind of objects.

But i still think there are better way to do that because just i'm new to this code

[Stephen]

Good one !

Yes I made the "out of view" update to make sure the player will get the last "packet" from this mob in a correct position.

This was made to prevent client keeping old mob speed info when getting out of view (example the mob sent its last update 8 seconds ago running towards you but in fact you outrunned him some time ago and he's returning to spawn)

mmm again , are you sure 100% of that? tested ?

Well we are talking about to send an objupdate if the obj is outofview that means the obj distance is > 5k loc

Another information i have now . My mobs ghost issues is limited for a short time.

Example: i do 10 pets with my thuergist, i move my character outofview and when the pet are removed , i move back where pet were spawned and my client displays for about 30 secs the pets , already removed but with the that last sendobjectupdate...after 30 secs my client-side cache is cleaned again.

So the issue seems to be that last update and restricted for 30 secs I still have to understand this mechanism.

I'll do some test I don't know maybe is correct to send very last update but after that make senses to me sending a objremove more than collect objs .

[Leodagan]

The last update is some experiment to reduce walking ghost, this is meant to try forcing client updating object more frequently at the limit of its view range, because objects in view range are already updated frequently when they change "heading" or when some animations are sent (spell cast, emote, attacks, a lot of "short state" packets)

This is really not the behavior of live server on handling updates, I understand this can bring some trouble as I implemented this quickly, but I hope we can keep this mechanism and prevent too much "erratic" NPC behavior (because live is still really buggy around this !)

About Client Cache, it's different depending on Game Version, around game.dll 1.109 the "client cache" is 30 seconds, in 1.110+ it's 15 seconds, and it may be shortened to 10 seconds in latest update 1.116~1.117

The WorldUpdateThread was made with the purpose of keeping Client Cache populated, trying to refresh client object at the expected rate or higher rate to prevent "blinking" or object disappearing, I simply reduce the cache time in there to match every game.dll so it's mostly around 10 seconds actually, 1.109 do support 10 sec refresh rate too, the trouble here is that this Update Thread wasn't made to keep track of what the client "may still have in cache"

We'll need some constants to match every game version cache time and either update the Current GameObjectUpdateArray Collection to hold some more data, or create an other Collection...

Your static method in Region.RemoveObject is also a solution I tried to implement locally, it sends remove packet to every player still having the object in cache (in Server Side Cache), but without using a "out of view collection"

the out of view collection would need to be updated when an object re-enter the player view range, and when we can guess that client removed this object from his cache, so we can't do "all the job" in the Region.RemoveObject Method, we have to change the Update Thread to keep this other collection clean (Clearing when changing Region ? Removing Object that didn't send update to player since "Cache Time" ? etc...)

An other change I tried locally is to change the Region nextObjectID update when removing object from Region, every object removed always register it's Own ID as the next Free ID this cause the most troubles you've seen even if it's nice to show WorldUpdateThread Wrong Implementations This nextObjectID update is only needed when the Region Object Array is Full to improve Next ID Lookup (this mean we still have to fix the Update Thread in case we're in a "Full" Region)

[Stephen]

Your static method in Region.RemoveObject is also a solution I tried to implement locally, it sends remove packet to every player still having the object in cache (in Server Side Cache), but without using a "out of view collection"

the out of view collection would need to be updated when an object re-enter the player view range, and when we can guess that client removed this object from his cache, so we can't do "all the job" in the Region.RemoveObject Method, we have to change the Update Thread to keep this other collection clean (Clearing when changing Region ? Removing Object that didn't send update to player since "Cache Time" ? etc...)

An other change I tried locally is to change the Region nextObjectID update when removing object from Region, every object removed always register it's Own ID as the next Free ID this cause the most troubles you've seen even if it's nice to show WorldUpdateThread Wrong Implementations This nextObjectID update is only needed when the Region Object Array is Full to improve Next ID Lookup (this mean we still have to fix the Update Thread in case we're in a "Full" Region)

Yes the "outofview" collection must be cleared like the GameCurrentArray and maybe when you move the player over the RefreshDistance in the player .MoveTo() -> .RefreshWorld() but you have to send a SendObjectRemove to every single obj in that list if exists "in a row".

I don't know if all this packet work is taking resources and is justified.

Also you need to keep the "outofview" collection clean to prevent huge collections, according with the client version cache for example checking when the obj was added and send an ObjRemove every 30 secs ( 1.109 cache) if the obj is still outofview. Also this will help with OID fast creations..

The "outofview" collection may become very huge, player is walking through the region collecting thousand of values.

Reason is why i prefer to send a .SendObjectRemove when the target is going out of view , it solves everything except your very last update for the client

[Leodagan]

we're ok on most of technical aspects

But on your last point :

We can't send "ObjectRemove" Packet for every object getting out of view range, this is a client-side performance hit, and it brings a lot of problem when player move high speed...

Once an object is "Removed" Client Side, when the Client receive a new "Update packet" for this object (getting back in view range for example), it won't display the Object Immediately, Client will need to send a request to the server for a "Create Packet".

Create Request Packet are somewhat handled sequentially by client that's why there is always some "latency" when porting on long range to make new NPC appear, update packet are meant to be concurrent for the client (they should even be handled as UDP, but we force them through TCP because we can't handle UDP Encryption), so you should better send too much "Update" than too much "Create"

[Stephen]

Create Request Packet are somewhat handled sequentially by client that's why there is always some "latency" when porting on long range to make new NPC appear, update packet are meant to be concurrent for the client (they should even be handled as UDP, but we force them through TCP because we can't handle UDP Encryption), so you should better send too much "Update" than too much "Create"

Oh i see.

Well so it looks easy now.

Step 1

We collect <RegID,OID>,long values for "outfoview" objects in new collection adding them when the WorldUpdateThread catchs the obj outofview with a method that also check if the current "outofview" objects are still outofview for the client over 30 secs. If yes , remove it from the collection without any packet send.

according to client versions, Is 30 secs the max period of time the client keep the obj data?

Step 2

Trigger an object delete in REGION.RemoveObject() with a method that checks for every client "outofview" collection if the obj need a .SendObjectDelete.


is extra Cleaning still necessary?

for every worldupdatethread client task:

If the obj is outview over 30 secs is removed from the collection..
if the obj is in-view is removed from the collection..
if the obj is deleted is removed from the collection and client-side updated.

So the client never keep a Wrong OID in cache for new objects around him with recycled OIDs.

Probalby a .Clean(); when the client is changing region or log out , same way of GameObjectUpdateArray.

is this the best way to do that?

Solved?

theoretically, it seems yes.

I'll write down and test it.

Sorry for my (bad) elemental english. i'm doing my best lol

[Leodagan]

English is not my natural language either

You've got the full specs that's exactly what was around my mind after our discussions with only one exception :

I was thinking of just adding an "OutOfView" Boolean in the GameObjectUpdateArray

This way we just need a short fix in the WorldUpdateThread to update this boolean to "True" when the "OutOfView Update" is sent (and NOT remove the object from GameObjectUpdateArray anymore), and change all methods that "Add Objects" to force the flag to "True" (or set some default to True for any Update)

So Instead of ReaderWriterDictionary<Tuple<ushort, ushort>, long> we should use a ReaderWriterDictionary<Tuple<ushort, ushort>, new { long UpdateTime, bool OutOfView }>

This will probably need a "mini-object" instead of my "dynamic" writing

Code: Select all

public sealed class ObjectUpdateData { long m_updateTime; public long UpdateTime { get { return m_updateTime; } set { m_updateTime = value; OutOfView = false; }; } public bool OutOfView { get; set; } public DataObjectUpdate(long time) { UpdateTime = time; } } public ReaderWriterDictionary<Tuple<ushort, ushort>, ObjectUpdateData> GameObjectUpdateArray...

I'm not really afraid of tracking every part of code using this array, it's a recent change anyway (I made this some months ago) so it can be updated freely to a more stable code...

If you want to make this thread safe you'll probably need to Update ReaderWriterDictionary<K, V> with a new method to "slim lock" on Object Update with a "lamba"

Code: Select all

public bool UpdateIfExists(TKey key, Func<TValue, TValue> func) { m_rwLock.EnterUpgradeableReadLock(); bool replaced = false; try { if (m_dictionary.ContainsKey(key)) { m_rwLock.EnterWriteLock(); try { m_dictionary[key] = func(m_dictionary[key]); replaced = true; } finally { m_rwLock.ExitWriteLock(); } } } finally { m_rwLock.ExitUpgradeableReadLock(); } return replaced; }

Usage :

Code: Select all

dict.UpdateIfExists(key, obj => { obj.OutOfView = true; return obj; } );

Sorry I can't add all this to DOL actually, I'm working on player inventory in an other branch and I can't make much change until commit But this Week end I could help you more, if you can test your own implementation already we'll know if there are any flows in the "specs"

[Stephen]

if you can test your own implementation already we'll know if there are any flows in the "specs"

yes ofc

Thanks for the support

[Stephen]

Well i did a first try using a second collection .

Basically if you move in the region you collect "outofview" objects that are removed after 30 secs using the current WorldUpdateThread except in the case the obj is deleted from the region therefore all clients of region who have this obj cached are updated with a .SendObjectRemove() and objkey removed from the collection.

Code: Select all

private static void UpdatePlayerWorld(GamePlayer player, long nowTicks) { // Update Player Player's if (ServerProperties.Properties.WORLD_PLAYERTOPLAYER_UPDATE_INTERVAL > 0) UpdatePlayerOtherPlayers(player, nowTicks); // Update Player Mob's if (ServerProperties.Properties.WORLD_NPC_UPDATE_INTERVAL > 0) UpdatePlayerNPCs(player, nowTicks); // Update Player Static Item if (ServerProperties.Properties.WORLD_OBJECT_UPDATE_INTERVAL > 0) UpdatePlayerItems(player, nowTicks); // Update Player Doors if (ServerProperties.Properties.WORLD_OBJECT_UPDATE_INTERVAL > 0) UpdatePlayerDoors(player, nowTicks); //update OutofviewCache UpdateClientOutOfViewCache(player, nowTicks); // Update Player Housing if (ServerProperties.Properties.WORLD_OBJECT_UPDATE_INTERVAL > 0) UpdatePlayerHousing(player, nowTicks); }

so basically, when the object is outofview...

Code: Select all

// We have a NPC in cache that is not in vincinity if (obj is GameNPC && !npcs.Contains((GameNPC)obj) && (nowTicks - objEntry.Value) >= GetPlayerNPCUpdateInterval) { // Update him out of View if (obj.IsVisibleTo(player)) player.Client.Out.SendObjectUpdate(obj); long dummy; // this will add the object to the cache again, remove it after sending... if(player.Client.GameObjectUpdateArray.TryRemove(objKey, out dummy)) { if (!player.Client.OutOfViewGameObjectUpdateArray.AddIfNotExists(objKey, GameTimer.GetTickCount())) player.Client.OutOfViewGameObjectUpdateArray[new Tuple<ushort, ushort>(obj.CurrentRegionID, (ushort)obj.ObjectID)] = GameTimer.GetTickCount(); } }

in the case you have the obj in-view within 30 secs and you go outofview again , the long value is updated so you will keep the objkey for another 30 secs with last update.

But we need a special handling for ControlledBrainNPCs infact here

Code: Select all

foreach (var objEntry in player.Client.GameObjectUpdateArray) { var objKey = objEntry.Key; GameObject obj = WorldMgr.GetRegion(objKey.Item1).GetObject(objKey.Item2); // Brain is updating to its master, no need to handle it. if (obj is GameNPC && ((GameNPC)obj).Brain is IControlledBrain && ((IControlledBrain)((GameNPC)obj).Brain).GetPlayerOwner() == player) continue;

continue; on controllednpc for the Owner client , because it is handled in the brain...

we need to send .SendObjectRemove() from the SummonSpellHandler.OnEffectExpires() checking if the owner (Caster) !IsWithinRadius(WorldMgr.VISIBILITY_DISTANCE).

In this way we are sure all "timed" summoned pets (theu,ani, etc etc ) are removed from client caches with the last update "outofview".

These are the results of my first test with some clients and lots of objects around the region , all my weird caches issues gone.

But maybe your way to do that with the boolean is better I'm just want to test the concept and it is working.

[Stephen]

there is another situation to handle for the same problem.

When a playing client crash or kicked by GM.

Obj is deleted but the client still keep OID data in cache .

[Stephen]

Btw....

i still think the major issue is around the OID assigment and how it's recycled in the current code...

here Gameplayer ---> public void OnLinkdeath()

Code: Select all

//Stop player if he's running.... CurrentSpeed = 0; foreach (GamePlayer player in GetPlayersInRadius(WorldMgr.VISIBILITY_DISTANCE)) { if (player.ObjectState != eObjectState.Active || player == null || player == this) continue; //Maybe there is a better solution? player.Out.SendObjectRemove(this); player.Out.SendPlayerCreate(this); }

A linkdead player is leaving his OID in cache of all clients within visibility distance.

[Leodagan]

SendPlayerCreate is updating the GameObjectUpdateArray.

What is the trouble here ? The ObjectRemove should be sent after Linkdead timer runs out

[Stephen]

SendPlayerCreate is updating the GameObjectUpdateArray.

What is the trouble here ? The ObjectRemove should be sent after Linkdead timer runs out

In this case , the player is in-view and it leaves his very last update in the worldupdatethread (we can't check the object remove trigger to remove a "outofview" obj in cache) This is what is happen in-game.After a kick, I see a ghost-char , visible and targettable but not a real target.

So for all clients within the kicked player and also when the linkdead timer is going on will keep the obj data in cache with that OID that could be recycled for other objects arounds.

[Leodagan]

I think I understand but I'm not sure how this can be a problem

The Linkdeath Remove/Create only work for player in range, and it wont update out of range player still having this object in cache ?

However the player OID isn't "released" when Linkdeath is triggered, the object is not removed from world until LinkdeathTimerCallback expires... which should trigger itself a correct SendObjectRemove to everyone still having this object in cache (in view or out of view)