Zone Locks on Massive NPC Death

For any problems with Dawn of Light website or game server, please direct questions and problems here.

Moderator: Support Team

Re: Zone Locks on Massive NPC Death

Postby Crazys » Mon Sep 29, 2014 10:26 pm

So do you have custom GameServer rules that does something when a mob dies? This is what I was alluding too above, the code that gets called must be optimized or it could cause issues unrelated to the core code.

Storm is on slower hardware than D2 so I imagine any mass death would cause issues. :)

D2 is almost all custom, the mob class is custom, the loot generators are custom, the server rules are custom.
No. thats what I said above None of this is custom on my server. all of it is 100% current SVN. As it is a PvP server its relying on the Abstract Server Rules handling for NPC Death.

Thats what i'm worried about... and if mass pulling of over 100 mobs works on yours something has to be wrong with the current svn that you fixed.
Crazys
Contributor
 
Posts: 346
Joined: Tue Nov 07, 2006 10:18 pm

Re: Zone Locks on Massive NPC Death

Postby Crazys » Mon Sep 29, 2014 10:46 pm

Leodagan!!! YOU SIR ARE RIGHT!!!

Test after changing the locks. 200 mob pull no errors or delays.

I call this a pass. THANK YOU for pointing me in the right direction!

I'll pull this into a patch for svn when i get some spare time next day or two


*** I lied... 300 mob pull for me fine... users pull 50 and freeze it :/ something yucky!
Crazys
Contributor
 
Posts: 346
Joined: Tue Nov 07, 2006 10:18 pm

Re: Zone Locks on Massive NPC Death

Postby Leodagan » Tue Sep 30, 2014 5:17 am

What did you change ?

(you should post partial "diff" log in a [ code ][ /code ] block to show the area of code you changed if it's trivial...)

Is there some different behavior from a Plvl+ character and a regular player ?

---

For D2 Code, if most collection browsing with locks have been moved to random loot compute it may have changed a lot of things ;)

I didn't check as far as locating the "inventory" code on mob loot drop to player inventory, but it can be an area with slowdowns too if we have some database queries or "transaction" queries running when putting stuff in player bag ?
User avatar
Leodagan
Developer
 
Posts: 1350
Joined: Tue May 01, 2012 9:30 am
Website: https://daoc.freyad.net
Location: Lyon

Re: Zone Locks on Massive NPC Death

Postby Tolakram » Tue Sep 30, 2014 7:17 am

Leodagan!!! YOU SIR ARE RIGHT!!!

Test after changing the locks. 200 mob pull no errors or delays.

I call this a pass. THANK YOU for pointing me in the right direction!

I'll pull this into a patch for svn when i get some spare time next day or two


*** I lied... 300 mob pull for me fine... users pull 50 and freeze it :/ something yucky!
Is a necromancer involved?
- Mark
User avatar
Tolakram
Storm / Storm-D2 Admin
 
Posts: 9189
Joined: Tue Jun 13, 2006 1:49 am
Location: Kentucky, USA

Re: Zone Locks on Massive NPC Death

Postby Crazys » Tue Sep 30, 2014 7:29 am


Is a necromancer involved?
Testing w/ chanter pet damage shield
2 players in group/area - 200 mob pull no issue. goes through fine.
8 players in group - 50 mob pull zone locks
The 6 extra plays stood there did not take any action in the fight or do anyhting other then join group and stand there during the fight/death
Crazys
Contributor
 
Posts: 346
Joined: Tue Nov 07, 2006 10:18 pm

Re: Zone Locks on Massive NPC Death

Postby Leodagan » Tue Sep 30, 2014 7:40 am

What happen with 8 players in the area but not in group ? :D

(Something may be wrong around aggro management with what you're telling there...a lot of aggrotable update are involved when fighting with group and pet, and all the aggro the pet can make must be transfered to the group !!)
User avatar
Leodagan
Developer
 
Posts: 1350
Joined: Tue May 01, 2012 9:30 am
Website: https://daoc.freyad.net
Location: Lyon

Re: Zone Locks on Massive NPC Death

Postby Crazys » Tue Sep 30, 2014 8:10 am

What happen with 8 players in the area but not in group ? :D

(Something may be wrong around aggro management with what you're telling there...a lot of aggrotable update are involved when fighting with group and pet, and all the aggro the pet can make must be transfered to the group !!)
2 in group - with 3 Plvled accounts near no issues.
I see where your going here!!! Will have to look tomorrow... WAY WAY WAY WAY WAY past bedtime...

I will note... I got mad @ this point and started doing some Chopping... like removing gold bag drops...
No reason to drop gold bags if we just autoloot them anyway.... :/
Crazys
Contributor
 
Posts: 346
Joined: Tue Nov 07, 2006 10:18 pm

Re: Zone Locks on Massive NPC Death

Postby Tolakram » Tue Sep 30, 2014 8:58 am

I would try the 50 mob test over and over to make sure it's repeatable. Random things happen when mobs die, it could be a particular loot generator causing the issue.
- Mark
User avatar
Tolakram
Storm / Storm-D2 Admin
 
Posts: 9189
Joined: Tue Jun 13, 2006 1:49 am
Location: Kentucky, USA

Re: Zone Locks on Massive NPC Death

Postby Crazys » Wed Oct 01, 2014 3:40 am

I would try the 50 mob test over and over to make sure it's repeatable. Random things happen when mobs die, it could be a particular loot generator causing the issue.
I finally think I got everything....
I removed gold bags - so they nolonger drop and just grant to the player/group instantly instead of the autoloot check and spamming clients with auto-loot gold bag create/deletion spam. (will be checking items the same way soon)
Removed the loot gens that i'm not using.
unlocked the all the locks i could find in the gamenpc code.
Final issue I was getting was Guild Bonus XP null error spam zilla.
Currently have this disabled.

Actually having an issue with all of the guild Event items giving me issues...
Most likely going to remove the entire Guild Event section and merge it into Server Rules as a few If checks.


but after all is said and done... Finally no more zone locks from pulling mobs...

I will note... Each of my changes seemed to improve the quality of the pulls so I can't say there was one root cause but It seems Many Many things mixed into a big issue.
Crazys
Contributor
 
Posts: 346
Joined: Tue Nov 07, 2006 10:18 pm

Re: Zone Locks on Massive NPC Death

Postby Tolakram » Thu Oct 02, 2014 12:28 pm

Some of the locks are needed, you might want to review that otherwise you might end up with inventory issues / items or money getting lost due to concurrent updates.

You might simply try removing just the guild code, that may have been the primary issue.
- Mark
User avatar
Tolakram
Storm / Storm-D2 Admin
 
Posts: 9189
Joined: Tue Jun 13, 2006 1:49 am
Location: Kentucky, USA

Re: Zone Locks on Massive NPC Death

Postby Leodagan » Thu Oct 02, 2014 2:05 pm

The bad thing around basic "lock" is that it locks that part of code for writing as well as reading...

You can't just lock in "Writing" method, as you could have incomplete "reading" in the meantime, so this has all the chance to prevent concurrent read even on thread-safe collections...

The best use-case for lock is when you want to be sure that a method won't be executed by multiple thread at the same time... (typically a method like Die() shouldn't allow concurrent calls at all, an object can only die() once at a given time)

Using too much synchronizing method (locks...) can reduce drastically the speed of a code part, methods could be waiting for a read lock to wear out from an other thread when they don't even need write access... (typically statistics thread will gather data periodically and may call methods to retrieve collection that will locks somewhere)

I don't know if there is anyway to handle read/write lock, I saw some code in DOL using this (around player login request I think), I have no idea if it could be more efficient or if it can turn into a "debug hell"...

Anyway locking collection is most of time a bad habit, we should handle access to the collection using method that can clearly choose between locking, getting a snapshot, updating asynchronously and not rely on objects that directly gives access to their Collection Property without checks and where a lot of parts of the game browse or update this collection without even "locking" it's syncroot everytime... (locking a syncroot is only usefull if every method does this before trying to access...)

For my part there are some area of DOL where I replace Generic Collections, with Concurrent Collections, but these aren't the best way to share collection, updates are asynchronous and can lead to racing condition, and "Getters" always get copies of the collections they use heavy memory and can have worse performance than a humanly "locked" collections, by default these objects are created with some "update queues" that allow to update them without even blocking (and the queue is then emptied asynchronously) to provide concurrency by default they create N queues where N is processor count (clearly overkill for most of our need)

There are some "blocking" collection in the "Concurrent" name space but it's meant for "producer->consumer" threaded scénarii, it block on read waiting for input (it's more like a FIFO)

They are also some good habit that are "Thread Safe", replacing a collection is always thread safe !!
Code: Select all
public void MethodUpdateCollection(int withValue) { var copy = new Collection(m_privateCollection); copy.Add(withValue); m_privateCollection = copy; }
At worst some "racing" conditions can retrieve the reference of the previous object... and be careful, the copy isn't thread safe ;) there should be some locking when copying the collection !
User avatar
Leodagan
Developer
 
Posts: 1350
Joined: Tue May 01, 2012 9:30 am
Website: https://daoc.freyad.net
Location: Lyon

Re: Zone Locks on Massive NPC Death

Postby Tolakram » Thu Oct 02, 2014 2:37 pm

IMO locking should not be an issue unless there is an uncontrolled delay while the lock is being held. Locking has been blamed for too many issues when instead it's a server speed issue. When a server is overloaded the first symptom is going to be the locking, but that does not indicate locking is the problem. A threaded application has to be made thread safe and locking is how this is accomplished. Toby put some new locking code into some areas of the core using newer ReaderWriterLock feature to help with specific instances where single write multiple read functionality was needed. It can be argued how many more places that is really needed.

Generally the best practice for locking is to create a dummy object and lock it. This creates a simple semaphore (flag) that acts to prevent other methods from locking if the object is already locked. The primary problem with this is that there is no failure mode, it simply waits until a lock can be established. For the most part this is EXACTLY what we want to do.

If someone decides to write smarter locking then the code that calls it will have to be refactored to handle failed lock attempts. For example, I pick an item up off the ground and place in my inventory. Current design will wait until lock is achieved to pickup object. A failure to lock would mean the item pickup should fail and the item will be left on the ground, or a loop would be required to keep attempting to lock the object until success, or we tell the player ... what? Sorry you can't pick that up right now?

It's helpful to write out the logic in plain English first to see if we really want to modify how some of this works. In Crazy's case, with removing locks, it won't be long until someone complains they lost items in inventory, or worse. :)

Just remember this is a game. If the object of the game is to kill foozle and get the I WIN spear, but the player can't pick up the I WIN spear because the server is overloaded, then the game failed. Might as well lock up. :)
- Mark
User avatar
Tolakram
Storm / Storm-D2 Admin
 
Posts: 9189
Joined: Tue Jun 13, 2006 1:49 am
Location: Kentucky, USA

Re: Zone Locks on Massive NPC Death

Postby Leodagan » Thu Oct 02, 2014 3:13 pm

Sure removing lock is not a good idea, DOL is pretty "lowly" threaded and it's still enough to have some weird behavior when not using synchronization methods ! (Network Stack is completely threaded...)

And as you said using newer method like ReaderWriterLock() may improve concurrent reads, at the cost of rewriting some part of code where it can be used...

My firsts "posts" pointed out I would rather investigate around which part of code takes too much CPU time (using some timestamp or other Profiler/Debugger) but locking around a heavy Client/Server exchange scenario can bring really bad things !!

I don't have the code in mind, but imagine 50 mobs are going to die, each one running about 10 full collections checks (checking attackers, checking xpgainers, checking drops, checking factions, checking player in the area for messages etc etc), this code is heavily "locked" around every collection browsing... Now all these checks are translated into "packet update" that we need to send to players to reflect what happen in world, we're gonna launch a packet storm (I/O are slows !) during Locks, and may have asynchronous reply that will steal a lock in the queue of the dieing locks methods, thus mob dying sequence may have to wait for one of the player object to handle an incoming packet before continuing onto a next loop or next mob die sequence...

1 people kill 50 mobs, triggering 50 die() sequence that will each send some packet to all player in the visibility range, complexity raise fast ;) we lock all the mobs, where only the player will get an update ! (mob will just be recycled into the respawn timer...)

You said Lock are exactly what we need being semaphore that "can't fail" (in fact there is ONE failure in lock, it's the deadlock, and this is the end of software execution...) but not exactly, here in die sequence, locks are mostly protecting for Reading, it's not that I want to use my data "all for my thread", it's that I want to browse this data without another thread changing it, I don't know why Reader/WriterLock should have more fail cases than any other lock (and by the way you can EnterMonitor() with timeout if you don't use the Keyword Lock...) but anyway I still think either lock should be avoided by using Copies of data or by using "smart" locking, synchronized code is really specific to some use-case in multi threading (producer->consumer scenario use a lot of synchronization, but in DOL we are in a Time Loop with Events...)
User avatar
Leodagan
Developer
 
Posts: 1350
Joined: Tue May 01, 2012 9:30 am
Website: https://daoc.freyad.net
Location: Lyon

Re: Zone Locks on Massive NPC Death

Postby Crazys » Thu Oct 02, 2014 4:08 pm

IMO locking should not be an issue unless there is an uncontrolled delay while the lock is being held. Locking has been blamed for too many issues when instead it's a server speed issue. When a server is overloaded the first symptom is going to be the locking, but that does not indicate locking is the problem. A threaded application has to be made thread safe and locking is how this is accomplished. Toby put some new locking code into some areas of the core using newer ReaderWriterLock feature to help with specific instances where single write multiple read functionality was needed. It can be argued how many more places that is really needed.

Generally the best practice for locking is to create a dummy object and lock it. This creates a simple semaphore (flag) that acts to prevent other methods from locking if the object is already locked. The primary problem with this is that there is no failure mode, it simply waits until a lock can be established. For the most part this is EXACTLY what we want to do.

If someone decides to write smarter locking then the code that calls it will have to be refactored to handle failed lock attempts. For example, I pick an item up off the ground and place in my inventory. Current design will wait until lock is achieved to pickup object. A failure to lock would mean the item pickup should fail and the item will be left on the ground, or a loop would be required to keep attempting to lock the object until success, or we tell the player ... what? Sorry you can't pick that up right now?

It's helpful to write out the logic in plain English first to see if we really want to modify how some of this works. In Crazy's case, with removing locks, it won't be long until someone complains they lost items in inventory, or worse. :)

Just remember this is a game. If the object of the game is to kill foozle and get the I WIN spear, but the player can't pick up the I WIN spear because the server is overloaded, then the game failed. Might as well lock up. :)
Never changed Items. ;) So no issue
Also - My changes to gold. Don't need locks as the item doesn't exist anymore.
The sections of locks I removed focused heavily on Players gaining XP and RPs which shouldn't matter if they gain out of order a Gain is a gain.
Crazys
Contributor
 
Posts: 346
Joined: Tue Nov 07, 2006 10:18 pm

Re: Zone Locks on Massive NPC Death

Postby Leodagan » Thu Oct 02, 2014 4:31 pm

Locking isn't made to enforce order ;)

It's made so that every "Process that have something to do on the same property" (like XP count) do it one after an other and not all at the same time, because if your XP count is 100, the first process try to add +20, and it stores 120 in there, but another process in the meantime wanted to add +40 and it read the value "100" before the first process add +20, so this second process will store 140, you can either end with 120 or 140 or something corrupted that doesn't mean anything, and by chance you can sometimes have the expected 160 results ;)

In fact DOL is not multithreaded around "rules" everything that "happens" in a region is always executed in the same order as the GameTimer for a Region is always the same the danger is only when you try to update object that are in another region or if you try to update object directly from client request handler...

Most players XP sources are in fact "rules" and "timers" when an object calls it dying method it will be triggered by a "hit" that is started from an AttackAction Timer or a SpellHandler Timer, Event Handler are executed "inline" with these methods, when calling a method "Notify()" you can assume every event handler will be executed when the method returns (except maybe some event that start another GameTimer...)

A lot of network player requests start a GameTimer to handle the data from client packet, this way the packet handling is done in the same thread as the "Region" it's smart to get rid of a lot of locking trouble, but it can give some latency to player actions ! (some handlers can't afford this, like player position handler !!)
User avatar
Leodagan
Developer
 
Posts: 1350
Joined: Tue May 01, 2012 9:30 am
Website: https://daoc.freyad.net
Location: Lyon


Return to “%s” Support

Who is online

Users browsing this forum: No registered users and 1 guest