Pathing Bug

[Fuzzylumpkins]

So currently I am getting the following error only sometimes when the server boots up, it seems selective.. I have tried to trace this and cannot figure out why its happening. Theres also currently only 2 paths in my db.

Code: Select all

System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value) at DOL.GS.Movement.MovementMgr.FillPathCache() in C:\GameServer\gameutils\MovementMgr.cs:line 50 at DOL.GS.Movement.MovementMgr.LoadPath(String pathID) in C:\GameServer\gameutils\MovementMgr.cs:line 124 at DOL.GS.GameNPC.AddToWorld() in C:\GameServer\gameobjects\GameNPC.cs:line 3021 at DOL.GS.Region.LoadFromDatabase(Mob[] mobObjs, Int64& mobCount, Int64& merchantCount, Int64& itemCount, Int64& bindCount) in C:\GameServer\world\Region.cs:line 938 at DOL.GS.WorldMgr.<>c__DisplayClass53_0.<Init>b__0(RegionData data) in C:\\GameServer\world\WorldMgr.cs:line 531 at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey) at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork() at System.Linq.Parallel.SpoolingTaskBase.Work() at System.Linq.Parallel.QueryTask.BaseWork(Object unused) at System.Linq.Parallel.QueryTask.<.cctor>b__0(Object o) at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of inner exception stack trace --- at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose) at System.Linq.Parallel.SpoolingTask.SpoolForAll[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, TaskScheduler taskScheduler) at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute() at System.Linq.Parallel.MergeExecutor`1.Execute() at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId) at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream) at System.Linq.Parallel.ForAllOperator`1.WrapPartitionedStream[TKey](PartitionedStream`2 inputStream, IPartitionedStreamRecipient`1 recipient, Boolean preferStriping, QuerySettings settings) at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.ChildResultsRecipient.Receive[TKey](PartitionedStream`2 inputStream) at System.Linq.Parallel.ListQueryResults`1.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient) at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient) at System.Linq.Parallel.QueryOperator`1.GetOpenedEnumerator(Nullable`1 mergeOptions, Boolean suppressOrder, Boolean forEffect, QuerySettings querySettings) at System.Linq.Parallel.ForAllOperator`1.RunSynchronously() at System.Linq.ParallelEnumerable.ForAll[TSource](ParallelQuery`1 source, Action`1 action) at DOL.GS.WorldMgr.Init(RegionData[] regionsData) in C:\GameServer\world\WorldMgr.cs:line 528 ---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value) at DOL.GS.Movement.MovementMgr.FillPathCache() in C:\GameServer\gameutils\MovementMgr.cs:line 50 at DOL.GS.Movement.MovementMgr.LoadPath(String pathID) in C:\GameServer\gameutils\MovementMgr.cs:line 124 at DOL.GS.GameNPC.AddToWorld() in C:\GameServer\gameobjects\GameNPC.cs:line 3021 at DOL.GS.Region.LoadFromDatabase(Mob[] mobObjs, Int64& mobCount, Int64& merchantCount, Int64& itemCount, Int64& bindCount) in C:\GameServer\world\Region.cs:line 938 at DOL.GS.WorldMgr.<>c__DisplayClass53_0.<Init>b__0(RegionData data) in C:\GameServer\world\WorldMgr.cs:line 531 at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey) at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork() at System.Linq.Parallel.SpoolingTaskBase.Work() at System.Linq.Parallel.QueryTask.BaseWork(Object unused) at System.Linq.Parallel.QueryTask.<.cctor>b__0(Object o) at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.Execute()<---

[Leodagan]

in Region.cs "LoadFromDatabase()" beginning around line 787.

This create Region Objects and Mobs before adding them to world...

When Adding to World the MovementMgr can be initiated when any Mobs need a path from cache...

Region loading is random due to the Parallel LINQ Query, so you may not experience the same Cache Fill Trigger at the same Region Loading...

What DOL version are you using ? this is not latest source anyway ? (the line in your log don't match anything in my code repository)

[Fuzzylumpkins]

Its a revision from right around the time you released your data careers update. But i've kept it pretty current with svn other than your recent db change... i just have added a few functions for region id checking to region.cs... other than that all default functions were left default, I havent touched them. I can post code of any function that may help you assist me with figuring out whats going on.

[Leodagan]

You know we are using "Git" now ?

https://github.com/Dawn-of-Light/DOLSharp

If you have your code base available as a Git Fork / Branch I could throw some debugger on it, but actually it's really too hard to follow code pasted in a forum post

I'm going to update the startup sequence a lot in the next DOL updates (and have already build recently new managers for some part of the startup), any reason not to use the latest Git Code ?

[Fuzzylumpkins]

You know we are using "Git" now ?

https://github.com/Dawn-of-Light/DOLSharp

If you have your code base available as a Git Fork / Branch I could throw some debugger on it, but actually it's really too hard to follow code pasted in a forum post

I'm going to update the startup sequence a lot in the next DOL updates (and have already build recently new managers for some part of the startup), any reason not to use the latest Git Code ?

Yes I know we're using git now, the reason im not using latest git code is because I have a lot changed in my current code and have cleaned up a lot of things im not using, that doesn't "jive" well with all of the current code... ('im emulating a non toa version of DAoC.) But as I said, nothing has been touched with pathing or mob loading etc.

[ontheDOL]

I'm not sure if this was ever fixed, but I still get this error from time to time. I hadn't happened in over a week and i thought it was randomly gone! But just now i got the error on three consecutive attempts to load the server...

Code: Select all

2017-04-02 12:06:29,540 - [MAIN] - ERROR - DOL.GS.WorldMgr - Init System.AggregateException: Se han producido uno o varios errores. ---> System.ArgumentException: Ya se agregó un elemento con la misma clave. en System.ThrowHelper.ThrowArgumentException(ExceptionResource resource) en System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) en System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value) en DOL.GS.Movement.MovementMgr.FillPathCache() en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\gameutils\MovementMgr.cs:línea 50 en DOL.GS.Movement.MovementMgr.LoadPath(String pathID) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\gameutils\MovementMgr.cs:línea 124 en DOL.GS.GameNPC.AddToWorld() en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\gameobjects\GameNPC.cs:línea 2888 en DOL.GS.Region.LoadFromDatabase(Mob[] mobObjs, Int64& mobCount, Int64& merchantCount, Int64& itemCount, Int64& bindCount) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\world\Region.cs:línea 912 en DOL.GS.WorldMgr.<>c__DisplayClass3.<Init>b__2(RegionData data) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\world\WorldMgr.cs:línea 531 en System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey) en System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork() en System.Linq.Parallel.SpoolingTaskBase.Work() en System.Linq.Parallel.QueryTask.BaseWork(Object unused) en System.Linq.Parallel.QueryTask.<>c.<.cctor>b__10_0(Object o) en System.Threading.Tasks.Task.InnerInvoke() en System.Threading.Tasks.Task.Execute() --- Fin del seguimiento de la pila de la excepción interna --- en System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose) en System.Linq.Parallel.SpoolingTask.SpoolForAll[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, TaskScheduler taskScheduler) en System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute() en System.Linq.Parallel.MergeExecutor`1.Execute() en System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId) en System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream) en System.Linq.Parallel.ForAllOperator`1.WrapPartitionedStream[TKey](PartitionedStream`2 inputStream, IPartitionedStreamRecipient`1 recipient, Boolean preferStriping, QuerySettings settings) en System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.ChildResultsRecipient.Receive[TKey](PartitionedStream`2 inputStream) en System.Linq.Parallel.ListQueryResults`1.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient) en System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient) en System.Linq.Parallel.QueryOperator`1.GetOpenedEnumerator(Nullable`1 mergeOptions, Boolean suppressOrder, Boolean forEffect, QuerySettings querySettings) en System.Linq.Parallel.ForAllOperator`1.RunSynchronously() en System.Linq.ParallelEnumerable.ForAll[TSource](ParallelQuery`1 source, Action`1 action) en DOL.GS.WorldMgr.Init(RegionData[] regionsData) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\world\WorldMgr.cs:línea 528 ---> (Nº de excepción interna 0) System.ArgumentException: Ya se agregó un elemento con la misma clave. en System.ThrowHelper.ThrowArgumentException(ExceptionResource resource) en System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) en System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value) en DOL.GS.Movement.MovementMgr.FillPathCache() en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\gameutils\MovementMgr.cs:línea 50 en DOL.GS.Movement.MovementMgr.LoadPath(String pathID) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\gameutils\MovementMgr.cs:línea 124 en DOL.GS.GameNPC.AddToWorld() en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\gameobjects\GameNPC.cs:línea 2888 en DOL.GS.Region.LoadFromDatabase(Mob[] mobObjs, Int64& mobCount, Int64& merchantCount, Int64& itemCount, Int64& bindCount) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\world\Region.cs:línea 912 en DOL.GS.WorldMgr.<>c__DisplayClass3.<Init>b__2(RegionData data) en c:\daoc server\DOLSharp-1.9.7.3595 work copy\GameServer\world\WorldMgr.cs:línea 531 en System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey) en System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork() en System.Linq.Parallel.SpoolingTaskBase.Work() en System.Linq.Parallel.QueryTask.BaseWork(Object unused) en System.Linq.Parallel.QueryTask.<>c.<.cctor>b__10_0(Object o) en System.Threading.Tasks.Task.InnerInvoke() en System.Threading.Tasks.Task.Execute()<---

alot of mumbo jumbo, but the top error stems from an element with the same key already. I have a few pathed mobs, but none are duplicate or using the same path.

Is there anything I can do to fix this without removing the mobs paths?

[ontheDOL]

could this be due to having path name too similar? For example i have some paths : MithraRoam01, MithraRoam02, MithraRoam03

i'm not sure how this thing searches/loads ... it is odd how it only happens randomly

[PlanarChaosRvrtwo]

To similiar names cant be the issue.
I have the guess its more an dirty script somewhere that create issue at some point and couse it dont throw out a clean result all scripts working with parts of that bug also.

I could be totally wrong on it but in my db paths work 100% fine but have the same issue from time to time and cause of that its ether an bug/ dirty script or an missmatch of scripts/code and DB.

[Leodagan]

It looks like a Race Condition...

MovementMgr.FillPathCache() is called multiple time when it shouldn't... (it retrieve all database content when initializing, this should only be called once in "MovementMgr.LoadPath()" when cache is empty)

There is no safety to prevent a concurrent cache initialization, some workaround would be to trigger the cache building before the Region are initialized in a Parallel Query, or to add some Initialization flag in the form of a "volatile bool" that support concurrent access in the LoadPath method...



in LoadPath

Code: Select all

if (m_pathCache.Count == 0) { FillPathCache(); }

this code part is not thread safe

Between the "Count Call" of the Cache collection and the end of the FillPathCache Method nothing prevent an other thread from getting inside the if statement (and there is a full Database query delaying the cache filling)

This kind of static Initialization should be protected for thread concurrency to make sure different access only trigger the method once...

This can be achieved with some code design too, having a private Locking Object, and moving methods that modify the collections to private visibility so you can handle the lock before allowing any access...

Code: Select all

// in the class body private static object LockObject = new object(); private static void FillPathCache() // ... public static PathPoint LoadPath(string pathID) { lock(LockObject) { if (m_pathCache.Count == 0) { FillPathCache(); } } // ... }

Concurrent threads can only run "lock" code part iteratively, they must wait for an other thread to release the lock before acquiring the lock for themselves, this way you make sure that once "FillPathCache()" has finished his job, the next thread will check the Cache Count with an up to date value and skip the cache build.

[ontheDOL]

hmm, this sounds like a good job for Leodagan

hmm so basically the movementmgr and region are loaded side by side to speed up the server initialisation? Could i not just change the load sequence so movementmgr is done after all the regions are loaded?

[Leodagan]

I'm sure LoadPath() is triggered when some mobs are loaded... (Bingo : there is a call in GameNPC.AddToWorld)

Regions and Paths are not loaded side-by-side, it's just that All Regions are loaded side-by-side, so you have multiple threads calling AddToWorld simultaneously but not on the same game objects (and at most 1 thread work on one region at a time), unfortunately MovementMgr is coded like a static class so all thread end up using the same method and sharing the same variables !

This is fun because you'll understand the mess that is when you'll try to have a GameGravestone Startup Sequence in the other topic we're discussing http://www.dolserver.net/viewtopic.php?f=6&t=23104

Someone probably didn't want to write a Startup Sequence for Paths, so they try to cheat by calling a Static Class in AddToWorld to make sure it's called at least once in the whole server life if you at least use one NPC in the entire game

In fact if you just want a quick'n dirty workaround, you should use some try/catch block that prevent the Duplicate Key Exception from raising all the way to Regions Startup Sequence and canceling the current region loading...

Try/Catch block are Error handler that should be used when you don't want to abort the execution stack in case of unexpected behavior, clearly an error in Paths Management should not prevent from adding a GameObject to the World, even if some NPC could fail to initialize a Patrol Path it's not related to values needed by Region Manager to render a GameObject...

[ontheDOL]

Regions and Paths are not loaded side-by-side, it's just that All Regions are loaded side-by-side, so you have multiple threads calling AddToWorld simultaneously but not on the same game objects (and at most 1 thread work on one region at a time), unfortunately MovementMgr is coded like a static class so all thread end up using the same method and sharing the same variables !

ok, the error makes a bit more sense now ^^

In fact if you just want a quick'n dirty workaround, you should use some try/catch block that prevent the Duplicate Key Exception from raising all the way to Regions Startup Sequence and canceling the current region loading...

quick and dirty is about the extent of my abilities, but i'm not sure what is better, loading server multiple times to get it to load, Or have server load with some mobs failing pathing init....

[ontheDOL]

just a follow up on this, i put that "lock" in that Leodagan suggested up above, and i havent had any issues in 25+ server startups, all paths working fine.