DOL on Linux with Mono3

[Graveen]

it was an attempt to make you update your guide

[Leodagan]

You could approve this one in the meantime : http://www.dolserver.net/articles/?article=47

[Argo]

You could approve this one in the meantime : http://www.dolserver.net/articles/?article=47

approved

[HunabKu]

Nice works, thanks Leodagan !

it was an attempt to make you update your guide

Hum i see catches missing ...

[Leodagan]

Update :

Found a great optimization for Mono and DOL. Using AOT compiler... (very experimental) it lower memory usage by 20% (we are still about 2.5x memory usage of .NET Runtime) and reduce load average a lot, DOL handles way better heavy system load with this.

after compiling dol use a script like that :

Code: Select all

#/bin/bash cd `dirname $0` mono-sgen --aot -O=all ./MySql.Data.dll mono-sgen --aot -O=all ./lib/ICSharpCode.SharpZipLib.dll mono-sgen --aot -O=all ./lib/MySql.Data.dll mono-sgen --aot -O=all ./lib/DOLDatabase.dll mono-sgen --aot -O=all ./lib/DOLBase.dll mono-sgen --aot -O=all ./lib/GameServerScripts.dll mono-sgen --aot -O=all ./lib/GameServer.dll

Tested with a fresh Mono 3.2, improvements could be related to version update too (sorry I'm not in a testing mood, but more in a stability improvement war !)

[Blue]

Is Mono still as laggy as I remember it? Some years ago the garbage collection completely stopped the server for up to 1000ms which disqualified Mono for me.

[Leodagan]

Well I won't talk for Mono 2 which is still the release version.

I'm compiling Mono 3 from sources and there is a new Garbage Collector (SGEN), a new JIT compiler (LLVM), and this AOT Compiler (it's an Advanced On Time compiler which can optimize code further than a JIT)

If you had trouble with garbage collector back then this could be solved by SGEN, but Mono in general is still heavier than .NET to do the same thing (but maybe not slower)

[Blue]

It was a lot slower and memory intensive back then. So much that I had to decide to use Microsoft.NET. Today's processing power is way bigger and that could be not such a big deal anymore. But some years ago it was very difficult to even host the server for 100 players und Mono.

[Leodagan]

Well honestly just using default distribution and mono package without any optimization is still a mess for 100 players !

That's the limit I've been running into some days ago on a Core i3 + 8 Go Memory !!

With some strong MySQL optimization I could leave some more space to DOLServer and prevent too much SQL long queries.

Then I've drop LLVM which use too much memory, it was a bit more CPU intensive, but I don't care, if memory is full on a Linux box your SQL server won't have any disk cache and it will get really bad !!

With this kind of config 125-150 players are well handled.

But this AOT compiler is really a life savier even on a strong machine (Core i3 Sandy Bridge can beat some 2-3 years old Xeon) As I understand it use static compile optimization to build ".SO" directly linking your Linux Libraries, and my libraries are all compiled with strongest GCC optimization and CPU Optimizations

My CPU usage has drop totally ! Memory usage some more, and it really handles better when I'm doing full database dump from desynching region. It handle really well when I'm doing Database Update on-live eating up one core of CPU, well instead of going in a death loop when server is lightly loaded it still goes well when server is going above 2 or 3 of load average !

Right now except for a memory usage still huge (3.5 GB used) It's feeling like going faster than .NET, but I used AOT to recompile the ENTIRE Mono DLL tree ! (there is a walktrough on Mono AOT page)

[Blue]

I doubt its faster than microsoft .NET but would be happy to be wrong as linux servers are in most cases cheaper. As comparison: Uthgard at the moment with 750 users takes about 15-20% cpu load on i920. If we have big events we get to 100% but then we also have up to 6mb/sec upload which itself is heavy. The more players see others the worse the performance. Zergevents are heavy load.

[Leodagan]

Interesting stats about a heavy loaded server.

For now on Freyad with 120 players we have 60% "average" of one core usage (on 2x2threads it shows around 15% by cores), this leaves some room for CPU.

an i7 920 is about 2 times faster than a Core i3. With a simple ratio we could say .NET is still way more efficient than Mono.

But I expect that the heavier memory usage is for something (caches ? Preload ? Reserved ?) and now I really want to see how server is behaving under an average load (which isn't reached anymore...) which is something that Linux should handle pretty well ! (and without starting trolls, way more better than windows as long as you have memory !)

And I'm really interested in this load raising when more players sees each other, I know it's normal that player seeing each other triggers more update and network traffic, but for now the code parts I modified were thought with this specific case in mind, which make them less efficient in low populated servers and I expect them to scale better on high population.

To achieve this I always need more memory ! (scaling is mostly made by caching or using one result for more than one client, I'm not a scientist or mathematician )

[Blue]

One important performance boost is giving Mysql 4GB memory for InnoDB.

[Leodagan]

I fixed MySQL Performance by reducing memory to leave more for DOL, and using mostly unsafe option with more frequent backup...

from a 16 seconds 100+ player save thread, I've drop to 4 seconds save thread (when it saves everything) for a 800M~1Gb memory usage.

I'll post the my.ini from work, but there are some specific "MariaDB" optimization (which doesn't make it better than MySQL for now...)

[Leodagan]

Config about mysqld

Code: Select all

[mysqld] thread_handling=pool-of-threads thread_pool_size=2 thread_pool_max_threads=20 wait_timeout=604800 max_connections=15 query_cache_limit = 16M query_cache_size = 256M # Turn on disk-ordered reads optimizer_switch='mrr=on' optimizer_switch='mrr_sort_keys=on' optimizer_switch='mrr_cost_based=off' optimizer_switch='index_merge_sort_intersection=on' # Size limit for the whole join join_buffer_space_limit = 128M # Limit for each individual table join_buffer_size = 64M # Turn on both Hash Join and Batched Key Access # should be 8 here for mariadb > 5.5.34 join_cache_level = 2 lower_case_table_names=1 performance_schema character-set-server = utf8 user = mysql port = 3308 socket = /var/run/mysqld/mysqld3.sock pid-file = /var/run/mysqld/mysqld3.pid log-error = /var/log/mysql/mysqld3.err basedir = /usr datadir = /var/lib/mysql3 key_buffer = 8M max_allowed_packet = 1M table_cache = 128 sort_buffer_size = 4M net_buffer_length = 32K read_buffer_size = 2M read_rnd_buffer_size = 4M myisam_sort_buffer_size = 16M thread_stack = 192K

Reducing Thread handling this is a huge memory sink, only authorize number of DOL Threads x 2 should be enough, each thread create a pool of memory, and it's not use to have too much threads and be bound to I/O.

MyISAM values are "conservative" they are not used except for performance_schema or information_schema.

Query cache need to accept big chunk of data as DOL do a lot of selecting entire tables. but it doesn't need to be so big much work come from inserts and updates...

Join buffer should be useless unless for maintenance query, I've never seen DOL doing a JOIN.

Config about innodb

Code: Select all

innodb_buffer_pool_size = 512M innodb_use_sys_malloc = 1 innodb_buffer_pool_instances = 2 # this is the default, increase it if you have lots of tables innodb_additional_mem_pool_size = 16M # the max is there to avoid run-away growth on your machine innodb_data_file_path = ibdata1:10M:autoextend:max:8G # we keep this at around 25% of of innodb_buffer_pool_size # sensible values range from 1MB to (1/innodb_log_files_in_group*innodb_buffer_pool_size) innodb_log_file_size = 256M # this is the default, increase it if you have very large transactions going on innodb_log_buffer_size = 16M # this is the default and won't hurt you # you shouldn't need to tweak it # see the innodb config docs, the other options are not always safe innodb_flush_log_at_trx_commit = 0 innodb_lock_wait_timeout = 50 innodb_flush_method=O_DIRECT

About buffers, don't give too much additional_mem_pool_size, this is for database with 500+ tables, don't give too much log_buffer_size, this is for transaction using a lot of data (DOL doesn't use Transaction from what I've seen)

The interesting part is at the end, flush_log_at_trx_commit if set to default will flush I/O after each implicit COMMIT (very bad with a lot of short queries)

Values at 0 or 2 delay flush but can be unsafe if MySQL or System Crash. innodb_flush_method and innodb_use_sys_malloc shouldn't do anything on a Windows server, it tries to use newer kernel hook for I/O.

[Blue]

We use:

Code: Select all

innodb_buffer_pool_size=3G

and this gives the most significant boost. We had especially problems with the huge inventoryitem table. At the moment its about 6 mio entries and 2.9GB around. And everytime a player connects it will query visible items of all chars from inventory. That always took a lot of time before. Now I guess most of the inventoryitem table sits in memory and its quite fast.

I also advise to convert all tables to InnoDB if enough memory is available as it will outperform in the end. But for using InnoDB plenty memory is absolutely critical.

http://www.mysqlperformanceblog.com/200 ... pool_size/