Help with data management

[Prever77]

Hi. Long story short.

I made year or two ago my own package of scripts for role play server, which i never started. 3 months ago i decided to rewrite everything. As i looked at it and said "what a mess. Who wrote this?" meanwhile i've gained experience in C and good habits, so choice to rewrite everything was obvious for me at that moment.

So i did basic optimisation. What can be stored and should be in bools is now stored in bools. not 1 and 20s lol. etc etc

But for now i hitted the wall. Which is named "to buffer. Or not to". By that i mean. My scripts are connected to outside database as built in database is.. well not so fast and secure. And i didn't used to do that mechanism in original code. But i think it's rational to store very often pulled data from database localy on server memory. And update tables in parts to database like every 30seconds. Also i want to use buffer data and database to validate what comes from client side. Im storing vehicles on map, their custom data etc etc.

And here's my main questions.

1.is it worth of effiord to make buffer if i plan 80-200 players on my server.

2.is validating client data with buffer and database worth concept

3.what are your coding advices for me?. Im Asking experienced ones that ruled servers with large player base. 

--[[btw i know that if server has 3 players buffer is not needed. Thanks, awesome advice.--]]

 

[IIYAMA]

17 hours ago, Prever77 said:

1.is it worth of effiord to make buffer if i plan 80-200 players on my server.

Buffers can be useful.
Because you can only pick a server with a good CPU once. Sometimes you can upgrade the CPU, but there is a limit.

• While with ram, it is easier to upgrade.
• MTA Lua processes are afaik still is single threaded.

But you need to invest a lot of time.

17 hours ago, Prever77 said:

2.is validating client data with buffer and database worth concept

I can be.

• An example db resource I made a while a go, not very efficient storage (text/string based), but easy to use.
  Not sure why I build the types text based, I would have build it differently now a days.
  • Also recommended to write automatic tests for your creation, like I did with mine in sub folder \dev\*.
• You can storage md5 hash validation strings for validation, if you are interested in that.
  Smaller hash storage [binary]: using the UNHEX(hash) function (convert back with HEX(binary).
  Example resource that uses hash validation for client screenshots.

 

17 hours ago, Prever77 said:

3.what are your coding advices for me?. Im Asking experienced ones that ruled servers with large player base. 

I am not one of them. But I do work often met databases.

I have put my recommendation in a spoiler if you are still interested:

Spoiler

While buffers can be useful.

• They can be also create bugs if not created correctly.
• Require lots of testing.
• But they do in fact make it easier to handle Async requests. Keeping your db from blocking you Lua processes.
  -> Using synced requests makes things easier, but could also create lag on your server.

There is 1 alternative to consider and that is to keep the active data in memory. While you do write to the db, you do not fetch data that you already know and keeping it around. This is a concept you could consider a buffer, the only difference is that you pre-load all the data you might need. And moving the 'truth' from the db to the memory when initialising. 

So when a player joins. You get his health, armor, money, house name,  name etc. etc. And put it in a table that represents his exact model in the db.
When values change, you only write to the db.

This keeps the data fast and persistent. But with the downside that if there is a bug in your code, your db becomes desynced with your memory. And when the player reconnects the 'truth' of the db leads again. It is important to mirror in Lua the database column limitations, which should solve most of those issues.

 

Last recommendation:

• Add rate limits for requests done by a client. Your db is one of your server it's weaknesses. (concept of a passive rate limit)

 

 

[Prever77]

Ty so much for reply.

Yes i had same thing in head about how buffer should look like. Changing only values that has changed etc. I didn't precised it well. By sending tables to database every 30s. i was thinking about mechanism that also stacks changed values to certain amount. Like old table and new table concept. Lets say im checking every 30s if cars changed their position. I compare old table with new. And values that has changed i move to some sort of stack. And stack is let say sent every 4 cycles of checking car positions. Before that stack is sent to database he's checked for latest values and reduced in size. 

By validating what comes from client i was thinking about putting clever anticheat trigger mechanism. Like while checking for changes. It's checking if that change was possible with my scripts for cars. Is car moving too fast or something. If it's. Then anticheat is triggered. and let me end here about anticheat mechanism. But ty also for mentioning your script. I'll take a look on it.

Also by outside database i was thinking about LAMP thing in LANetwork on the other physical device. As i wanna only let one device to talk with the world. Just to reduce risk. Im sharing same network so i need to keep packs small so i have transfer reserved for server talking to trustfull world. Not exact desciption on purpouse here.

Im having physical hardware so CPU and RAM are not problem. Problem is rational use of power. As i want to run my server independent. And made it most watt per player ratio effective. And keep it cheap. As im not planing earn money from server.

Sorry for not mentioning details, but i didn't knew how to describe it all earlier and now too. as english is not my great side. Also ty for yours recommendations. Especialy about last one i forgot about that nice feature 

EDIT ADD

Isn't it better for me to just make mta module that handle this stuff with multi core rather than single threaded lua? TheoreticaIy it should save me cycles on lua. And let them to be used more productive on bare bone server mechanics and scripting. And if so. what are good sources to begin with mta modules.

 

 

Edited 17 hours ago by Prever77

[IIYAMA]

10 hours ago, Prever77 said:

i was thinking about mechanism that also stacks changed values to certain amount. Like old table and new table concept

For some changes it is a good way to save resource. Like for example statistics or car fuel.

But do not do it for critical data. When for example your power shuts down, it could create weir de-syncs. (Like if you were buying a house ingame: you do not receive the house [in buffer] but the money has already been withdrawn [not in buffer])

 

10 hours ago, Prever77 said:

Isn't it better for me to just make mta module that handle this stuff with multi core rather than single threaded lua? TheoreticaIy it should save me cycles on lua. And let them to be used more productive on bare bone server mechanics and scripting. And if so. what are good sources to begin with mta modules.

You can use MySQL + dbConnect, instead of writing a custom module.
The current MySQL module available is blocking the CPU thread, so that is not really an option for 200 players in my opinion.

---

Also a way to save resources, is to enable multi_statements:

local connection = dbConnect("sqlite", "database/database.db", "", "", "multi_statements=1")

When for example if you want to remove data at multiple tables.

dbExec(connection,
		"DELETE FROM shared_memory_file WHERE clientId = ?;DELETE FROM shared_memory_frame WHERE clientId = ?;DELETE FROM shared_memory_frame_position WHERE clientId = ?",
		clientId, clientId, clientId)

Or get data from multiple tables:

dbQuery(processRequestSharedMemory, { player, clientId }, connection, [[
SELECT variantKey, item, fileData FROM shared_memory_file WHERE clientId = ?;
SELECT x, y, item FROM shared_memory_frame WHERE clientId = ?;
SELECT x, y, z FROM shared_memory_frame_position WHERE clientId = ? LIMIT 1
]],
clientId, clientId, clientId)

 

[Prever77]

4 hours ago, IIYAMA said:

For some changes it is a good way to save resource. Like for example statistics or car fuel.

But do not do it for critical data. When for example your power shuts down, it could create weir de-syncs. (Like if you were buying a house ingame: you do not receive the house [in buffer] but the money has already been withdrawn [not in buffer])

Ofc you are right. and this should always be good advice for everyone to not mess with critical data. i know it, and more programmers should know that. i like to organize everything in hierarchy of importance. like cars possitions can be lost if something happens wrong. as that's not so crucial to server working after shutdown. and in database are still hold relativeley "fresh" data, like vehicle possitions and so on. But money and housing systems for examples are 1st or 2nd importance. so it must be sure that after buying a house money amount is right and house owner is assigned afap in buffer and database.  and ofc validate everything.

4 hours ago, IIYAMA said:

You can use MySQL + dbConnect, instead of writing a custom module.
The current MySQL module available is blocking the CPU thread, so that is not really an option for 200 players in my opinion.

---

yeah i know that i can do that this way. but i was thinking about directly working with server objects. 

like buffermodule takes objects how they real look like. and scan only for lets say vehicles. and all previous ideas of how it will be implemented in lua. and make them in Cpp instead of lua. oldtable newtable etc. direct calling from module to built in dbFunctions should be faster in theory. and if i use some coroutines i can prevent that way server from lagging. but still keeping the lua code just to drive module. but i can mistaken somewhere and what i think can be fast. can also be not fast. on one side i can then keep everything hardly bounded together. but on the other side it can be error generating nightmare. what i know from experience in Cpp xD. and my question is. do i'm right about it? and if so.  are there good sources to start with it? cause personaly i didn't found anything more usefull on the internet than example module. and ofc mta blue repo. im having in mind some sort of documentation. cause looking at servers source code just to get an idea of how module should look like and work pushes me back even from trying. im doing it everyday, so getting point of my mates is my second nature. but mta blue server is a lot of code.  so i guess my natural questions are. if there is a chance that server can save 5% to 10% of power and space. is there any code documentation? so i can short my working time on module. ofc if im right that it can save cycles. i never wrote mta module, and didn't heard opinions of people who did.

maybe someone is reading this rn. please get in to discussion. you can help me and others just by getting in to discussion. 

EDIT

also thanks for earlier replying @IIYAMA 

Edited 2 hours ago by Prever77

[IIYAMA]

1 hour ago, Prever77 said:

and if i use some coroutines i can prevent that way server from lagging. but still keeping the lua code just to drive module. but i can mistaken somewhere and what i think can be fast. can also be not fast. on one side i can then keep everything hardly bounded together. but on the other side it can be error generating nightmare. what i know from experience in Cpp xD.

I have no experience with C++ myself nor with modules.
But I do know that Lua is fast enough for a basic gamemode. The moment C++ might become important is when you want to implement something like pathfinding.

 

1 hour ago, Prever77 said:

so i guess my natural questions are. if there is a chance that server can save 5% to 10% of power and space.

There are other things that can save you more resources. For example using less timers. Most of the time getTickCount / getRealTime is more than enough.

Or use as less as possible element data.

 

1 hour ago, Prever77 said:

is there any code documentation?

Sort of : https://wiki.multitheftauto.com/wiki/Modules_Introduction