Lag issues

[ciclano]

Currently I have been suffering with lag problems, delay to open panels, voice cutting. The only things that are used a lot in the performancebrowser is the voice, which is constantly and a group script that I use but it is a matter of seconds and back to normal.

Is voice causing this problem of voice locking and the delay to open panels, send messages and others?

 

netstat:

 

Edited November 10, 2020 by ciclano

[IIYAMA]

26 minutes ago, ciclano said:

Is voice causing this problem of voice locking and the delay to open panels, send messages and others?

ah sorry, I forgot to reply to your previous topic.

It is not very complex to test that assumption, just disable voice and check if the problem still occurs. (fastest way would be out-comment all export functions in the meta.xml of the voice resource, you will see a lot of errors, but it will do it's job)

<!-- -->

---

If your assumption is correct, then still that doesn't mean that the resource is fully responsible. Since voice is just an API, that receive instructions from another resource. That will be a second objective in that case.

 

Edited November 10, 2020 by IIYAMA

[ciclano]

4 minutes ago, IIYAMA said:

ah sorry, I forgot to reply to your previous topic.

It is not very complex to test that assumption, just disable voice and check if the problem still occurs. (fastest way would be out-comment all export functions in the meta.xml of the voice resource, you will see a lot of errors, but it will do it's job)

<!-- -->

---

If your assumption is correct, then still that doesn't mean that the resource is fully responsible. Since voice is just an API, that receive instructions from another resource. That will be a second objective in that case.

 

The problem is that if I disable the voices of all players, they will become active. Which ends up causing a general problem

If it is the voice, what solution for that?Is there a function in the performancebrowser that I can be sure is the problem?

Edited November 10, 2020 by ciclano

[IIYAMA]

11 hours ago, ciclano said:

If it is the voice, what solution for that?Is there a function in the performancebrowser that I can be sure is the problem?

There is.

 

You can add a debug hook.

https://wiki.multitheftauto.com/wiki/AddDebugHook

 

This can be used to monitor the activity of MTA functions and events.

Attach the hook to (in the voice resource):

• https://wiki.multitheftauto.com/wiki/SetPlayerVoiceBroadcastTo
• https://wiki.multitheftauto.com/wiki/SetPlayerVoiceIgnoreFrom

Monitor the activity of all functions every 1min. Then log the result and restart counting. Do this for 30 mins.

 

If the activity is for example around than 100 (per 1 min). Then you need to monitor a bit more.

If the activity is higher than 1000 (per 1 min), you know something is wrong.

---

Your developer must be able to do this.

 

Edited November 11, 2020 by IIYAMA

[ciclano]

6 hours ago, IIYAMA said:

There is.

 

You can add a debug hook.

https://wiki.multitheftauto.com/wiki/AddDebugHook

 

This can be used to monitor the activity of MTA functions and events.

Attach the hook to (in the voice resource):

• https://wiki.multitheftauto.com/wiki/SetPlayerVoiceBroadcastTo
• https://wiki.multitheftauto.com/wiki/SetPlayerVoiceIgnoreFrom

Monitor the activity of all functions every 1min. Then log the result and restart counting. Do this for 30 mins.

 

If the activity is for example around than 100 (per 1 min). Then you need to monitor a bit more.

If the activity is higher than 1000 (per 1 min), you know something is wrong.

---

Your developer must be able to do this.

 

The voice problem was solved, it was one of the problems that was causing this delay on my server. Now there is only one script, and the same I believe that there are not many ways to solve it.

I will explain how it works, it is a group system it saves information about groups and their players in a table. The problem is that I have many players and many groups, which ends up causing the execution of "triggerClientEvent" and "triggerServerEvent" at all times, these executions are most often sending data from the group to some table.

Is there another way to do this? I see that scripts that save data in tables using a lot in the performancebrowser, most of the time when the script is sending a data to the table the use in performance / ipb is very high, I use tables to save data because it is better than "elementData "but apparently the performance values are similar to when I used elementData.

[IIYAMA]

3 hours ago, ciclano said:

Is there another way to do this? I see that scripts that save data in tables using a lot in the performancebrowser, most of the time when the script is sending a data to the table the use in performance / ipb is very high, I use tables to save data because it is better than "elementData "but apparently the performance values are similar to when I used elementData.

Using triggerClientEvent/triggerServerEvent instead of elementdata doesn't reduce network usage while using them the exact same way.

In some cases it will be come worse or it might also improve depending on the context.

 

trigger(Client/Server)Event is like an 1 way trip. That starts at 1 side and stops at 1 side.

• This trip uses more data than element data, for each client and server.
• But when some clients are excluded from this trip, data reduction can occur.
  • Like triggerServerEvent(1 client  to server)
  • or triggerClientEvent(send to a % of clients, instead of all)

 

When using elementData, you are enhancing an element with a specific property. The best practice of using this is to inform all clients about this specific property.

• This trip uses less data than trigger(client/server)Event, for each client and server.
• Before the latest update there was no data reduction possible, now there is:
  • https://wiki.multitheftauto.com/wiki/AddElementDataSubscriber
  • Before the addElementDataSubscriber, we had no way to stop the data from spreading to all clients and server. Each time the data was set on either side, all clients and the server were busy receive and process the data.
• It has a benefit of auto sharing the information new joiners. (this can also be seen as a downside)

 

Nice to know, but it is not just picking the right tool for the right job.

You need to reduce your data rate. A double player count shouldn't double the activity of the triggerClientEvent function. The data may scale, but the rate should remain the same.

 

 

 

 

 

Edited November 11, 2020 by IIYAMA

[Tekken]

So we can do addElementDataSubscriber

addElementDataSubscriber(player, "data", player); 

And this will only be synced for the specific player?

What will be the disadvantages ?

[IIYAMA]

49 minutes ago, Tekken said:

What will be the disadvantages ?

- It requires a lot of management.

- Missing documentation, examples and benchmarks. (So I can't you all of them yet, but maybe you can give them to us by trying the functions out)

52 minutes ago, Tekken said:

And this will only be synced for the specific player?

You probably have to unsubscribe all the other players to make that work.

[ciclano]

4 hours ago, IIYAMA said:

- It requires a lot of management.

- Missing documentation, examples and benchmarks. (So I can't you all of them yet, but maybe you can give them to us by trying the functions out)

You probably have to unsubscribe all the other players to make that work.

I am studying about, but there is little wiki. Also, is there no other solution for me to save this data without running triggerServerEvent and triggerClientEvent at all times?

 

Edit: Something that works on both client-side and server-side

Edited November 12, 2020 by ciclano

[IIYAMA]

1 hour ago, ciclano said:

I am studying about, but there is little wiki. Also, is there no other solution for me to save this data without running triggerServerEvent and triggerClientEvent at all times?

Some data doesn't have to be updated every time.

Like: playTime

 

Server sends it to the client 1x (or maybe once per 30 mins).

Then each client keeps per player the following data:

{
  [player] = {
      serverPlayTime = 1453436465,
      clientPlayTimeStart = getTickCount()
  }
}

With that, at any moment the client can compute the playTime of itself and other players, without the need of a continuous synchronization rate with the server.

local playTime = serverPlayTime + (getTickCount() - clientPlayTimeStart)

There might occur some small differences over an amount of time (0.1 > 1 seconds ???), because the operation and execution time may differ: client VS server

But that is just a little sacrifice for an enormous performance boost.

 

This can also be applied to values like: "hunger", "time cycles", "health restore". (some values require a correction from serverside)

Every update that has a continuous value change without randomization, can be optimised like that.

 

 

 

 

Edited November 12, 2020 by IIYAMA

[ciclano]

48 minutes ago, IIYAMA said:

Some data doesn't have to be updated every time.

Like: playTime

 

Server sends it to the client 1x (or maybe once per 30 mins).

Then each client keeps per player the following data:

{
  [player] = {
      serverPlayTime = 1453436465,
      clientPlayTimeStart = getTickCount()
  }
}

With that, at any moment the client can compute the playTime of itself and other players, without the need of a continuous synchronization rate with the server.

local playTime = serverPlayTime + (getTickCount() - clientPlayTimeStart)

There might occur some small differences over an amount of time (0.1 > 1 seconds ???), because the operation and execution time may differ: client VS server

But that is just a little sacrifice for an enormous performance boost.

 

This can also be applied to values like: "hunger", "time cycles", "health restore". (some values require a correction from serverside)

Every update that has a continuous value change without randomization, can be optimised like that.

 

 

 

 

I do it like this, the big question is that it is a group system so whenever someone leaves a group, they log into the server and etc. the server needs to update the group data.

So, in addition, he saves the groups in a table, so whenever someone logs in he performs the function of saving the list of groups in the table. At the end of the day he needs to do this but it ends up causing self-use in IPB since the server has a lot of players, every time someone updates a group and someone enters the server.

 

The issue of sending groups to the client-side is necessary since I use shared functions to pull information from the player, if he is a member of the group, if he is a leader. So

 

Before I set an elementData, "player.groups" so I could pull the player's group information by both client-side and server-side. But for tables I have to send to the client and pro server, so they are trigger running at all times, both on the server and on the client.

I could use elementData, but it would also continue to lag.

Edited November 12, 2020 by ciclano

[IIYAMA]

23 minutes ago, ciclano said:

But for tables I have to send to the client and pro server, so they are trigger running at all times, both on the server and on the client.

Have you considered to send only the groupID and let the client remove the data by itself?

 

[ciclano]

Just now, IIYAMA said:

Have you considered to send only the groupID and let the client remove the data by itself?

 

Already, but how would I do to pull the group information from the client, if they are saved only on the server.

Does the amount of data sent to the client have any difference in the use of the script?

[IIYAMA]

1 minute ago, ciclano said:

Already, but how would I do to pull the group information from the client, if they are saved only on the server.

Caching on clientside is important, that way serverside doesn't have to send data twice or send large amount of data.

 

1 minute ago, ciclano said:

Does the amount of data sent to the client have any difference in the use of the script?

Yes, it takes longer to send the data and therefore other messages have to wait longer. (like opening a panel)

 

[ciclano]

7 minutes ago, IIYAMA said:

Caching on clientside is important, that way serverside doesn't have to send data twice or send large amount of data.

 

Yes, it takes longer to send the data and therefore other messages have to wait longer. (like opening a panel)

 

Is there a way for me to pull just that script which is causing more lag?

I believe that the big problem is not this sending of data, because it is not always that they are using the script.

Edited November 12, 2020 by ciclano

[IIYAMA]

1 minute ago, ciclano said:

Is there a way for me to pull just that script which is causing more lag?

Something like this: (untested)

local eventUsage = {}

function onPreEvent( sourceResource, eventName, eventSource, eventClient, luaFilename, luaLineNumber, ... )
    local args = { ... }
    local srctype = eventSource and getElementType(eventSource)
    local resname = sourceResource and getResourceName(sourceResource)
    local plrname = eventClient and getPlayerName(eventClient)
  	if not eventUsage[eventName] then
    	eventUsage[eventName] = {}
    end
    eventUsage[eventName][#eventUsage[eventName] + 1] = "preEvent" .. " " .. tostring(resname) .. " " .. tostring(eventName) .. " source:" .. tostring(srctype) .. " player:" .. tostring(plrname) .. " file:" .. tostring(luaFilename) .. "(" .. tostring(luaLineNumber) .. ")"
 .. " numArgs:" .. tostring(#args)
 .. " arg1:" .. tostring(args[1]);
  
end
addDebugHook( "preEvent", onPreEvent )


setTimer(function () {
    local eventUsageSorted = {}
	for eventName, eventData in pairs (eventUsage) do
        eventUsageSorted[#eventUsageSorted + 1] = {eventName = eventName, eventData = eventData}
	end
	table.sort(eventUsageSorted, function(a, b) return #a.eventData > #b.eventData end)
      
	iprint("1 = ", eventUsageSorted[1].eventName, "\n2 = ",eventUsageSorted[2].eventName, "\n3 = ", eventUsageSorted[3].eventName)

    eventUsage = {}
}, 60000, 0)

https://wiki.multitheftauto.com/wiki/AddDebugHook

 

 

 

 

[ciclano]

14 minutes ago, IIYAMA said:

Something like this: (untested)

local eventUsage = {}

function onPreEvent( sourceResource, eventName, eventSource, eventClient, luaFilename, luaLineNumber, ... )
    local args = { ... }
    local srctype = eventSource and getElementType(eventSource)
    local resname = sourceResource and getResourceName(sourceResource)
    local plrname = eventClient and getPlayerName(eventClient)
  	if not eventUsage[eventName] then
    	eventUsage[eventName] = {}
    end
    eventUsage[eventName][#eventUsage[eventName] + 1] = "preEvent" .. " " .. tostring(resname) .. " " .. tostring(eventName) .. " source:" .. tostring(srctype) .. " player:" .. tostring(plrname) .. " file:" .. tostring(luaFilename) .. "(" .. tostring(luaLineNumber) .. ")"
 .. " numArgs:" .. tostring(#args)
 .. " arg1:" .. tostring(args[1]);
  
end
addDebugHook( "preEvent", onPreEvent )


setTimer(function () {
    local eventUsageSorted = {}
	for eventName, eventData in pairs (eventUsage) do
        eventUsageSorted[#eventUsageSorted + 1] = {eventName = eventName, eventData = eventData}
	end
	table.sort(eventUsageSorted, function(a, b) return #a.eventData > #b.eventData end)
      
	iprint("1 = ", eventUsageSorted[1].eventName, "\n2 = ",eventUsageSorted[2].eventName, "\n3 = ", eventUsageSorted[3].eventName)

    eventUsage = {}
}, 60000, 0)

https://wiki.multitheftauto.com/wiki/AddDebugHook

 

 

 

 

In a period of 5 minutes he had several executions of the function of picking up the groups. Really what is causing me problems is the way these groups are managed.

The big question is how can I re-do this, the function that was most performed is the "requestGroups" ( https://prnt.sc/vi33zp ), it basically sends a request to the server by the groups and it returns with the list of groups ( https://prnt.sc/vi34cz ) that are loaded as soon as the script is started. In addition, it also updates the player's groups ( https://prnt.sc/vi34ra ), sending the groups he is part of to the client and saving the groups he is part of on the server.  

I save both on the client and on the server because I need to pull information through other scripts using "exports", and I pull on both the client and the server. If I add only on the server and try to pull this information with exports on a client, I can't get this information.

I've already made several changes to make it as light as possible, as groups only load if the player presses F5 and several others in order to make the script lighter. But I don't know what I can do other than that, since I need to save this data anyway.

Edited November 12, 2020 by ciclano

[IIYAMA]

2 minutes ago, ciclano said:

In a period of 5 minutes he had several executions of the function of picking up the groups. Really what is causing me problems is the way these groups are managed.

What is the quantity?

 

3 minutes ago, ciclano said:

The big question is how can I re-do this

You can queue it (buffering it).

------------------------

Is there a request?

Wait 0,3/0.5 seconds + put the requests in a table.

Then send it.

 

While putting it in the table, make sure to only send the latest groups-list to each player.

 

 

 

 

 

 

 

[ciclano]

6 minutes ago, IIYAMA said:

What is the quantity?

 

You can queue it (buffering it).

 

 

 

 

 

 

 

Because the server has 200 online, it had 300 executions

EDIT: It is worth remembering that the 300 executions include not only requesting the groups but also changes in the members and the group in general.

I will go for a while to perform these executions, for example, if a player pressed a button to load the groups, the others must wait 5 seconds to load the groups.

This should already help, but I believe the problem will remain.

 

In addition to this requestGroups function, I have other functions that perform executions when a player is removed from a group, or the group is updated as is:

https://prnt.sc/vi3fw6

https://prnt.sc/vi3g76

https://prnt.sc/vi3gb1

https://prnt.sc/vi3gr0

I believe that at this point there is no problem, but what would be your opinion. Could it cause any instability?

Edited November 12, 2020 by ciclano

[IIYAMA]

2 minutes ago, ciclano said:

Could it cause any instability?

The database can also be an factor.

Which fields have you already optimised through applying indexes?

[ciclano]

3 minutes ago, IIYAMA said:

The database can also be an factor.

Which fields have you already optimised through applying indexes?

This now to request the groups, I put a general delay. If one player opens the other, he will only be able to open it 8 seconds later.

I removed the functions of loading groups when a player logs in.

And I removed elementData by saving everything in tables.

But still the use is a little high, the problem is that the use is not constant but when I have many players online between 250-300 it gets to use 70% of the IPB and falls, but it keeps going up all the time.

Another problem that may be causing lag (https://prnt.sc/vi3p2s) whenever a group is modified all players on the server have their table on the "client-side" updated.

These executions for everyone only happen when an ADMIN creates or modifies a group.

Another doubt, I see this high usage in the performancebrowser but only in the server-side tab, it means that the problem comes from there or that it is going there

[IIYAMA]

11 minutes ago, ciclano said:

But still the use is a little high, the problem is that the use is not constant but when I have many players online between 250-300 it gets to use 70% of the IPB and falls, but it keeps going up all the time.

 

Database

Are all database requests a-synced? < (check that)

Experiment with indexes: https://www.youtube.com/watch?v=uyLy462Fmk8

At limits for search queries:

LIMIT 1

 

triggerClientEvent - personalize the data

Combine those related to the group system and only trigger over 1 event name.

Combine all triggered events that are executed within 1 function -> In that 1 event, you can give all kinds of instructions that have to be done with the group system.

You probably have to create a new file for how that is going to work.

 

 

 

 

 

[ciclano]

1 hour ago, IIYAMA said:

 

Database

Are all database requests a-synced? < (check that)

Experiment with indexes: https://www.youtube.com/watch?v=uyLy462Fmk8

At limits for search queries:

LIMIT 1

 

triggerClientEvent - personalize the data

Combine those related to the group system and only trigger over 1 event name.

Combine all triggered events that are executed within 1 function -> In that 1 event, you can give all kinds of instructions that have to be done with the group system.

You probably have to create a new file for how that is going to work.

 

 

 

 

 

Thank you very much for the tips, I managed to solve the problem of this script. But the delay on the server still continued, it is not something that always happens but when it happens it lasts several minutes.

Is there any way I can find out what is causing this or if it can be my vps?

There is nothing high about IPB

https://prnt.sc/vi5h1s

https://prnt.sc/vi5jwn

Edited November 12, 2020 by ciclano

[ciclano]

1 hour ago, IIYAMA said:

 

Database

Are all database requests a-synced? < (check that)

Experiment with indexes: https://www.youtube.com/watch?v=uyLy462Fmk8

At limits for search queries:

LIMIT 1

 

triggerClientEvent - personalize the data

Combine those related to the group system and only trigger over 1 event name.

Combine all triggered events that are executed within 1 function -> In that 1 event, you can give all kinds of instructions that have to be done with the group system.

You probably have to create a new file for how that is going to work.

 

 

 

 

 

Apparently the higher the Logic Thread CPU the longer the delay, but I'm not sure.

https://prnt.sc/vi5u61

I don't know if this is normal, but the MTA processes on my VPS are many.

Edited November 12, 2020 by ciclano

[IIYAMA]

22 minutes ago, ciclano said:

Is there any way I can find out what is causing this or if it can be my vps?

You could check if there is a player who thinks it is funny to spam a command, so that your server starts to lag (no joke, when no limit is applied, it is an absolute vulnerability.):

https://wiki.multitheftauto.com/wiki/OnPlayerCommand

 

---

 

You could rent another VPS and see if that one performs better. Since it is a VPS and not a dedicated server, you will not be able to monitor the global statistics. If your neighbour uses a lot of data,CPU etc., you might not be able to notice that through your own server statistics, while still receiving bad performance.