Server crashing HELP

[IIYAMA]

6 hours ago, juaosilv said:

I sent all the log data in your private, there you can see the data they gave. With 15-20 the server started to slow down and crashing, I did not send here because it contains data from my users.

 

  ["nil/onElementDataChange/elem:vehicle[Police LV]0x11e5"] = 389,

["nil/onElementDataChange/elem:vehicle[Towtruck]0x123a"] = 400,

  ["nil/onElementDataChange/elem:vehicle[Hustler]0x11e0"] = 310,

  ["nil/onElementDataChange/elem:vehicle[Police LV]0x11e5"] = 524,

  ["nil/onElementDataChange/elem:vehicle[Towtruck]0x123a"] = 317,

["nil/onElementDataChange/elem:vehicle[Police LV]0x11e5"] = 956,

 

It looks like it is related to vehicles. Maybe a fuel resource? (Maybe stop that resource and see if it makes any difference)

 

I am currently very busy, so will help later further today.

 

Edited August 6, 2019 by IIYAMA

[IIYAMA]

 

    local eventList = {}

    setTimer(
      function () 
        if next(eventList) then
          outputDebugString(inspect(eventList))
          eventList = {}
        end
      end, 10000, 0
    )


    addDebugHook( "preEvent", 
      function ( sourceResource, eventName, eventSource, eventClient, luaFilename, luaLineNumber, ... )
    	local args = { ... }
        local i =  "resource: " .. (sourceResource and getResourceName(sourceResource) or "[unkown] ") .. ", key: " .. tostring(args[1]) .. ", file: " .. tostring(luaFilename) .. ", line-number: " .. tostring(luaLineNumber)
        eventList[i] = (eventList[i] and eventList[i] + 1) or 1
      end, {"onElementDataChange"}
    )

 

If this code works, then you can start fixing things.

@juaosilv

Edited August 6, 2019 by IIYAMA

[Addlibs]

Quote

["nil/onElementDataChange/elem:vehicle[Police LV]0x11e5"] = 389,
["nil/onElementDataChange/elem:vehicle[Towtruck]0x123a"] = 400,
["nil/onElementDataChange/elem:vehicle[Hustler]0x11e0"] = 310,

These seem to be what's causing you most of the lag. However, these changes don't seem to be associated with any resource. I'd guess that means they're triggered remotely from a client. My bet this is some sort of fuel system, and it probably has every client send fuel updates instead of only the driver and/or element syncer.

---

If you're running the server on 1.5.5 r11856 or above, the debug hook should have more parameters to help narrow down where it's coming from:

resource eventResource, string eventName, element eventSource, element eventClient, string eventFilename, int eventLineNumber, resource functionResource, string functionFilename, int functionLineNumber, ...eventArgs

if you use preEventFunction instead of preEvent debug hook.

Pretty sure functionResource would be the resource of the function that handles the event, which should be more reliable (perhaps?) than eventResource. So you could incorporate

inspect(functionResource)
-- or
getResourceName(functionResource)

into one of the debug hook codes above, such as

local eventList = {}

setTimer(
  function () 
    if next(eventList) then
      outputDebugString(inspect(eventList))
      eventList = {}
    end
  end, 10000, 0
)

addDebugHook( "preEventFunction", 
  function(eventResource, eventName, eventSource, eventClient, eventFilename, eventLineNumber, functionResource, functionFilename, functionLineNumber, ...)
    local args = { ... }
    local i = "resource: " .. (eventResource and getResourceName(eventResource) or "[unkown]") .. ", handled by: " .. (functionResource and getResourceName(functionResource) or "[unkown]") .. ", key: " .. tostring(args[1]) .. ", file: " .. tostring(functionFilename) .. ", line-number: " .. tostring(functionLineNumber)
    eventList[i] = (eventList[i] and eventList[i] + 1) or 1
  end, {"onElementDataChange"}
)

 

Edited August 6, 2019 by MrTasty

[juaosilv]

7 hours ago, MrTasty said:

These seem to be what's causing you most of the lag. However, these changes don't seem to be associated with any resource. I'd guess that means they're triggered remotely from a client. My bet this is some sort of fuel system, and it probably has every client send fuel updates instead of only the driver and/or element syncer.

---

If you're running the server on 1.5.5 r11856 or above, the debug hook should have more parameters to help narrow down where it's coming from:

resource eventResource, string eventName, element eventSource, element eventClient, string eventFilename, int eventLineNumber, resource functionResource, string functionFilename, int functionLineNumber, ...eventArgs

if you use preEventFunction instead of preEvent debug hook.

Pretty sure functionResource would be the resource of the function that handles the event, which should be more reliable (perhaps?) than eventResource. So you could incorporate

inspect(functionResource)
-- or
getResourceName(functionResource)

into one of the debug hook codes above, such as

local eventList = {}

setTimer(
  function () 
    if next(eventList) then
      outputDebugString(inspect(eventList))
      eventList = {}
    end
  end, 10000, 0
)

addDebugHook( "preEventFunction", 
  function(eventResource, eventName, eventSource, eventClient, eventFilename, eventLineNumber, functionResource, functionFilename, functionLineNumber, ...)
    local args = { ... }
    local i = "resource: " .. (eventResource and getResourceName(eventResource) or "[unkown]") .. ", handled by: " .. (functionResource and getResourceName(functionResource) or "[unkown]") .. ", key: " .. tostring(args[1]) .. ", file: " .. tostring(functionFilename) .. ", line-number: " .. tostring(functionLineNumber)
    eventList[i] = (eventList[i] and eventList[i] + 1) or 1
  end, {"onElementDataChange"}
)

 

 

With this code how do I know which feature is bad?

["resource: ap_items, handled by: admin, key: trashID, file: admin_server.Lua,
 line-number: 1561"] = 1, <= When is this number high?

[IIYAMA]

22 minutes ago, juaosilv said:

["resource: ap_items, handled by: admin, key: trashID, file: admin_server.Lua,
 line-number: 1561"] = 1, <= When is this number high?

Yes, when that number exceeds around 20.

20 = overall 2 updates per second. Which is not good, because it is not only 2 updates, it is: 2 X playerCount = impact on the server network.

100 = bad

300 = really bad

1000 = I am not joining your server for sure

Edited August 6, 2019 by IIYAMA

[juaosilv]

3 hours ago, IIYAMA said:

Yes, when that number exceeds around 20.

20 = overall 2 updates per second. Which is not good, because it is not only 2 updates, it is: 2 X playerCount = impact on the server network.

100 = bad

300 = really bad

1000 = I am not joining your server for sure

Quote

[19-08-06 18:57:20] INFO: {
  ["resource: [unkown], handled by: admin, key: active:itemID, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: active:itemSlot, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.showedPlayerInventory, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemID, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemSlot, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.showedPlayerInventory, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_nametag, handled by: admin, key: nameEnabled, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_nametag, handled by: admin, key: visibleName, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_nametag, handled by: ap_detran, key: nameEnabled, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_nametag, handled by: ap_detran, key: visibleName, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_samucair, handled by: admin, key: data.playerID, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_samucair, handled by: ap_detran, key: data.playerID, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_uptime, handled by: admin, key: Tempo, file: admin_server.Lua, line-number: 1561"] = 29,
  ["resource: ap_uptime, handled by: ap_detran, key: Tempo, file: sFulano.Lua, line-number: 29"] = 29

Quote

[19-08-06 18:59:00] INFO: {
  ["resource: [unkown], handled by: admin, key: active:itemID, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: active:itemSlot, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.afk, file: admin_server.Lua, line-number: 1561"] = 2,
  ["resource: [unkown], handled by: admin, key: char.console, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.showedPlayerInventory, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.typing, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: hunger, file: admin_server.Lua, line-number: 1561"] = 4,
  ["resource: [unkown], handled by: admin, key: price, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: sede, file: admin_server.Lua, line-number: 1561"] = 4,
  ["resource: [unkown], handled by: ap_detran, key: active:itemID, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemSlot, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.afk, file: sFulano.Lua, line-number: 29"] = 2,
  ["resource: [unkown], handled by: ap_detran, key: char.console, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.showedPlayerInventory, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.typing, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: hunger, file: sFulano.Lua, line-number: 29"] = 4,
  ["resource: [unkown], handled by: ap_detran, key: price, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: sede, file: sFulano.Lua, line-number: 29"] = 4,
  ["resource: ap_nametag, handled by: admin, key: nameEnabled, file: admin_server.lu[19-08-06 18:59:10] INFO: {
  ["resource: [unkown], handled by: admin, key: afk, file: admin_server.Lua, line-number: 1561"] = 2,
  ["resource: [unkown], handled by: admin, key: char.afk, file: admin_server.Lua, line-number: 1561"] = 3,
  ["resource: [unkown], handled by: admin, key: hunger, file: admin_server.Lua, line-number: 1561"] = 3,
  ["resource: [unkown], handled by: admin, key: sede, file: admin_server.Lua, line-number: 1561"] = 3,
  ["resource: [unkown], handled by: ap_detran, key: afk, file: sFulano.Lua, line-number: 29"] = 2,
  ["resource: [unkown], handled by: ap_detran, key: char.afk, file: sFulano.Lua, line-number: 29"] = 3,
  ["resource: [unkown], handled by: ap_detran, key: hunger, file: sFulano.Lua, line-number: 29"] = 3,
  ["resource: [unkown], handled by: ap_detran, key: sede, file: sFulano.Lua, line-number: 29"] = 3,
  ["resource: ap_uptime, handled by: admin, key: Tempo, file: admin_server.Lua, line-number: 1561"] = 100,
  ["resource: ap_uptime, handled by: ap_detran, key: Tempo, file: sFulano.Lua, line-number: 29"] = 100
}

Quote

[19-08-06 18:59:30] INFO: {
  ["resource: [unkown], handled by: admin, key: char.afk, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.typing, file: admin_server.Lua, line-number: 1561"] = 2,
  ["resource: [unkown], handled by: ap_detran, key: char.afk, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.typing, file: sFulano.Lua, line-number: 29"] = 2,
  ["resource: ap_painel-samu, handled by: admin, key: KitMedico, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_painel-samu, handled by: ap_detran, key: KitMedico, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_police, handled by: admin, key: player:Algemado, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_police, handled by: admin, key: player:Preso, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_police, handled by: ap_detran, key: player:Algemado, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_police, handled by: ap_detran, key: player:Preso, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_save, handled by: admin, key: Action:Background, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Action:Pro, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Items:Agua, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Items:Hamburguer, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Items:HotDog, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Items:MedicKit, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Items:Pizza, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key: Items:Refrigerante, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: ap_save, handled by: admin, key:[19-08-06 18:59:40] INFO: {
  ["resource: [unkown], handled by: admin, key: active:itemID, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: active:itemSlot, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.afk, file: admin_server.Lua, line-number: 1561"] = 6,
  ["resource: [unkown], handled by: admin, key: char.console, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.showedPlayerInventory, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: char.typing, file: admin_server.Lua, line-number: 1561"] = 4,
  ["resource: [unkown], handled by: admin, key: price, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemID, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemSlot, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.afk, file: sFulano.Lua, line-number: 29"] = 6,
  ["resource: [unkown], handled by: ap_detran, key: char.console, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.showedPlayerInventory, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: char.typing, file: sFulano.Lua, line-number: 29"] = 4,
  ["resource: [unkown], handled by: ap_detran, key: price, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: ap_uptime, handled by: admin, key: Tempo, file: admin_server.Lua, line-number: 1561"] = 100,
  ["resource: ap_uptime, handled by: ap_detran, key: Tempo, file: sFulano.Lua, line-number: 29"] = 100

Quote

19-08-06 19:03:41] INFO: {
  ["resource: [unkown], handled by: admin, key: active:itemID, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: active:itemSlot, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: afk, file: admin_server.Lua, line-number: 1561"] = 4,
  ["resource: [unkown], handled by: admin, key: char.afk, file: admin_server.Lua, line-number: 1561"] = 11,
  ["resource: [unkown], handled by: admin, key: char.showedPlayerInventory, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: fuel, file: admin_server.Lua, line-number: 1561"] = 3772,
  ["resource: [unkown], handled by: admin, key: hunger, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: admin, key: i:left, file: admin_server.Lua, line-number: 1561"] = 5,
  ["resource: [unkown], handled by: admin, key: i:right, file: admin_server.Lua, line-number: 1561"] = 5,
  ["resource: [unkown], handled by: admin, key: player:Voice, file: admin_server.Lua, line-number: 1561"] = 54,
  ["resource: [unkown], handled by: admin, key: sede, file: admin_server.Lua, line-number: 1561"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemID, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: active:itemSlot, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: afk, file: sFulano.Lua, line-number: 29"] = 4,
  ["resource: [unkown], handled by: ap_detran, key: char.afk, file: sFulano.Lua, line-number: 29"] = 11,
  ["resource: [unkown], handled by: ap_detran, key: char.showedPlayerInventory, file: sFulano.Lua, line-number: 29"] = 1,
  ["resource: [unkown], handled by: ap_detran, key: fuel, file: sFulano.Lua, line-number: 29"] = 3772,
  ["resource: [unkown], handled by: ap_detran, key: hunger, file: sFulano.Lua, line-number: 29"] = 1,

This ap_uptime is lagging, I could identify but the ones that have "unkown" as I can solve the problem without knowing the resource?

[Addlibs]

This is definitely (part of) the problem:

Quote

["resource: [unkown], handled by: admin, key: fuel, file: admin_server.Lua, line-number: 1561"] = 3772,
["resource: [unkown], handled by: ap_detran, key: fuel, file: sFulano.Lua, line-number: 29"] = 3772,

This is definitely your fuel system, since we're looking at 3.7k data changes per 10 seconds (6 updates per frame if you run 60 fps, more if lower fps). I can't narrow down exactly which resource it is (but you should probably know) nor which lines specifically, but you could share parts of the fuel system code which call setElementData for key "fuel" so we can help you reduce this number.

Edited August 7, 2019 by MrTasty

[juaosilv]

1 hour ago, MrTasty said:

This is definitely (part of) the problem:

This is definitely your fuel system, since we're looking at 3.7k data changes per 10 seconds (6 updates per frame if you run 60 fps, more if lower fps). I can't narrow down exactly which resource it is (but you should probably know) nor which lines specifically, but you could share parts of the fuel system code which call setElementData for key "fuel" so we can help you reduce this number.

Later I will look calmly and send here.

What is the average number a script can reach?

From what I saw, there are some standard MTA scripts above "20", what would be the maximum value below "100" that a script can reach?

For me to have an idea if I need to fix scripts that use 20-30.

[Addlibs]

20 or 30 would mean 2 and 3 Hz update rate respectively. These don't sound bad. I'd assume they don't need fixing. What needs fixing are resources that calling between 300 and 600 times a second per client, as that means they're syncing element data every frame, which is very very rarely necessary, and even when so, sync itself is often unnecessary. If you know your element data is consistent between client and server (that is, no client changes the data locally only or the server on server only, for example) than you should be able to use a simple guard

if getElementData(elem, "key") ~= intended_value then
  setElementData(elem, "key", intended_value)
end

If they are out of sync, they probably don't need syncing or your scripts are badly designed. If they don't need sync you should use

setElementData(elem, "key", intended_value, false)

However, if you're getting, say, 35 clients each sending 6 Hz updates, the server is receiving 210 updates a second on the server. This number need to be flexible. It must be able to accommodate your server player capacity (player slots), otherwise during peak hours when there's more people, there's more lag.

Edited August 7, 2019 by MrTasty