Sync fetchRemote

[uriid1]

I want to achieve synchronous behavior for the fetchRemote function. However, instead, I'm getting an infinite loop with the while statement. Could it be because MTA uses different addressing?
 

local expectedResponse = {}

function fetchRemoteCallback(responseData)
  table.insert(expectedResponse, responseData)
end

function fetchRemoteAsync(url, opts)
  expectedResponse = {}
  fetchRemote(url, opts, fetchRemoteCallback)
end

function waitForResponse()
  while not expectedResponse[1] do
    -- Wait...
  end
end

fetchRemoteAsync('http://site.com/foo/bar', { method = 'POST', headers = ... })
waitForResponse()

if expectedResponse[1] then
  return expectedResponse[1]
end

 

Edited September 8, 2023 by uriid1

[DiSaMe]

I don't know what you're calling "different addressing". The reason it enters an infinite loop is because multiple threads cannot execute code in the same Lua state at the same time. So the callback cannot stop the loop because the loop needs to end to allow the callback to execute in the first place.

And even if that wasn't a problem, there's another one: Lua executes synchronously with the rest of the stuff in the server, so even if it wasn't an infinite loop, it would still be blocking the server from updating other things until the response from remote server arrives.

Instead of calling fetchRemote synchronously, you probably want to execute your own Lua code asynchronously, over multiple Lua invocations. You can achieve that using coroutines, which allow the function execution to stop and resume later.

An example:

function fetchRemoteAsync(url, opts)
  local co = coroutine.running() -- Store the current coroutine in a variable

  local function callback(responseData, errorCode)
    -- When the response arrives, resume the coroutine stored in variable co. Also
    -- pass responseData and errorCode as values to be returned by coroutine.yield.
    coroutine.resume(co, responseData, errorCode)
  end

  fetchRemote(url, opts, callback)

  -- Pause the execution of the current coroutine. It will be
  -- resumed when coroutine.resume gets called for this coroutine
  -- and any additional arguments will be returned.
  local responseData, errorCode = coroutine.yield()

  -- The coroutine has resumed, return the data.
  return responseData, errorCode
end

function testFetchAndOutput()
  local responseData, errorCode = fetchRemoteAsync('http://site.com/foo/bar', { method = 'POST', headers = ... })
  outputDebugString(responseData)
end

-- Create a coroutine that will execute testFetchAndOutput
local testCo = coroutine.create(testFetchAndOutput)

-- Execute the coroutine
coroutine.resume(testCo)

As a result, fetchRemoteAsync isn't blocking so you can call it and everything else on the server (including Lua code) can execute before it returns. But it can only be called from a coroutine.

Edited September 8, 2023 by DiSaMe

[uriid1]

1 hour ago, DiSaMe said:

I don't know what you're calling "different addressing". The reason it enters an infinite loop is because multiple threads cannot execute code in the same Lua state at the same time. So the callback cannot stop the loop because the loop needs to end to allow the callback to execute in the first place.

And even if that wasn't a problem, there's another one: Lua executes synchronously with the rest of the stuff in the server, so even if it wasn't an infinite loop, it would still be blocking the server from updating other things until the response from remote server arrives.

Instead of calling fetchRemote synchronously, you probably want to execute your own Lua code asynchronously, over multiple Lua invocations. You can achieve that using coroutines, which allow the function execution to stop and resume later.

An example:

function fetchRemoteAsync(url, opts)
  local co = coroutine.running() -- Store the current coroutine in a variable

  local function callback(responseData, errorCode)
    -- When the response arrives, resume the coroutine stored in variable co. Also
    -- pass responseData and errorCode as values to be returned by coroutine.yield.
    coroutine.resume(co, responseData, errorCode)
  end

  fetchRemote(url, opts, callback)

  -- Pause the execution of the current coroutine. It will be
  -- resumed when coroutine.resume gets called for this coroutine
  -- and any additional arguments will be returned.
  local responseData, errorCode = coroutine.yield()

  -- The coroutine has resumed, return the data.
  return responseData, errorCode
end

function testFetchAndOutput()
  local responseData, errorCode = fetchRemoteAsync('http://site.com/foo/bar', { method = 'POST', headers = ... })
  outputDebugString(responseData)
end

-- Create a coroutine that will execute testFetchAndOutput
local testCo = coroutine.create(testFetchAndOutput)

-- Execute the coroutine
coroutine.resume(testCo)

As a result, fetchRemoteAsync isn't blocking so you can call it and everything else on the server (including Lua code) can execute before it returns. But it can only be called from a coroutine.

Thank you for your response. Unfortunately, the example you provided will only work synchronously for the testFetchAndOutput function.

The behavior I would like to achieve:
 

-- Create a coroutine that will execute testFetchAndOutput
local co = coroutine.create(
  function()
    local responseData, errorCode = fetchRemoteAsync(url, opts)
    return responseData, errorCode
  end
)

-- Execute the coroutine
local coRes, responseData, errorCode = coroutine.resume(co)

outputDebugString('coRes: '..tostring(coRes)) -- true
outputDebugString('responseData: '..tostring(responseData)) -- '{data: foo}'
outputDebugString('errorCode: '..tostring(errorCode)) -- nil

I want to avoid callbacks. Your example provides some insight into working with coroutines in MTA. Is there somewhere I can read more about this? I haven't been able to find any information on this. Thanks again.

Edited September 8, 2023 by uriid1

[uriid1]

I tried to implement this logic in plain Lua(5.1), but I didn't get any results.
 

local function fetchRemote(url, opts, callback)
  callback("data: Yes, it's data", 200)
end

local function fetchRemoteAsync(url, opts)
  local co = coroutine.running()

  local function callback(responseData, errorCode)
	-- print(responseData) -- data: Yes, it's data
    coroutine.resume(co, responseData, errorCode)
  end

  fetchRemote(url, opts, callback)

  return coroutine.yield()
end

local function testFetchAndOutput()
  local responseData, errorCode = fetchRemoteAsync('http://site.com/foo/bar', { method = 'POST' })
  print('responseData:', responseData) -- nil
  return responseData, errorCode
end

local testCo = coroutine.create(testFetchAndOutput)

print(coroutine.status(testCo)) -- suspended
coroutine.resume(testCo)
print(coroutine.status(testCo)) -- suspended
coroutine.resume(testCo)
print(coroutine.status(testCo)) -- dead

 

Edited September 8, 2023 by uriid1

[uriid1]

Works if written like this:

  local function callback(responseData, errorCode)
    coroutine.yield(responseData, errorCode)
  end

But MTA doesn't support this notation.


I don't understand why this works:

local function test()
  return 'Hello'
end

local testCo = coroutine.create(test)

local result, data = coroutine.resume(testCo)

outputDebugString('Data: '..tostring(data)) -- 'Data: Hello'

And this doesn't work...

local function testFetchAndOutput()
  local responseData, errorCode = fetchRemoteAsync(url, opts)
  return responseData, errorCode
end

local testCo = coroutine.create(testFetchAndOutput)

local result, responseData, errorCode = coroutine.resume(testCo)

outputDebugString('responseData: '..tostring(responseData)) -- nil

I find this very strange.

Edited September 8, 2023 by uriid1

[DiSaMe]

10 hours ago, uriid1 said:

Unfortunately, the example you provided will only work synchronously for the testFetchAndOutput function.

That's the point, you're supposed to do the work from within the coroutine. testFetchAndOutput is an example of such function. It looks like a single function but executes over multiple invocations of the main thread. So you need to transform testFetchAndOutput into something that does all the work you need to do.

9 hours ago, uriid1 said:

local testCo = coroutine.create(testFetchAndOutput)

print(coroutine.status(testCo)) -- suspended
coroutine.resume(testCo)
print(coroutine.status(testCo)) -- suspended
coroutine.resume(testCo)
print(coroutine.status(testCo)) -- dead

Well, you shouldn't be resuming this coroutine like that. The coroutine should be resumed when the response data arrives, which the callback in fetchRemoteAsync already takes care of. What you need to do from the main thread is what my example already does, which is creating and starting the coroutine using coroutine.create and a single coroutine.resume call (optionally you can pass arguments to coroutine.resume to be used as arguments for coroutine's function). The rest of the coroutine will execute when response arrives. Until then, the coroutine will be in a suspended state and the main thread (the rest of the stuff that happens in the server, including the code following the initial coroutine.resume call) will be executing.

That basically turns the coroutine into a thread - not in a technical "OS thread" sense, but in a logical sense. fetchRemoteAsync doesn't execute instantaneously - between calling and returning, the time passes and other things update on the server. Coroutines allow you to avoid callbacks, but they can't just make asynchronous code run synchronously with respect to the main thread. Even if they could - unless it's some initialization code that runs once on start, you don't want to freeze your server in case the remote server takes a while to respond, do you?

In general, coroutines are one of my favorite Lua features. Unfortunately, they're very unpopular, and I'm hardly aware of sources of more info on this. Maybe searching in http://lua-users.org/wiki/ will give you some results.

It's not just in Lua that they're unpopular. An analogous feature in other languages that I'm aware of is generator functions in JavaScript (using function*, yield and yield* syntax and iterators) and Python (yield and yield from syntax and iterators). In those languages, it's probably overshadowed by async functions, using async/await syntax - in fact, most likely that's what we would be using for this fetchRemote stuff if it was in those languages instead of Lua.

Maybe I should make a tutorial on coroutines someday. But in the end, it's a simple concept, the coroutine execution just advances between coroutine.yield calls every time you resume it.

Edited September 9, 2023 by DiSaMe

[uriid1]

8 hours ago, DiSaMe said:

Well, you shouldn't be resuming this coroutine like that. The coroutine should be resumed when the response data arrives, which the callback in fetchRemoteAsync already takes care of. What you need to do from the main thread is what my example already does, which is creating and starting the coroutine using coroutine.create and a single coroutine.resume call (optionally you can pass arguments to coroutine.resume to be used as arguments for coroutine's function). The rest of the coroutine will execute when response arrives. Until then, the coroutine will be in a suspended state and the main thread (the rest of the stuff that happens in the server, including the code following the initial coroutine.resume call) will be executing.

Well, I understand where I went wrong and why it's not working as I expected. But now I'd like to learn a bit more about how asynchronous functions are executed in MTA and in which thread. Is this thread similar to an event loop, like in Node.js, or does it use a different model? And can you influence this thread, for example, by adding your own asynchronous functions?

On an interesting note, there's a project called luvit that also uses an asynchronous model, but they've figured out how to wrap asynchronous functions in coroutines. For example, you can see this in action here: https://github.com/luvit/lit/blob/master/deps/coro-net.lua.

This is something I would also like to achieve in MTA. However, for now, it seems impossible.

[DiSaMe]

3 hours ago, uriid1 said:

Well, I understand where I went wrong and why it's not working as I expected. But now I'd like to learn a bit more about how asynchronous functions are executed in MTA and in which thread.

Maybe for a more intuitive understanding, you shouldn't think of them as asynchronous code that executes "alongside" the rest of the code, but rather as code that executes "inside" coroutine.resume. Because that's exactly what happens. When execution enters coroutine.resume, it enters the coroutine, then when it reaches coroutine.yield, it leaves the coroutine and returns from coroutine.resume. Consider this concept of a coroutine-like construct:

function resumeCustomCoroutine(co)
  local nextFunction = co.functions[co.index]
  nextFunction()
  co.index = co.index+1
end

local testCo = {
  index = 1,
  functions = {
    function() print("first") end,
    function() print("second") end,
    function() print("third") end
  }
}

resumeCustomCoroutine(testCo) -- prints: first
resumeCustomCoroutine(testCo) -- prints: second
resumeCustomCoroutine(testCo) -- prints: third

You have no problem understanding that all code inside testCo.functions is executed inside resumeCustomCoroutine, do you? And yet, testCo.functions looks like a piece of code that executes "in parallel" to the code that calls resumeCustomCoroutine. You can think of coroutine.resume the same way, only instead of separate functions, you have "segments" between the start of the function, coroutine.yield calls and the end of the function.

3 hours ago, uriid1 said:

Is this thread similar to an event loop, like in Node.js, or does it use a different model? And can you influence this thread, for example, by adding your own asynchronous functions?

Lua itself has no such thing like JavaScript's event loop. And I don't know how MTA does things internally, but that doesn't matter - it just makes calls into Lua through event handlers or various callbacks, including those passed to fetchRemote. And wherever you call coroutine.resume from, it will advance the coroutine's execution up to the next coroutine.yield call.

3 hours ago, uriid1 said:

On an interesting note, there's a project called luvit that also uses an asynchronous model, but they've figured out how to wrap asynchronous functions in coroutines. For example, you can see this in action here: https://github.com/luvit/lit/blob/master/deps/coro-net.lua.

This is something I would also like to achieve in MTA. However, for now, it seems impossible.

Actually, if you look at makeCallback in that link, you can see it calls coroutine.running to store the current coroutine into variable thread, which is later passed (indirectly through nested function, then assertResume) to coroutine.resume. connect function uses makeCallback to create such coroutine-resuming function and passes it as a callback to socket:connect, then calls coroutine.yield to pause its own execution. So not only it's possible in MTA - it's the very same thing I did with fetchRemote, only structured slightly differently.