Software Rendering in MTA:SA Lua

The_GTA · February 22, 2020

Dear MTA community,

I have been spending my last 8 weeks on mathematical problems. One of them is the 3D Frustum-Plane intersection that is used by GPUs to draw triangles onto your screen. If you want to learn more about this please consider reading this thread.

Promotional Video: https://www.youtube.com/watch?v=RQy3Q4Xe110

Prerequisites
This tutorial is aimed at people who are capable of scientific thinking and are willing to playfully learn with Lua code. To execute steps in this tutorial minimal knowledge of Linear Algebra and Lua is required.

Required MTA Resource: https://github.com/quiret/mta_lua_3d_math

Description of the math
Imagine that we have got a frustum and a plane in a 3D room described by coordinates plus their boundaries. By intersecting both you obtain all coordinates on a screen along with their depth values.

Now think about how your vision works. You see distant objects smaller than closer ones. You rotate your eyes to angles of vision. If we were to put this concept into terms of math we could say: the plane of vision is bigger in the distance than in close proximity. The frustum is a seamless row of vision planes starting from the pyramid tip to the bottom.

How to use the MTA Resource
Just download the GitHub repository into a folder of your MTA Resources, name it "math_3d_nonlin" and start it. You can execute the following commands for quick testing:

send_bbuf: draws a simple depth test
draw_model: draws the DFF file "gfriend.dff"

Now we have got the basics out of the way. Time to start coding.

Please create a new "_math_test.Lua" script file in the resource and include it server-side at the bottom of meta.xml.

Tutorial: software rendering a plane on screen
Open your _math_test.Lua and include the following code:

local viewFrustum = createViewFrustum(
    createVector(0, 0, 0),      -- position
    createVector(10, 0, 0),     -- right
    createVector(0, 0, 10),     -- up
    createVector(0, 20, 0)      -- front
);
local plane = createPlane(
    createVector(-3, 10, -3),
    createVector(6, 0, 0),
    createVector(0, 0, 6)
);
local function task_draw_scene(thread)
    local bbuf = create_backbuffer(640, 480, 255, 255, 0, 50);
    local dbuf = createDepthBuffer(640, 480, 1);
    
    local time_start = getTickCount();
    
    do
        local gotToDraw, numDrawn, numSkipped = draw_plane_on_bbuf(viewFrustum, bbuf, dbuf, plane, true);
        
        if ( gotToDraw ) then
            outputDebugString( "drawn " .. numDrawn .. " pixels (skipped " .. numSkipped .. ")" );
        end
    end
    
    local time_end = getTickCount();
    local ms_diff = ( time_end - time_start );
    
    outputDebugString( "render time: " .. ms_diff .. "ms" );
    
    taskUpdate( 1, "creating backbuffer color composition string" );
    
    local bbuf_width_ushort = num_to_ushort_bytes( bbuf.width );
    local bbuf_height_ushort = num_to_ushort_bytes( bbuf.height );
    
    local pixels_str = table.concat(bbuf.items);
    
    local bbuf_string =
        pixels_str ..
        ( bbuf_width_ushort ..
        bbuf_height_ushort );
        
    taskUpdate( false, "sending backbuffer to clients (render time: " .. ms_diff .. "ms)" );
        
    local players = getElementsByType("player");
    
    for m,n in ipairs(players) do
        triggerClientEvent(n, "onServerTransmitImage", root, bbuf_string);
    end
    
    outputDebugString("sent backbuffer to clients");
end
addCommandHandler( "testdraw", function(player)
        spawnTask(task_draw_scene);
    end
);

Result:

Try executing the "testdraw" command.

At the top of file you see the definition of our frustum cone as well as a plane. By calling the function "draw_plane_on_bbuf" we put color information into bbuf for exactly the pixels that make up the rectangle. If you change the plane definition to...

local plane = createPlane(
    createVector(-2, 10, -4),
    createVector(6, 0, 3),
    createVector(-2, 0, 6)
);

you instead get this image:

Try changing around the coordinates of frustum and plane to obtain different pictures!

Tutorial: software rendering a triangle on screen
Take the same code as in the tutorial above but change line 19 to:

        local gotToDraw, numDrawn, numSkipped = draw_plane_on_bbuf(viewFrustum, bbuf, dbuf, plane, true, "tri");

This way we have changed the primitive type to triangle (rectangle is the default). Try executing the "testdraw" command again to inspect the new result!

Tutorial: drawing a DFF file onto screen
Instead of writing triangle definitions by hand we can take them from a DFF file instead. DFF files are storage of triangle and vertex information along with 3D rotation and translation information. By extacting the triangles from the DFF file we can put them into our algorithm to software-render them!

Here is a related excerpt from math_server.Lua:

local modelToDraw = false;
do
    local modelFile = fileOpen("gfriend.dff");
    
    if (modelFile) then
        modelToDraw = rwReadClump(modelFile);
        fileClose(modelFile);
    end
end
local function task_draw_model(thread)
    local bbuf = create_backbuffer(640, 480, 255, 255, 0, 50);
    local dbuf = createDepthBuffer(640, 480, 1);
    
    local time_start = getTickCount();
    
    local num_triangles_drawn = 0;
    
    if (modelToDraw) then
        -- Setup the camera.
        local geom = modelToDraw.geomlist[1];
        local mt = geom.morphTargets[1];
        local centerSphere = mt.sphere;
        
        local camPos = viewFrustum.getPos();
        camPos.setX(centerSphere.x);
        camPos.setY(centerSphere.y - 3.8);
        camPos.setZ(centerSphere.z);
        
        local camFront = viewFrustum.getFront();
        camFront.setX(0);
        camFront.setY(5 + centerSphere.r * 2);
        camFront.setZ(0);
        
        local camRight = viewFrustum.getRight();
        camRight.setX(centerSphere.r * 2);
        camRight.setY(0);
        camRight.getZ(0);
        
        local camUp = viewFrustum.getUp();
        camUp.setX(0);
        camUp.setY(0);
        camUp.setZ(centerSphere.r * 2);
    
        local triPlane = createPlane(
            createVector(0, 0, 0),
            createVector(0, 0, 0),
            createVector(0, 0, 0)
        );
        
        local vertices = modelToDraw.geomlist[1].morphTargets[1].vertices;
        local triangles = modelToDraw.geomlist[1].triangles;
        
        local tpos = triPlane.getPos();
        local tu = triPlane.getU();
        local tv = triPlane.getV();
        
        for m,n in ipairs(triangles) do
            taskUpdate( m / #triangles, "drawing triangle #" .. m );
            
            local vert1 = vertices[n.vertex1 + 1];
            local vert2 = vertices[n.vertex2 + 1];
            local vert3 = vertices[n.vertex3 + 1];
            tpos.setX(vert1.x);
            tpos.setY(vert1.y);
            tpos.setZ(vert1.z);
            
            tu.setX(vert2.x - vert1.x);
            tu.setY(vert2.y - vert1.y);
            tu.setZ(vert2.z - vert1.z);
            
            tv.setX(vert3.x - vert1.x);
            tv.setY(vert3.y - vert1.y);
            tv.setZ(vert3.z - vert1.z);
            
            local gotToDraw, numDrawn, numSkipped = draw_plane_on_bbuf(viewFrustum, bbuf, dbuf, triPlane, false, "tri");
            
            if (gotToDraw) and (numDrawn >= 1) then
                num_triangles_drawn = num_triangles_drawn + 1;
            end
        end
    end
        
    local time_end = getTickCount();
    local ms_diff = ( time_end - time_start );
    
    (...)
end

The code first loads a DFF file called "gfriend.dff" and stores it inside the "modelToDraw" variable. Once you execute the "draw_model" command the code looks up the first geometry in the DFF file and fetches all triangles associated with it. The rendering camera is set up to point at the middle of the model. Then all triangles are drawn one-by-one.

https://twitter.com/rplgn/status/1230650912345067520

Try swapping the DFF file for another one, like biker.dff, and examine the results! Maybe extract a different DFF file from GTA:SA and replace gfriend.dff with that one.

External references:

math calculation on paper example: https://imgur.com/gallery/rLvln3X
German thread on mta-sa.org: https://www.mta-sa.org/thread/38693-3d-frustum-ebene-schneidung-in-Lua/

Do you have any questions related to the math or the implementation? Do not shy away from asking! I want to provide you with as much insight as I can.

Edited February 23, 2020 by The_GTA
beautification, fixes for links, added images of frustum, added promotional video

-ffs-PLASMA · February 22, 2020

Doom port soon in MTA?

Good work :3

The_GTA · February 22, 2020

1 hour ago, -ffs-PLASMA said:

Doom port soon in MTA?

Good work :3

Thanks! <3 I like hearing about your appreciation.

About the possibility of a Doom port, this could take a lot of work ;) But not impossible because Doom is a retro game.

Haxardous · February 23, 2020

Imagine what i can do with this, good job!

The_GTA · February 23, 2020

3 hours ago, Haxardous said:

Imagine what i can do with this, good job!

Hello Haxardous!

Thank you for complimenting me. I spent a lot of work into it and just want people to have a good starting foundation. Later the performance of the implementation will be improved by choosing better data structures as well as faster algorithms (already have plans).

As a side note, I was heavily inspired by Direct3D 9's frustum definition when setting out my math formulas. So if you read their documentation you will find many parallels. Pretty sure this definition is industry standard.

Edited February 23, 2020 by The_GTA

The_GTA · March 4, 2020

After spending some time bugfixing and optimizing the implementation, the performance has been improved. It still is not as fast as a hardware GPU but you should understand why

To celebrate the improvements I want to show you guys how to check with 100% accuracy whether a triangle is on the screen. We basically use the same code we do for the rendering but skip the rasterization part. Take a look at this code (test_client.Lua clientside file at the end of meta.xml):

local triangle = createPlane(
    createVector(0, 0, 5),
    createVector(25, 0, 0),
    createVector(0, 0, 15)
);

local frustum_pos = createVector(0, 0, 0);
local frustum_right = createVector(0, 0, 0);
local frustum_up = createVector(0, 0, 0);
local frustum_front = createVector(0, 0, 0);

local frustum = createViewFrustum(
    frustum_pos,
    frustum_right,
    frustum_up,
    frustum_front
);

local function set_frustum_from_camera()
    local camMat = getElementMatrix(getCamera());
    local camPos = camMat[4];
    local camRight = camMat[1];
    local camFront = camMat[2];
    local camUp = camMat[3];
    local farClip = getFarClipDistance();
    
    local cam_frontX = camFront[1] * farClip;
    local cam_frontY = camFront[2] * farClip;
    local cam_frontZ = camFront[3] * farClip;
    
    local sW, sH = guiGetScreenSize();
    
    local s_ratio = sW / sH;
    
    local _, _, _, _, _, _, _, fov = getCameraMatrix();
    local fovRad = math.rad(fov/2);

    local cam_side_dist = farClip * math.tan(fovRad);
    local cam_up_dist = cam_side_dist / s_ratio;

    local cam_rightX = camRight[1] * cam_side_dist;
    local cam_rightY = camRight[2] * cam_side_dist;
    local cam_rightZ = camRight[3] * cam_side_dist;
    
    local cam_upX = camUp[1] * cam_up_dist;
    local cam_upY = camUp[2] * cam_up_dist;
    local cam_upZ = camUp[3] * cam_up_dist;
    
    frustum_pos.setX(camPos[1]);
    frustum_pos.setY(camPos[2]);
    frustum_pos.setZ(camPos[3]);
    
    frustum_right.setX(cam_rightX);
    frustum_right.setY(cam_rightY);
    frustum_right.setZ(cam_rightZ);
    
    frustum_up.setX(cam_upX);
    frustum_up.setY(cam_upY);
    frustum_up.setZ(cam_upZ);
    
    frustum_front.setX(cam_frontX);
    frustum_front.setY(cam_frontY);
    frustum_front.setZ(cam_frontZ);
end

addEventHandler("onClientRender", root,
    function()
        local triPos = triangle.getPos();
        local triU = triangle.getU();
        local triV = triangle.getV();
        
        local vert1 = {
            triPos.getX(),
            triPos.getY(),
            triPos.getZ(),
            tocolor(255, 255, 255)
        };
        
        local vert2 = {
            triPos.getX() + triU.getX(),
            triPos.getY() + triU.getY(),
            triPos.getZ() + triU.getZ(),
            tocolor(255, 255, 255)
        };
        
        local vert3 = {
            triPos.getX() + triV.getX(),
            triPos.getY() + triV.getY(),
            triPos.getZ() + triV.getZ(),
            tocolor(255, 255, 255)
        };
        
        dxDrawPrimitive3D("trianglelist", false, vert1, vert2, vert3);
        
        -- Check whether the triangle is on screen.
        set_frustum_from_camera(frustum);
        
        local inter = frustum.intersectWithTrianglePlane(triangle);
    
        dxDrawText("is intersecting: " .. tostring(not (inter == false)), 100, 300);
    end
);

The function "set_frustum_from_camera" does apply the GTA:SA camera metrics onto our frustum object, thus accurately representing the game camera. The triangle object is located at the center of the GTA:SA map (the farming area). If you teleport there and start the resource you will see a white triangle as well as the text: "is intersecting: true". If you move the camera away from the triangle then the text will change to "is intersecting: false". Thus it is checking whether the triangle is visible through the game camera.

The script is pretty slow at the moment so the framerate does stutter (Lua is to be partially blamed). But I have plans to improve the performance even futher, so please follow this thread

Edited March 4, 2020 by The_GTA

Ren_712 · May 26, 2021

The_GTA This resource is beyond impressive. I found this topic just a week ago, immediately threw every other thing to study it.
I'm particularly focused right no on using rw_shared.lua to read model data.
This is a current result I've got with the standard model provided in the resource archive drawn using primitiveMaterial3D

The problems started when i decided to load some different models.
The ones provided with the resource load with no issues, however Any other model (including original ones from gta3.img) fail to load.
The same thing happens with clean math_3d_nonlin after typing draw_model (have replaced gf model with skyscr01_lawn.dff from original game)

the err string for this particular file is:
failed to read geometrylist: failed to read geometry #1: failed to read morph target vertex translation
The script iterates through numVertices (a value much higher than an actual number) until this exception occurs
the values from start to end don't make much sense...

x 5.8774857670961e-39 y 5.8774997800807e-39 z 1.7060598161754e-24
x 5.8774724547607e-39 y 5.8774829644992e-39 z 1.7060598161754e-24
x 0 y 0 z 5.8774752573576e-39

Edited May 26, 2021 by Ren_712

The_GTA · May 26, 2021

2 hours ago, Ren_712 said:

The_GTA This resource is beyond impressive. I found this topic just a week ago, immediately threw every other thing to study it.
I'm particularly focused right no on using rw_shared.lua to read model data.

(...)

The problems started when i decided to load some different models.
The ones provided with the resource load with no issues, however Any other model (including original ones from gta3.img) fail to load.
The same thing happens with clean math_3d_nonlin after typing draw_model (have replaced gf model with skyscr01_lawn.dff from original game)

the err string for this particular file is:
failed to read geometrylist: failed to read geometry #1: failed to read morph target vertex translation
The script iterates through numVertices (a value much higher than an actual number) until this exception occurs
the values from start to end don't make much sense...

Thank you very much for your valuable feedback!

Based on the example DFF file that you have provided me (skyscr01_lawn.dff) I was able to detect an issue in the geometry chunk vertex-prelighting color reading. Instead of reading the color channels as uint8/byte, I was reading them as float About 20 minutes ago I have pushed a new commit to the GitHub repository which can now load the mentioned DFF file. If you find any further bugs then please post again, I will evaluate whether I want to support the DFF file and then fix any bugs.

I am very interested in the MTA-backed DFF renderer that you have created. It seems like a fun project and I like it.

Ren_712 · May 26, 2021

Man, that was fast I also noticed that textureInfo.texName string has a last NULL character, and probably others in rwReadTextureInfo but not sure. Will test tomorrow.

The_GTA · May 26, 2021

5 minutes ago, Ren_712 said:

Man, that was fast I also noticed that textureInfo.texName string has a last NULL character, and probably others in rwReadTextureInfo but not sure. Will test tomorrow.

OK. I forgot to subtract a zero there, should trim off the trailing '\0' now.

Ren_712 · May 27, 2021

This will do great for alpha objects and allow perfect control of the rendering order. Thank You so much for fixing the parser.

Edited May 27, 2021 by Ren_712

The_GTA · May 27, 2021

1 hour ago, Ren_712 said:

This will do great for alpha objects and allow perfect control of the rendering order. Thank You so much for fixing the parser.

You're welcome! Happy to be part of your vision. Not so sure about your alpha fix claims because I happen to be deeply into this research topic of per-pixel rendering order but it sure does help a little, maybe enough to fit your cause Replacing the internal GTA:SA rendering calls with MTA versions sounds fun, I hope you achieve great new creative control over MTA server content this way!

maximumdrive · October 7, 2021

This job is just great! That way of applying science gives outstanding results, still wondering why do I find such interesting topics in years!

Among other things I'm focused at the parser as Ren_712 did. The only problem found, it doesn't 'accept' larger DFFs from about 5-10 MB breaking with 'infinite running script'.

But that is 'too little things' in front of the research made on RW and maths applied. Wish you all the best on the way of your interest!

The_GTA · October 7, 2021

4 hours ago, maximumdrive said:

This job is just great! That way of applying science gives outstanding results, still wondering why do I find such interesting topics in years!

Among other things I'm focused at the parser as Ren_712 did. The only problem found, it doesn't 'accept' larger DFFs from about 5-10 MB breaking with 'infinite running script'.

But that is 'too little things' in front of the research made on RW and maths applied. Wish you all the best on the way of your interest!

Glad to hear that you like it! In order to stop the "infinite running script" error, there is this little-known secret to use "debug.sethook(nil)" to allow unbounded execution. Use it wisely because if there really is an infinite loop then you cannot properly terminate it anymore (game hangs, server hangs, data loss, etc).

Sign In

Software Rendering in MTA:SA Lua

Recommended Posts

The_GTA

Link to comment

-ffs-PLASMA

Link to comment

The_GTA

Link to comment

Haxardous

Link to comment

The_GTA

Link to comment

The_GTA

Link to comment

Ren_712

Link to comment

The_GTA

Link to comment

Ren_712

Link to comment

The_GTA

Link to comment

Ren_712

Link to comment

The_GTA

Link to comment

maximumdrive

Link to comment

The_GTA

Link to comment

Recently Browsing 0 members

Browse

Activity

MTA Network

Social Media