[DOC] CrystalMV’s Traffic Resource File structure

[Nurupo]

Motivation
Recently I'm working on vice city stories map conversion project, thinking whatever I can also convert the path from the single-player to the MTA. there's a traffic resource made by one of our lovely community members CrystalMV @Reyomin. however, a few documentation were in there regarding its traffic file structure since it is encoded in binary. For sure, we do need a bit more understanding of the basics of how this thing works before we can start doing some magic on that.

Disclaimer: The information might contain errors due to extracting by eyeball directly guessing from code 

The Variables

Traffic Map File Structure

Description

Identifier	Type	Description
Node Block
X	int (4 bytes)	X, / 1000 for actual value
Y	int (4 bytes)	Y, / 1000 for actual value
Z	int (4 bytes)	Z, / 1000 for actual value
RX	int (2 bytes)	RotX, / 1000 for actual value
RY	int (2 bytes)	RotY, / 1000 for actual value
Connection Block
Start	int (4 bytes)	The Start node index of a connection (Refer by index from node block)
End	int (4 bytes)	The End node index of a connection (Refer by index from node block)
Bend	int (4 bytes)	The blending point of a connection (Refer by index from node block)
Traffic type	unsigned integer (1 byte)	Traffic Type, 1: ped, 2: car, 3: boat, 4: plane
Lights	unsigned short (2 bytes)	Traffic light direction has two sub value, can obtain them by light_1= lights % 4 , light_2= floor(lights/4) it can be: 1: “NS”, 2: “WE” 3: “PED” (Red for all cars, no matter what direction, use when peds are crossing roads)
Speed	unsigned integer (1 byte)	Value from 0 – 255, indicate the max speed limit when passing through the node
Left lane		Number of left lanes does this node have
Right lane		Number of right lanes does this node have
Density		Indicate the density of how many traffic can generate
forbidden turn Block
C1	int (4 bytes)	The Start node index of a connection (Refer by index from node block)
C2	int (4 bytes)	The End node index of a connection (Refer by index from node block)

At last 

I also posted another copy in my blog, Feel free to correct me anything you found in the following reply.

At last, I'd like to appreciate all the hard work that the author made into this work, although it's an old stuff

Edited December 30, 2021 by Nurupo

[DiSaMe]

Wow, I'm very happy to see my scripts are still relevant, thanks! But it can be a negative thing as well, I mean it wasn't supposed to end up this way, back then I tried to improve the script, then I tried to make a new, a more advanced ped system from scratch but didn't have enough time, and to make things even worse, the code I wrote then looks horrible to me now So happy to feel appreciated, yet the fact that everyone's focused on my work that still wasn't satisfying enough to myself, gives me this unpleasant feeling that I gotta clean up this mess, gotta finish the work I once started but I don't know when I'm going to try scripting on MTA again

Conversion of game paths to my path format sounds interesting, but making my own paths instead of converting the ones from the game was the reason I came up with my format and traffic editor to begin with. I was having trouble understanding some things in the game's format, and the paths have things like multiple nodes in a straight line, which needlessly uses more data.

Now, to the documentation details.

node_conns tells the connection ID from node IDs. That is, node_conns[node1_id][node2_id] = connection_id.

Also, node coordinates are decimal fixed-point numbers, not floats, and RX/RY each takes 2 bytes as opposed to 4. Reading the value as integer and dividing it by 1000, you get the actual value. I had some problems converting between bytes and floats, and integers were faster to process.

Finally, traffic lights. The value 3: "PED" means red for all cars, and that's when the peds cross the road. Normally it would work according to pedestrian traffic lights, but as far as I know, those are neither synced, nor scriptable - at least they weren't when I made the resource.

[Platin]

8 hours ago, Reyomin said:

yet the fact that everyone's focused on my work that still wasn't satisfying enough to myself, gives me this unpleasant feeling that I gotta clean up this mess, gotta finish the work I once started

Everyone thinks about their past work like this sometimes. It just means that you got better over time and that you are more experienced. There is nothing wrong about it. Of course it would be nice to see you working again on great MTA:SA Resources, but please don't feel pressured to do so.

[DiSaMe]

32 minutes ago, Platin - xNikoXD said:

Everyone thinks about their past work like this sometimes. It just means that you got better over time and that you are more experienced. There is nothing wrong about it. Of course it would be nice to see you working again on great MTA:SA Resources, but please don't feel pressured to do so.

You're right. But I already could tell there was something wrong with the code even then. Which is why I always considered it an unresolved matter from the beginning. Never stopped thinking it would be nice to make a better ped system. You could say it's not pressure, but motivation.

In addition, by making the resource I wanted to demonstrate that ped scripting is not really that hard, and encourage more people to try it, hoping that after a while we would have more ped/traffic resources. That didn't come true to the extent that I was hoping

[Platin]

2 hours ago, Reyomin said:

In addition, by making the resource I wanted to demonstrate that ped scripting is not really that hard, and encourage more people to try it, hoping that after a while we would have more ped/traffic resources. That didn't come true to the extent that I was hoping

Well, there is a freeroam that made their own version of the traffic system + police and gangs. And I think there is other server that also has an NPC system but I am not so sure about it. Because of the developer of Prime Freeroam things like doing enter_exit on peds is now trivial, when before it needed to be scripted.

Maybe the reason for these systems not being more popular is because they are not open source and are very hard to create (or time consuming) / are great performance destroyers for some users.

Anyway, if you are motivated then go for it, a lot of people would probably appreciate it and maybe even use it on a moderate scale. Who knows.

[Nurupo]

3 hours ago, Reyomin said:

You're right. But I already could tell there was something wrong with the code even then. Which is why I always considered it an unresolved matter from the beginning. Never stopped thinking it would be nice to make a better ped system. You could say it's not pressure, but motivation.

In addition, by making the resource I wanted to demonstrate that ped scripting is not really that hard, and encourage more people to try it, hoping that after a while we would have more ped/traffic resources. That didn't come true to the extent that I was hoping

I'd say the extensibility of the resources at least is good enough, you have seperate the NPC controls and traffic, there's does some rushes code in there though, but i woundn't consider it's wrong there, like @Platin - xNikoXD said you just geting better over time and that you are more experienced. but at least the npchlc did a good job, it's easy to extend, i've made some experiment on that, like cops chasing, arrest,gangs, actually it works pretty smooth.

Speaking in terms of considering refactoring, I suggest we can take a look at how the original game traffic's code is designed, there are several reverse engineering projects available, RE3, GTASA-Reversed, see if we can do something based on them, I think this is the most accurate approach to simulate the original game traffic.

But in overall, it's you did a relly good job @Reyomin

 

[The_GTA]

1 hour ago, Nurupo said:

... there are several reverse engineering projects available, *exhibit 1*, *exhibit 2*, ...

It is quite a shame how certain projects were handled and that the code was released to the public in such a complete form, claiming to be the original but lacking the DRM protections of the originals, just giving room for the most misunderstanding, emotionally childish people to abuse the GTA modding situation. Every GTA modder was hurt by the grievous IP rights breach. Since most of you hard working MTA engineers know reverse engineering from the "expose the necessities, tie together with the original game being present" you are not likely to breach the core principles of respectful IP rights handling. But those people even tried to establish their own funding-based IP rights touching product were they had no permission. Please be careful about mentioning reverse engineering projects because one of them I quite disagree with. I know, this situation has gone quite out-of-hand in this passing year and you might not have followed the situation as closely as I did.

Going into these source codes is risky. I would like to give you an example of how risky other people that touch other IP works see it. The ReactOS project disallows contributors if they even worked for Microsoft, let alone took a look into the original source code. Quite the extreme stand-point, but I guess they had quite the breaches based on their many years of experience with random newcomers trying to contribute code they could not faithfully trace back to the real origin. Just be careful and put in your own work after all, alright? 

Edited December 30, 2021 by The_GTA

[Nurupo]

1 hour ago, The_GTA said:

It is quite a shame how certain projects were handled and that the code was released to the public in such a complete form, claiming to be the original but lacking the DRM protections of the originals, just giving room for the most misunderstanding, emotionally childish people to abuse the GTA modding situation. Every GTA modder was hurt by the grievous IP rights breach. Since most of you hard working MTA engineers know reverse engineering from the "expose the necessities, tie together with the original game being present" you are not likely to breach the core principles of respectful IP rights handling. But those people even tried to establish their own funding-based IP rights touching product were they had no permission. Please be careful about mentioning reverse engineering projects because one of them I quite disagree with. I know, this situation has gone quite out-of-hand in this passing year and you might not have followed the situation as closely as I did.

Going into these source codes is risky. I would like to give you an example of how risky other people that touch other IP works see it. The ReactOS project disallows contributors if they even worked for Microsoft, let alone took a look into the original source code. Quite the extreme stand-point, but I guess they had quite the breaches based on their many years of experience with random newcomers trying to contribute code they could not faithfully trace back to the real origin. Just be careful and put in your own work after all, alright? 

You're right, that's a issue, I'll keep that in mind, definitely go with my own. maybe those projects are probably best to go with study purpose, e.g. what elements are there in traffic, like when the game first comes out, people are researching & investigate how the TXDs & DFFs structure laid out, and come up with their own code to parse the file.

Edited December 30, 2021 by Nurupo

[The_GTA]

11 minutes ago, Nurupo said:

... come up with their own code to parse the file.

Yes, I think that this practice of "creating (...) own code" along with documentation does teach the developers the respect for other people's work. For example, let's take a look at my Magic.TXD which has a parser for just about all the commercial RW TXD file formats. The code and parsing models inside of that tool which are related to TXD files are essentially my work, with the help of accredited third party code as side contribution. Of course, much of the data layout is fixed due to expectations of the target architectures but there was a lot of ingenuity involved at connecting the dots. From reading your post I think that you get the idea.

I am a big fan of good-founded binary file formats, such as @Reyomin's node file format as discussed in this thread. The advantage of binary file formats is their nativity to the platform as well as the deterministic structure (you know the file contents by hard). There is no room for unnecessary ambiguity. And from what I gather Reyomin has even potentially improved the mathematical structure by compressing multiple nodes in one line into a two-node minimal representation. The science does shine through, which is splendid! 

Edited December 30, 2021 by The_GTA

[Nurupo]

20 hours ago, Reyomin said:

Wow, I'm very happy to see my scripts are still relevant, thanks! But it can be a negative thing as well, I mean it wasn't supposed to end up this way, back then I tried to improve the script, then I tried to make a new, a more advanced ped system from scratch but didn't have enough time, and to make things even worse, the code I wrote then looks horrible to me now So happy to feel appreciated, yet the fact that everyone's focused on my work that still wasn't satisfying enough to myself, gives me this unpleasant feeling that I gotta clean up this mess, gotta finish the work I once started but I don't know when I'm going to try scripting on MTA again

Conversion of game paths to my path format sounds interesting, but making my own paths instead of converting the ones from the game was the reason I came up with my format and traffic editor to begin with. I was having trouble understanding some things in the game's format, and the paths have things like multiple nodes in a straight line, which needlessly uses more data.

Now, to the documentation details.

node_conns tells the connection ID from node IDs. That is, node_conns[node1_id][node2_id] = connection_id.

Also, node coordinates are decimal fixed-point numbers, not floats, and RX/RY each takes 2 bytes as opposed to 4. Reading the value as integer and dividing it by 1000, you get the actual value. I had some problems converting between bytes and floats, and integers were faster to process.

Finally, traffic lights. The value 3: "PED" means red for all cars, and that's when the peds cross the road. Normally it would work according to pedestrian traffic lights, but as far as I know, those are neither synced, nor scriptable - at least they weren't when I made the resource.

Thanks for the confirmation@Reyomin, speaking in terms of RY,RY, how do they excally work? i've looked over the PATH's wiki from https://www.grandtheftwiki.com/PATH_(IDE_Section) seems the original game path does not have rotation information, instead it has the "Median" for reprenting the width of walkway, kind of simular idea. I'm looking for a way to convert them into your traffic map, I've managed converted the nodes position from the original game path file, I need a bit help on that 

Edited December 30, 2021 by Nurupo

[DiSaMe]

8 hours ago, Platin - xNikoXD said:

Well, there is a freeroam that made their own version of the traffic system + police and gangs. And I think there is other server that also has an NPC system but I am not so sure about it. Because of the developer of Prime Freeroam things like doing enter_exit on peds is now trivial, when before it needed to be scripted.

Maybe the reason for these systems not being more popular is because they are not open source and are very hard to create (or time consuming) / are great performance destroyers for some users.

Anyway, if you are motivated then go for it, a lot of people would probably appreciate it and maybe even use it on a moderate scale. Who knows.

Yes, peds are more popular than they used to be, it's just that I was expecting them to be even more popular.

Not that I have any particular plans to return to scripting, but it's a possibility.

7 hours ago, Nurupo said:

I'd say the extensibility of the resources at least is good enough, you have seperate the NPC controls and traffic, there's does some rushes code in there though, but i woundn't consider it's wrong there, like @Platin - xNikoXD said you just geting better over time and that you are more experienced. but at least the npchlc did a good job, it's easy to extend, i've made some experiment on that, like cops chasing, arrest,gangs, actually it works pretty smooth.

Speaking in terms of considering refactoring, I suggest we can take a look at how the original game traffic's code is designed, there are several reverse engineering projects available, RE3, GTASA-Reversed, see if we can do something based on them, I think this is the most accurate approach to simulate the original game traffic.

But in overall, it's you did a relly good job @Reyomin

 

I love these videos!

I might have gotten some things right, but my goal was to make ped scripting easy so that even beginners could do it. Didn't come close enough because I only implemented very basic tasks. Was thinking of implementing pathfinding but never did.

Analyzing how the game does it may help getting the peds behave the way they do in original game, but I'm not a fan of this approach. Reimplementing original behavior wouldn't be satisfying enough, I always wanted peds to perform complex tasks regardless of how the game does it.

3 hours ago, The_GTA said:

And from what I gather Reyomin has even potentially improved the mathematical structure by compressing multiple nodes in one line into a two-node minimal representation. The science does shine through, which is splendid! 

I'm not sure if I understood what you're saying. I mean I didn't do anything special regarding that. But you're making it sound like I did something that I didn't actually do. When I was playing around with nodes from original game, I found that there were many redundant nodes (or so it looked). But I created my own paths in the end (which is what I created npchlc_traffic_editor for), and obviously, I wouldn't create nodes that I find redundant. So Rockstar created more nodes in their paths than I created in mine. That's it.

49 minutes ago, Nurupo said:

Thanks for the confirmation@Reyomin, speaking in terms of RY,RY, how do they excally work? i've looked over the PATH's wiki from https://www.grandtheftwiki.com/PATH_(IDE_Section) seems the original game path does not have rotation information, instead it has the "Median" for reprenting the width of walkway, kind of simular idea. I'm looking for a way to convert them into your traffic map, I've managed converted the nodes position from the original game path file, I need a bit help on that 

If I remember correctly, it's just a vector that tells lane position relative to the node position. When you drag while creating a node in the editor, that's when you set RX and RY.

[The_GTA]

7 minutes ago, Reyomin said:

I'm not sure if I understood what you're saying. (...) I found that there were many redundant nodes (or so it looked). But I created my own paths in the end (which is what I created npchlc_traffic_editor for), and obviously, I wouldn't create nodes that I find redundant. So Rockstar created more nodes in their paths than I created in mine. That's it.

You understood me correctly and I did understand you correctly, too!

[DiSaMe]

1 hour ago, The_GTA said:

You understood me correctly and I did understand you correctly, too!

Oh, okay. It was the wording that confused me