modding HUGE Problem with Modding RDR2 That No One is Talking About

[RLTYProds 1]

Thank you, got it working! Thanks for making solutions to this problem, hopefully we get some from Rockstar themselves too, but I ain't holding my breath.

[Edegon 11]

2 hours ago, werwolf969 said:

Since this code not for Python but for , as I specified on Pastebin, best bet will be to save script code as .ps1 file.

 

 

I see, thank you.

I think deadwoodd made a mistake here then:

11 hours ago, deadwoodd said:

which is where the Python script comes into play.

 

[werwolf969 32]

43 minutes ago, RLTYProds said:

Thanks for making solutions to this problem

You're welcome, but I wouldn't call them solutions. Because the root cause of game logic breaks is much more complex than simply inserting files into game archives with different method. When the situation is sorted out, we can use streaming again, and I think all this will be redundant.

Looking at the GTA V game configuration and additional ASI plugins, manipulating in-game memory limits and pools, I think that is where we should look. And maybe, after that, LML for RDR2 will support adding external RPF archives with modified data inside the game, which will solve this problem.

[Edegon 11]

2 minutes ago, werwolf969 said:

When the situation is sorted out, we can use streaming again, and I think all this will be redundant.

dominatorgt said that:

Quote

File Streaming:

1. Process: In file streaming, the file is loaded into memory in chunks rather than the entire file at once. This means that only the parts of the file that are needed at a given time are loaded into memory
2. This can be taxing for SSDs, as they are not as efficient at handling small files
3.  

• Frequent Random Access: If rapplication constantly jumps around to different small files on the SSD, then it can be less efficient.
• Many Temporary Files: If streaming creates a large number of temporary small files that are constantly written and deleted, it can impact SSD lifespan over time.

Here.
Do you think points 2 and 3 can be fixed too?

[werwolf969 32]

Just now, Edegon said:

Do you think points 2 and 3 can be fixed too?

I can't say anything until we get RPF archive support in LML for RDR2. It will depend on how the data from them is inserted into the game. Currently, any new files you add to the game will be streamed (there is nothing to replace when adding new content). Therefore, all the thousands of clothing texture dictionaries from WhyEm/EEE/GFA, all additional YMAP locations, and all new YMT definitions for newly added content will be streamed into the game through LML. You can use replacements (and many mods do that) for existing files, and this surely has less strain on your storage and on the game engine, as DominatorGT mentions in his posts. However, this will not solve the problem of newly added content, and for me, this is what matters the most.

How external RPF support could possibly work, I don't know. I'm too lazy to install GTA V, start modding it, and search for answers.

[Edegon 11]

1 hour ago, werwolf969 said:

Here's an example of this file. You can paste any path there, including the path to any file in the game archive, even if it is not in the source directory (making "source paths" a cheat sheet for the script). The script will only choose paths to files in the mod directory. However, for performance reasons, it's best to filter the data in this file to something similar to the source directory content.

source_paths.txt 322.34 kB · 4 downloads

So I need to manually find, copy and paste the original path of every file in the mod from OpenIV and paste it there?

Sometimes the file is in more paths, all of them (excluding the one of the mod itself) must be put in source_paths.txt?

Edited November 5 by Edegon

[werwolf969 32]

*doubleposted*

Edited November 5 by werwolf969

[werwolf969 32]

11 minutes ago, Edegon said:

So I need to manually find, copy and paste the original path of every file in the mod from OpenIV and paste it there?

Correct.

 

11 minutes ago, Edegon said:

Sometimes the file is in more paths, I guess I should choose just the one of update.rpf, right?

Just copy them all. The script will insert a path with LML mount points for each of them if there is a file with the same name in the source folder.

In fact, for even easier "non-brainer" install.xml file generation, you should use this version of the script.

https://pastebin.com/ZyZePbjX

A made changes. So, if there is more than one mount point for a source file in LML, the script will add the "allowDuplicate=true" option to the node. Thus, LML will replace the file in every archive within the game. And you will be sure that you actually loaded the replacement file into the game.

You can search for files not by their full name, but by some similar part of their name. For example, if you have files to replace with names like:
crp_berry.ytd

crp_berry_aa_sim.yft

crp_berry_har_aa_sim.yft

 

You can search for crp_berry and copy all the paths into the source_paths. The script will choose only existing files from the source folder to paste into install.xml.

Edited November 5 by werwolf969

[deadwoodd 1]

8 hours ago, Edegon said:

 

I see, thank you.

I think deadwoodd made a mistake here then:

 

Can confirm, I am indeed an idiot 😁 I'll edit the post to fix that and will clarify it's a powershell script, not python

[BitTitan 1]

@werwolf969 -- Dude you are a hero among men 😄  

 

I'm looking at modifying your updated (Duplicates=True) version so that it also outputs to /nested/game/path/folder/moddedfile.ext.txt.   Reasoning for this is I use ModOrganizr to manage RDR mods.  After over a decade of Bethesda modding, I'm spoiled in what a mod manager should do.  I'm still limited by what RDR requires to inject modded files, but your scripts and masochist methods of managing modded files is working well -- I think.  Because I'm choose to not rely on LMM to handing conflict detection; frankly I'm not even sure LMM has conflict detection outside of looking the log file for OVERWRITE.

 

My PowerShell-fu is weak and I'm converting Linux shell logic into PowerShell, but I'll get there soon as I figure out the translation and logic differences.  My napkin logic (which at this point looks like the broken English of coding) is:

mkdir -p nested\${RemainingPath}\${MatchingFile}.txt

I'm not 100% on which variables make the most sense and obviously that syntax is now PowerShell.

 

 

[werwolf969 32]

Thank you for your kind words. I'm just trying to be helpful to the community.

 

1 hour ago, BitTitan said:

LMM has conflict detection

LMM/LML does not have any conflict detection. It will overwrite files with the same name, choosing only by their order in the mods.xml file inside the LML folder. If you enable the overwrite flag for a mod, then the LML will load this mod files exclusively last. And if you have two mods with the overwrite flag enabled, and they have same file to load, then it will load the last one in the load order.

So, to be sure you haven’t messed up your load order, it’s best to make sure you don’t have any of the same files in any mods. Do a manual merge for the same files, put them in a separate folder, create an install.xml file for them, and rename the original mod files that you replaced with something.ext_org so LML doesn’t load them at game start.

1 hour ago, BitTitan said:

mkdir -p nested\${RemainingPath}\${MatchingFile}.txt

I'm not sure if I fully understand the problem you are trying to solve. However, I assume that you want to create an .xml file and then upload all the files from it to a separate folder, like making a backup or something? If you want, PM me, and we'll try to figure something out.

Edited November 6 by werwolf969

[TheCubanMissile2 1]

I've been doing some testing and have the train blimps mod installed. After completing a mission a bunch of blimps get removed from 9 to 3 or 4. Anyone else experience this? I have my lml mods disabled for now and a couple asi mods.

[JKL409 8]

Did this original issue ever get fixed or was progress made on it...? I'm unclear as I've not been checking the thread properly for a few months now. 

[xerofilontoXerofilonto 1]

Hello, this research on trying to combat the problem is amazing but we need to present it in a more straighforward way maybe in other post or something so that people who want to patch out their own texture mods can do so without scrolling various pages of the forum for nesessary information i myself i'm very confused but i haven't had time to read all the comments in this post

[werwolf969 32]

4 hours ago, JKL409 said:

Did this original issue ever get fixed

This problem has not been fixed since the RAGE engine mods were introduced, only partially solved through workarounds such as expanding memory through third-party plugins and packaging mods in RPF archives that are understandable to the engine. For GTA V, it took several years until such tools were created, the RDR 2 community is an order of magnitude smaller, so everything is done longer here. Perhaps the recent release of the RDR remaster will help speed up the appearance of such tools.

 

1 hour ago, xerofilontoXerofilonto said:

more straighforward way

So far, there are guesses posted here, some attempts to solve the problem, but there is no solution. It will be correct to wait for the solution to appear, and I am sure it will be issued as a separate mod on all sites.

[Edegon 11]

@werwolf969 Thanks a lot for the script but I find faster to make the install.xml manually on Notepad++. IDK if I'm using it wrong, but if I'm not then it could be optimized. I still thank you because you took your time to make it for free and without it I wouldn't be able to have an idea on how install.xml works.

Anyway the way I do it manually is to check if all the mod files are in the same .rpf, if they are then they can be all put in a folder (let's call it folderX) inside the mod folder and then I use this as base and it's done. It's way faster than copy-pasting all and each of the files I need to replace inside the source_paths.txt for your script.
 

<EasyInstall>
    <Name>ModName</Name>
    <Author>AuthorName</Author>
    <Version>1.0</Version>
    <Resources>
        <Resource>
            <FileReplacements>
                <GamePath>platform:/pathOfTheRPF</GamePath>
                <FilePath>folderX</FilePath>
            </FileReplacements>
        </Resource>
    </Resources>
</EasyInstall>

If they aren't all in the same .rpf then I just make different folders for each .rpf (putting the mod files in the right folders) and add them in like this:
 

<EasyInstall>
    <Name>ModName</Name>
    <Author>AuthorName</Author>
    <Version>1.0</Version>
    <Resources>
        <Resource>
            <FileReplacements>
                <GamePath>platform:/pathOfTheRPF1</GamePath>
                <FilePath>folderX</FilePath>
            </FileReplacements>
            <FileReplacements>
                <GamePath>platform:/pathOfTheRPF2</GamePath>
                <FilePath>folderY</FilePath>
            </FileReplacements>
        </Resource>
    </Resources>
</EasyInstall>

[Click and drag to move]

I repeat that I'm not an expert and am doing this imitating other mods install.xml and don't fully understand how this stuff works, it's just intuition and assumptions.

Anyway for files with duplicates I still use your script because I don't know how that works.

 

Edit: Here's an example on how I did it with Beta Wrinklemasks by SomeClassyToast
 

<EasyInstall>
    <Name>Wrinklemasks</Name>
    <Author>SomeClassyToast</Author>
    <Version>1.0</Version>
    <Link>https://www.nexusmods.com/reddeadredemption2/mods/1865</Link>
    <Resources>
        <Resource>
            <FileReplacements>
                <GamePath>platform:/packs/base/models/metaped_textures_cutscene_a_f</GamePath>
                <FilePath>metaped_textures_cutscene_a_f</FilePath>
            </FileReplacements>
            <FileReplacements>
                <GamePath>platform:/packs/base/models/metaped_textures_cutscene_g_l</GamePath>
                <FilePath>metaped_textures_cutscene_g_l</FilePath>
            </FileReplacements>
            <FileReplacements>
                <GamePath>platform:/packs/base/models/metaped_textures_cutscene_m_r</GamePath>
                <FilePath>metaped_textures_cutscene_m_r</FilePath>
            </FileReplacements>
            <FileReplacements>
                <GamePath>platform:/packs/base/models/metaped_textures_cutscene_s_z</GamePath>
                <FilePath>metaped_textures_cutscene_s_z</FilePath>
            </FileReplacements>
            <FileReplacements>
                <GamePath>platform:/packs/base/models/metaped_textures_player</GamePath>
                <FilePath>metaped_textures_player</FilePath>
            </FileReplacements>
        </Resource>
    </Resources>
</EasyInstall>

The mod folder:


The files inside the first folder:

Edited November 7 by Edegon

[werwolf969 32]

31 minutes ago, Edegon said:

but I find faster to make the install.xml manually on Notepad++

Well, in your example you use FileReplacements node, that replace files from specified folder. You could even use it without gamepath, LML still will find proper rpf where files needs to be replaced. Issue with this method, as I found by testing, is performance decrease in the game. But it’s indeed an option and you don’t need any script to do that, just some copy pasting into folders and few edits in install.xml.

[Edegon 11]

6 minutes ago, werwolf969 said:

Well, in your example you use FileReplacements node, that replace files from specified folder. You could even use it without gamepath, LML still will find proper rpf where files needs to be replaced. Issue with this method, as I found by testing, is performance decrease in the game. But it’s indeed an option and you don’t need any script to do that, just some copy pasting into folders and few edits in install.xml.

Oh I see. Thanks again for the script

[UncleISRED 2]

9 hours ago, werwolf969 said:

Well, in your example you use FileReplacements node, that replace files from specified folder. You could even use it without gamepath, LML still will find proper rpf where files needs to be replaced. Issue with this method, as I found by testing, is performance decrease in the game. But it’s indeed an option and you don’t need any script to do that, just some copy pasting into folders and few edits in install.xml.

werwolf u are an absolute gentleman my good sir!! I was wrong about the configs too the AB10 is best for most consistent train spawns - thank you so much for the tools for utilizing the replacement method, I've replaced all my textures and my stream folder is now empty and everything is replaced and working wonderfully, not noticed any difference in spawns from replacement alone, mostly the config mixed with systemsettings.ymt - I mean't to ask, the system.xml in docu, should that be locked to keep the perframe=25 along with the systemsettings.ymt or best to keep it un-hidden if using systemsettings.ymt

 

absolute legend tho - you've helped me sososo much with the informaiton n tools you've shared <3

 

EDIT: further testing is showing Tkoul config may be better, I rarely see ambient trains with AB10 only when near towns etc

Edited November 8 by UncleISRED

[werwolf969 32]

2 minutes ago, UncleISRED said:

absolute gentleman

Thank you kindly. There’s many people in this community, who are done much more than me, they’re true legends.

3 minutes ago, UncleISRED said:

system.xml in docu, should that be locked

Yes, make it read-only after you finish with the changes. So game not overwrite it when started.

[Edegon 11]

19 hours ago, TheCubanMissile2 said:

I've been doing some testing and have the train blimps mod installed. After completing a mission a bunch of blimps get removed from 9 to 3 or 4. Anyone else experience this? I have my lml mods disabled for now and a couple asi mods.

 

Try changing the hour with Rampage and see if the number of blips changes

Edit: Thinking about it, there should always be at least 4 train blips, since trams count as trains. If it actually goes down to 3 there's a serious problem, if it goes to 4 it means there are no trains at all. Are you using a save file used with mods installed?

Edited November 7 by Edegon

[Edegon 11]

I don't know how to make one (yet), but I think a script to fill source_paths.txt semi-automatically can be done (I guess it should also be easy), and would be great since it's very time consuming to copy-paste stuff hundreds of times.
My idea was that some mods replace files that are inside a single .rpf, so you have a folder full of files that have the same path. I guess a script can be made where the user puts the .rpf path and the mod's folder path and it makes the list automatically.

Edit: made one with ChatGPT:
removed
 

Feel free to edit and upgrade it.

 

Edit 2: I broke it, will update it as soon as I fix it.

 

Edit 3: Now it works (don't forget to delete the .ytd from the path if you copy it from a file):

removed

 

Edit 4: No, it didn't. Sorry, this time I will test it more carefully before posting it.

Edit 5: Seems to be working right this time:
https://pastebin.com/70cFWVUv

 

Edited November 7 by Edegon

[werwolf969 32]

1 hour ago, Edegon said:

copy-paste stuff hundreds of times

You can search for each file inside the .rpf archive in CodeX or OpenIV using a mask like *.ext or *.* If you have different file types in your mod, just copy all the results into source_paths.txt. The script will choose only files from the source folder, so you won't bloat your install.xml file. Or better, just do one round of copying and pasting the most common file types like *.ymt, *.ytd, and *.ydd, etc., to source_paths.txt and then use that same source_paths. txt for every mod you want to create install.xml. I think it's much simpler way.

[Edegon 11]

16 minutes ago, werwolf969 said:

Or better, just do one round of copying and pasting the most common file types like *.ymt, *.ytd, and *.ydd, etc., to source_paths.txt

 

I don't know how to do it, it was the first thing I wanted to do when I got your script but don't know how

Edit: I mean, I can copy the path of multiple files in mass or you mean to manually copy-paste every single *.ymt, *.ytd, etc. path? Because just the .ytd files are about 90000.

Edited November 7 by Edegon

[werwolf969 32]

11 hours ago, Edegon said:

I don't know how to do it

Here's the source_paths.txt file with all possible paths to every file in the rpf archives.

However, my install.xml generator script won't work with such a large file. You need to shrink it down to only the files you need to replace from mod, or modify my script so it use different method for parsing source_paths.txt.

The source_paths.txt only exists because we can't parse the rpf archives without CodeX or OpenIV, and neither of those has any CLI or API for interacting with them.

My logic for creating the install.xml generator was that mod authors might make mistakes like:

1) wrong file name;

2) wrong mount point in the mod install.xml, or wrong method for some files (like replacing a file that doesn't exist inside the rpf archive, or streaming a file that can be replaced);

3) duplicates in paths in install.xml.

By manually adding paths to source_paths, we check 1 and 2, and 3 is resolved by the script. But if you certain that mod files are inside single rpf archive, you totally can make much simpler script, in this case, you really don't need source_paths.txt. you just paste paths inside xml nodes directly with correct pattern. I think such script was posted few pages ago.

source_paths.7z

Edited November 8 by werwolf969