[WIP] Mator Smash

[Mator]

 

Mator Smash

VIDEO TUTORIAL - For v0.0.7

GamerPoets Video - For v0.4.1

 

Mator Smash is now available on Nexus Mods. 

Please use the Nexus Mods Mod page for all future discussion.

Nexus Mods Mod Page

 

 

 

As of v1.0.0, Mator Smash has a "Quick Patch" button

As of v0.5.1, Mator Smash will copy records with "errors" in them (such as unresolved/unexpected references)

As of v0.5.0, Mator Smash has new algorithm features and is more stable.

As of v0.4.1, Mator Smash merges redundant plugins properly.

As of v0.4, Mator Smash now produces complete record prototypes

As of v0.3, Mator Smash supports Fallout 4 and Skyrim SE

As of v0.2.2, Mator Smash allows you to modify tags on plugins from within the program.

As of v0.2.1, Mator Smash has Smash Settings for Skyrim for every Bash Tag Wrye Bash offered for Oblivion.

 

Recent Changes:

v1.0.0
- No longer cleaning masters (the smashed patch will require all patched plugins as masters)
- Increased saturation of profile panel colors
- Removed unimplemented buttons from the QuickBar
- Removed MO Integration - it's unnecessary
- Added easter egg
- Added "quick patch" button
- Fixed FO3/FNV load order issue
- Added Fallout3 and FalloutNV Smash.All settings  
- Updated FO4 and TES4 Smash.All settings
- Fixed switch smash profile button

 

v0.5.1
- Fixed inactive plugins being enabled for SSE/FO4
- Updated Smash.All so Perk Effect Conditions are treat as single entity
- Fixed case sensitivity when determining if patch destination directory is the game's data path
- Fixed bug with not creating directories when building a patch when user is not using MO
- Now allowing copying of records with "errors" in them
- Improved handling of out of order subrecords

 

v0.5.0
- Added override deletions algorithm feature
- Added force value algorithm feature
- Fixed some SSE/FO4 load order issues
- Added and updated smash settings
- Fixed issue with CTDs due to counter elements being set to 0 when their associated array is not present.
- Fixed issue with cells being deleted incorrectly due to bad ITM/ITPO cleanup code.

 

Purpose
Mator Smash is an application built on the xEdit framework developed to replace the primary functionality of Wyre Bash - the creation of patch files to combine (or merge) conflicting overrides to overcome Bethesda's "Rule of One". Wyre Bash used to provide this functionality in the days of Oblivion through the usage of bash tags, keywords that would be put in the description of the mod that would then tell Wyre Bash what types of records it should try patching. However, the current status is that Wyre Bash only provides this for a handful of record types for Skyrim, and this has been the case for quite a while now. There are several users (most notably Sharlikran) who have been working to update Wyre Bash's codebase to work with Skyrim, but it's been an uphill battle. As of now, it's not certain whether Wyre Bash will ever offer the same functionality it did in the days of Oblivion for Skyrim mods. That's where Mator Smash comes in.

 

Why is this necessary?
As per Skyrim's "Rule of One", any two mods that modify the same record but in different ways will conflict with each other in a way that can't be resolved asides from the creation of a compatibility patch. Because the number of Skyrim mods exceeds 40,000, there is no conceivable way to manually build enough compatibility patches to make every possible load order (or even a small fraction of them) work with each mod operating as intended. Mator Smash automates the most important part of the task of compatibility patch creation - the combining of conflicting edits. This is something that we've been needing for a very, very long time.

 

How does it work?
Warning: technical jargon ahead. Where Wyre Bash has separate procedures for the patching of individual record types, Mator Smash has a few generic procedures for the patching of ALL record types. This means that Mator Smash has, with very little development effort, achieved the capacity to patch more records than Wyre Bash has ever been capable of patching. This is achieved through a recursive traversal method which traverses override record structure, comparing subrecords between a master record, a source override record, and a destination override record. Upon the basis of this comparison, certain subrecords are written to the destination record while others are skipped and others are deleted from it.

 

Development Status
I began development of Mator Smash as a standalone application built on top of the xEdit API 10/18/2015. Here's the GitHub repository. Check out the design folder in the repository for various files relating to my design process. I'll be live streaming most of my development on livecoding.tv. I'll be streaming mainly on weekends 9:00AM-9:00PM PST. Because of major code reuse from merge plugins standalone, smash will be entering public beta very soon.

Mator Smash is currently available as a proof of concept script (which is slow and lacking in features/a clean user experience). You can download v0.9.4 of that script here. This script can currently correctly patch a massive number of different types of records. FAR MORE than Wyre Bash or any other currently available solution, and more than any solution that has ever existed for Skyrim or any other game.

 

Implemented Features

• Dynamic, flexible settings: Smash settings are comprised of a tree which has information on how records/subrecords should be handled. This will allow users to control how a patch is created on a per-subrecord basis. This will make smash relevant regardless of the nature of the mods being patched, or the game the mods come from. Each node (record/subrecord) will have the following data associated with it-
  • Process: Whether or not smash should process this node and perform conflict resolution for it if a conflict is found.
  • Preserve Deletions: If a mod deletes an element inside of this node, that deletion will be preserved in the generated patch.
  • Treat as single entity: If a mod overrides any subrecord in this node, the entire node will be replaced with that mod's version of it.
  • Chaining: Linking allows you to copy multiple side-by-side elements when any one of those elements changes.
  • Override Deletions: Allows a plugin to restore deleted elements.
  • Force Value: Forces values from a particular plugin to be used, and only allows plugins which require this plugin to perform further conflict resolution on affected records.
• Tags: As of v0.2.1, all Bash tags that were offered with Wrye Bash for Oblivion are now available for Skyrim.  Smash will automatically detect tags and apply smash settings to the plugins that have them, assuming you have smash settings that correspond to the tags.
• Managed smashed patches: Patches you build will be tracked in the program, so you can quickly and easily rebuild them when your load order changes. You can also have multiple patches for your load order, if you feel that is desirable.
• Multi-language support: I built a really clean, extensible, and easy to use language system when I developed Merge Plugins Standalone. Smash will use the same language system to support multiple languages out-of-the-box. As a translator, all you have to do is make a text file to create a translation and have users install it in their lang folder. The rest is handled by the program.
• Fast and easy setting creation: Select some plugins to build a setting to specifically handle the conflicts they are capable of creating in a load order.  Toggle or set flags for multiple nodes at once.  Select similar nodes in a setting tree.  Conflict resolution has never been quite so powerful, flexible, and easy.

 

Planned Features

Smash is now pretty much feature complete.  Further development energy will be spent on porting it to a zEdit application mode, zSmash, which will include a new, streamlined workflow.

 

[alt3rn1ty]

In your opinion, what would be the optimal way of ordering your load order prior to using Mator Smash ?

 

Also, what is the output of the program - MatorSmash.esp ? ( forgive me if I missed it in the OP )

 

I hope to be able to contribute here, unfortunately I am just about to get busy with my career again starting tomorrow

[Mator]

In your opinion, what would be the optimal way of ordering your load order prior to using Mator Smash ?

 

Also, what is the output of the program - MatorSmash.esp ? ( forgive me if I missed it in the OP )

 

I hope to be able to contribute here, unfortunately I am just about to get busy with my career again starting tomorrow

 

Optimal way of ordering load order: shouldn't be one.  If multiple mods modify the same subrecord, then the last override value will be what's used, so I suppose that could be taken into consideration.

 

Output of the program is an ESP file of any name the user desires.  The ESP file will be identified as a Smashed patch by the author field.  This allows you to have multiple smashed patches in your load order, but you can't open multiple (>1) smashed patches in xEdit at the same time without hindering the script's functionality.  The script will automatically use the first smashed patch ESP file it can find (based on the author field) if such a plugin is loaded in xEdit.

[alt3rn1ty]

Would there also be any indication when Smash was able to fully merge a mods records, so that those mods could be deselected and not count towards the maximum mods allowed by the game ?

 

And lastly is there a possibility when Zilav brings DBash along, that it could be integrated to be used by Smash for more speed of execution ( I am thinking along the lines of CBash utilised by Python in the Oblivion incarnation of Wrye Bash, which was far swifter at processing large load orders than Python Bash on its own ) .. Probably too early to know.

[alt3rn1ty]

One more - How to scroll a list which is longer than your screen length ..

 

 

 

Bring Out Your Dead is not the last mod in my list

[Mator]

Would there also be any indication when Smash was able to fully merge a mods records, so that those mods could be deselected and not count towards the maximum mods allowed by the game ?

You mean when those mods are comprised of override records only? I can work on developing a notification or log message that could alert the user when a mod is comprised entirely of override records and has been/can be entirely included in a smashed patch.

 

And lastly is there a possibility when Zilav brings DBash along, that it could be integrated to be used by Smash for more speed of execution ( I am thinking along the lines of CBash utilised by Python in the Oblivion incarnation of Wrye Bash, which was far swifter at processing large load orders than Python Bash on its own ) .. Probably too early to know.

I have no idea what DBash is or how it's going to be implemented. If I knew I could probably answer this question. The approach of the script only requires the capacity to traverse records, copy records/subrecords, and create a new ESP file. So, hypothetically, any application that is capable of all of that could use the approach in Mator Smash.

One more - How to scroll a list which is longer than your screen length ..

Bring Out Your Dead is not the last mod in my list

Supposed to be a scrollbar. Guess that must have gotten lost in development. Will fix in v0.8.2, which I'll put out soon (next 30 minutes maybe?).

[alt3rn1ty]

I will have to go after this post, but here's your first out of the wild debug log - Its failed the first run .. But this will be one less problem on the road afterwards

 

Smash failed.  Exception: List index out of bounds (-1)

https://dl.dropboxusercontent.com/u/86040572/Alt3rn1ty-smashfailexception.7z

 

 

Edit : I am on Win 7 x32, with Aero off ( using a classic theme if that makes any diff to the script windows and gadgetry )

[Arthmoor]

I got the same failure after it spent nearly 30 minutes processing

 

Last bit was this:

Smashing record [CELL:0000BCF0] (in Tamriel "Skyrim" [WRLD:0000003C] at 43,-25) from file: Open Cities Skyrim.esp
    CELL \ Worldspace
    Skipping record header.
    CELL \ EDID - Editor ID
      Found differing values: RiftenCitySoutheastOCS and
    CELL \ DATA - Flags
        CELL \ DATA - Flags \ Has Water
        CELL \ DATA - Flags \ Hand Changed
    CELL \ XCLC - Grid
        CELL \ XCLC - Grid \ X
        CELL \ XCLC - Grid \ Y
        CELL \ XCLC - Grid \ Force Hide Land
    CELL \ TVDT - Unknown
    CELL \ MHDT - Max Height Data
    CELL \ LTMP - Lighting Template
    CELL \ XCLW - Water Height
    CELL \ XCLR - Regions
    Sorted array found: CELL \ XCLR - Regions
    CELL \ XLCN - Location
      Found differing values: RiftenLocation "Riften" [LCTN:00018A58] and

Smash failed.  Exception: List index out of bounds (-1)

 

Oddly still generated most of a file from what I can see.

 

IMO, I like what Bash tags bring to the table. Allowing you to selectively decide which types of records in each mod should and should not be carried forward. If I'm reading this right, your script only allows "inclusive" or "exclusive" for the entire plugin with no further management possibilities. And I'm honestly not clear on the difference or why I'd want to change one.

[Mator]

I will have to go after this post, but here's your first out of the wild debug log - Its failed the first run .. But this will be one less problem on the road afterwards

 

Smash failed.  Exception: List index out of bounds (-1)

https://dl.dropboxusercontent.com/u/86040572/Alt3rn1ty-smashfailexception.7z

 

 

Edit : I am on Win 7 x32, with Aero off ( using a classic theme if that makes any diff to the script windows and gadgetry )

Here's what I think happpened-

 

Line where failure occurred:

ini := TMemIniFile(lstSettings[slSettings.IndexOf(slOptions[slFiles.IndexOf(fn)])]);

Why it happened:

slFiles.IndexOf(fn) returned -1 OR

slSettings.IndexOf(..) returned -1

Likelihood of former: Medium.

Likelihood of latter: Should be impossible.

Why would slFiles.IndexOf(fn) return -1?

-> Some filename not added to stringlist. Possible, but unlikely.

-> Some filename added to stringlist but altered. Possible.

Possible fixes:

Break the line into multiple lines and put everything into a try-except block, assigning default setting upon failure.

 

What you should do:

Please use less ESPs when testing.

What I should do:

Test similar plugin smashing with fewer plugins and more debug messages.

[Mator]

IMO, I like what Bash tags bring to the table. Allowing you to selectively decide which types of records in each mod should and should not be carried forward. If I'm reading this right, your script only allows "inclusive" or "exclusive" for the entire plugin with no further management possibilities. And I'm honestly not clear on the difference or why I'd want to change one.

 

I'm sorry that that wasn't clear for you.  Smash does allow you to selectively decide which types of records in each mod should and should not be carried forward.  As I wrote in the OP:

 

 

The setting tells the script what records to process or skip, and what subrecords to process or skip.

 

Basically the exact same words you just said.  Here's an illustration if it still isn't clear:

 

 

 

[Mator]

Uploaded v0.8.3, now catching the exceptions you guys got there, so the script will no longer fail, it'll just print a log message and move on to patching the next record.

 

The log messages should help me to better understand what's going wrong so I can fix the actual problem.

[Arthmoor]

*DING* And the light goes on. Yes, that makes things more clear. Though somewhat painful since it'll require manually going through everything to set that stuff up.

 

Would there be any possibility of having Smash be able to read Bash tags from a mod's description field and setting itself up appropriately based on those? Maybe as an initial setup option or something?

[Mator]

*DING* And the light goes on. Yes, that makes things more clear. Though somewhat painful since it'll require manually going through everything to set that stuff up.

 

Would there be any possibility of having Smash be able to read Bash tags from a mod's description field and setting itself up appropriately based on those? Maybe as an initial setup option or something?

 

Already planned:

 

 

Because of exclusion you can just use the default setting for most mods without having problems.  There will also be more setting presets packaged with Mator Smash in the future for users to use right out of the box.

[Arthmoor]

If I may suggest, you should permanently ignore the processing of navmesh and landscape records (NAVM, NAVI, LAND). They cannot be effectively merged and produce useful results in the game. Yet they seem to be taking up the bulk of the processing time. Nearly 40 minutes to run an out of the box default set on 100 mods. Seems like a lot of mods, but I've seen plenty of people with twice that. If it takes them 80 minutes to run this, they won't bother

 

You'll probably also want to look at this thread: http://www.afkmods.com/index.php?/topic/3940-skyrim-tes5edit-records-that-merge-at-runtime/ for stuff that shouldn't be processed as well. Records that already merge during runtime don't need to be handled by a tool like this and it would just waste time to do so.

[Mator]

If I may suggest, you should permanently ignore the processing of navmesh and landscape records (NAVM, NAVI, LAND). They cannot be effectively merged and produce useful results in the game. Yet they seem to be taking up the bulk of the processing time. Nearly 40 minutes to run an out of the box default set on 100 mods. Seems like a lot of mods, but I've seen plenty of people with twice that. If it takes them 80 minutes to run this, they won't bother

 

You'll probably also want to look at this thread: http://www.afkmods.com/index.php?/topic/3940-skyrim-tes5edit-records-that-merge-at-runtime/ for stuff that shouldn't be processed as well. Records that already merge during runtime don't need to be handled by a tool like this and it would just waste time to do so.

 

I've seen that thread and that's how I built the default setting.  The default setting skips all of the records/subrecords mentioned in that thread.

 

When a record is set to be excluded, it isn't processed any more than is necessary.

        // build list of records with multiple overrides
        lbl.Caption := 'Processing '+fn;
        LogMessage('Processing '+fn);
        application.processmessages;
        ini := TMemIniFile(lstSettings[slSettings.IndexOf(slOptions[k])]);
        records := StringReplace(ini.ReadString('Setting', 'records', ''), '#13', #13#10, [rfReplaceAll]);
        recordMode := ini.ReadString('Setting', 'recordMode', '0');
        for j := 0 to RecordCount(f) - 1 do begin
          r := MasterOrSelf(RecordByIndex(f, j));
          if ((Pos(Signature(r), records) > 0) and (recordMode = '0'))
          or ((Pos(Signature(r), records) = 0) and (recordMode = '1'))
          or ((Pos(Signature(r), global_records) > 0) and (global_recordMode = '0'))
          or ((Pos(Signature(r), global_records) = 0) and (global_recordMode = '1')) then
            continue;
          rn := Name(r);
          if (nbsOverrideCount(r) > 1) then
            if slRecords.IndexOf(rn) = -1 then begin
              slRecords.AddObject(rn, TObject(r));
            end;
        end;

^ performed for each file loaded.

 

So for records that are skipped we're perfoming a total of 12 operations. There's no conceivable way I can reduce that.

 

I can assure you, the majority of the processing time isn't being spent on NAVM/NAVI records.  Are LAND records merged at runtime?  They aren't listed in your thread, so I'm not sure.  If so then they can be added to the default setting's record exclusion list.  Honestly though, I can't even find LAND records in TES5Edit...

 

Additional speed and efficiency is attainable, it's just not a priority at the present time.  You're free to tweak your settings to get the maximum speed and reliability while still patching what you want to have patched.  That's why they're there.

 

The majority of processing time is probably being spent dealing with the unofficial patches, assuming you have them loaded.  Set them to skip if you want to patch things faster.

 

EDIT: I just patched a load order of over 140 ESP files, skipping the unofficial patches, with the default setting in 06:14 (6 minutes 14 seconds).  I disabled all of the debug booleans, which made a large difference in execution time.  My load order includes a lot of merged plugins, so the actual number of unmerged plugins is probably closer to 200.

 

EDIT 2: Including the unofficial patches more than doubles the number of records to be patched, from 1420 to 3054.  So I'm right about them soaking up the majority of execution time.  Will edit again when this session of the script finishes executing.

 

EDIT 3: 16:33 for my load order with the unofficial patches.  So they added 10 minutes to the script's execution.

[Arthmoor]

Some quick things.

 

Properties on some scripts are being ignored:

 

 

Graphics changes for some things are getting ignored:

 

Winning records are being unnecessarily duplicated into the patch:

 

Dialogue INFO records cannot be merged like this. It changes the fundamental meaning of the entire CDTA:
 

 

[zilav]

I wanted to post my ideas on how bashing should work long time ago, but always getting distracted.

We all know that PBash stands for Python Bash, CBash for C/C++ Bash, so DBash is Delphi Bash according the programming language it uses. Sounds kind of dumb, but hey, it is not me who started it!

 

The main problem of all bashers is the requirement of so called tags, or a set of named logical groups that point to the specific data in records to process by patcher. Processing includes copying their values to the bash patch from the winning override record having the same tag. Pretty simple principle that worked for years in Oblivion and to some point in Fallouts. "Lists" include a slightly different processing method like removing elements from them, but still it is the same base principle.

 

How do mod authors (and BOSS/LOOT maintainers) assign tags to mods? They check them in xEdit and see what that mod changes from it's master files. Why? Because PBash/CBash doesn't have a conflict detection mechanism, it is simply a library to load esp/esm data, modify it or create new one, and save. It is certainly possible to add conflicts detection in those bashers too, but that will require:

1) Someone very knowledgable to pick up their development

2) Add additional definitions for fields that should be ignored (treated as ITM) similar to xEdit

 

The 6(?) years history of Wrye Bash showed it is unlikely to happen and I really don't know why. Python and C++ are far more popular languages than Delphi (object pascal), and still noone from programmers is willing to even maintain them regulary, not even talking about improving or rewriting bashing code. I admire Sharlikran persistence, but he obviously lacks the knowledge, at least for now. Maybe he will learn enough some day

 

So here comes the main idea of DBash - use xEdit's features to detect changes and create bashed patch at the same time. Actually what this script tries to do. However I believe that meaning of tags are still needed when bashing.  By "meaning" I understand the logical groups of data that should be processed together. Lets demonstrate it on 2 made up examples.

 

1. User wants to spice up his Skyrim and installs a pair of mods. First adds attacks of bandits on Whiterun and doubles the health of guards to compensate them. Second mod from a different author adds attacks of necromancers and doubles the level of guards instead. When bashing everything that changes guards will end up with double health and level and will easily destroy any opposition, probably making the game even easier than before. This is certainly not the effect user expects when using those mods together. Keep in mind I don't say this script will do that, it is just an example why bashing all changes is undesirable.

 

2. More real life example. A cell record with changes in XCLL lighting, LTMP template and XCIM image space. All those values, and only TOGETHER will provide the correct visuals in game, especially for mod that changes interior lighting since, while creating a lighting mod, mod author sees the net effect of all those values as a whole. If one lighting mod modifies lighting colors, and second one changes imagespace, bashing those changes will most likely cause a visual mess in game. However there are some exclusions like if a mod modifies only fog in interiors, it can be bashed with other lighting mods.

 

And that's why tags are still necessary to define the grouping of data in records that should be analyzed and propagated into a bashed patch together. But mods are not needed to be tagged beforehand since changes can be determined by xEdit on the go. Tags can be defined using config files that point to particular data in records. For the first example with guards

{ACBS.Stats}

NPC_ \ ACBS \ Level

NPC_ \ ACBS \ Calc min level

NPC_ \ ACBS \ Calc max level

NPC_ \ DNAM \ Health

 

There tags for NPCs already exist, just showing as to how they can be defined and configured.

 

Second example for interior fog changing mod

{C.Fog}

CELL \ XCLL \ Inherits [Flags] Fog near, Fog far, Clip distance, Fog power, Fog max

CELL \ XCLL \ Fog Near

CELL \ XCLL \ Fog Far

CELL \ XCLL \ Fog Clip Distance

CELL \ XCLL \ Fog Power

CELL \ XCLL \ Fog Max

 

The first line could mean particular flags in Inherits flags since they affect from where values will be taken - XCLL subrecord or LTMP Lighting template.

If mod is already tagged by the author in TES4 header description, DBash can easily use those settings and don't try to detect other changes itself. It would also be possible to skip tags completely and allow mod author to explicitly add specific datas he wants to be bashed from his mod in the mod's description field, using this syntax for example

 

{

NPC_ \ HCLF - Hair Color

NPC_ \ NAM9 - Face morph

}

 

As you can see, the possibilities are unlimited and could range from full changes autodetection to bashing of specific fields for any mod if desired. This is my idea of how bashing should work, and how DBash can work if I ever get time to write it. I believe that blind "smashing" of any changes is not the right approach to the problem. And while it can be customized too, I doubt a lot of people will bother with it. We already have a working list of tags definitions that comes from years of modding experience, better to use it as a base and expand instead of creating something new.

[KNakamura]

The reason why only Sharlikan is that Utumno is deep in refactoring: https://github.com/wrye-bash/wrye-bash/commits/dev and is tagging any improvement to code for 307.

 

It's a good idea, but means any improvements to Bash are going to take a while.

 

(I'd say more but I'm a rather poor C# programmer so this is all beyond my knowledge.)

[Mator]

Some quick things.

 

Properties on some scripts are being ignored:

I'm aware of this.  Just one of the three current array element merging issues.

 

Graphics changes for some things are getting ignored:

Wasn't aware of this.  Looks like an element assignment issue.

 

Winning records are being unnecessarily duplicated into the patch:

This record is actually different from the winning override, if you look at the KWDA - Keywords element.  However, I am aware of this as well.

 

Dialogue INFO records cannot be merged like this. It changes the fundamental meaning of the entire CDTA:

Already aware of this.  This is because I'm currently using recursive traversal for all unsorted arrays instead of using a MergeUnsortedArray function that attempts to determine element uniqueness and then performs operations similar to MergeSortedArray.

 

Thanks for the notes, however, it should be noted that the type of record is actually irrelevant to Mator Smash.  All records are dealt with by the same functions, so the issue isn't one of not handling a type of record correctly but of subrecord data structure.

[Mator]

I wanted to post my ideas on how bashing should work long time ago, but always getting distracted.

We all know that PBash stands for Python Bash, CBash for C/C++ Bash, so DBash is Delphi Bash according the programming language it uses. Sounds kind of dumb, but hey, it is not me who started it!

 

The main problem of all bashers is the requirement of so called tags, or a set of named logical groups that point to the specific data in records to process by patcher. Processing includes copying their values to the bash patch from the winning override record having the same tag. Pretty simple principle that worked for years in Oblivion and to some point in Fallouts. "Lists" include a slightly different processing method like removing elements from them, but still it is the same base principle.

 

How do mod authors (and BOSS/LOOT maintainers) assign tags to mods? They check them in xEdit and see what that mod changes from it's master files. Why? Because PBash/CBash doesn't have a conflict detection mechanism, it is simply a library to load esp/esm data, modify it or create new one, and save. It is certainly possible to add conflicts detection in those bashers too, but that will require:

1) Someone very knowledgable to pick up their development

2) Add additional definitions for fields that should be ignored (treated as ITM) similar to xEdit

 

The 6(?) years history of Wrye Bash showed it is unlikely to happen and I really don't know why. Python and C++ are far more popular languages than Delphi (object pascal), and still noone from programmers is willing to even maintain them regulary, not even talking about improving or rewriting bashing code. I admire Sharlikran persistence, but he obviously lacks the knowledge, at least for now. Maybe he will learn enough some day

 

So here comes the main idea of DBash - use xEdit's features to detect changes and create bashed patch at the same time. Actually what this script tries to do. However I believe that meaning of tags are still needed when bashing.  By "meaning" I understand the logical groups of data that should be processed together. Lets demonstrate it on 2 made up examples.

 

1. User wants to spice up his Skyrim and installs a pair of mods. First adds attacks of bandits on Whiterun and doubles the health of guards to compensate them. Second mod from a different author adds attacks of necromancers and doubles the level of guards instead. When bashing everything that changes guards will end up with double health and level and will easily destroy any opposition, probably making the game even easier than before. This is certainly not the effect user expects when using those mods together. Keep in mind I don't say this script will do that, it is just an example why bashing all changes is undesirable.

 

2. More real life example. A cell record with changes in XCLL lighting, LTMP template and XCIM image space. All those values, and only TOGETHER will provide the correct visuals in game, especially for mod that changes interior lighting since, while creating a lighting mod, mod author sees the net effect of all those values as a whole. If one lighting mod modifies lighting colors, and second one changes imagespace, bashing those changes will most likely cause a visual mess in game. However there are some exclusions like if a mod modifies only fog in interiors, it can be bashed with other lighting mods.

 

And that's why tags are still necessary to define the grouping of data in records that should be analyzed and propagated into a bashed patch together. But mods are not needed to be tagged beforehand since changes can be determined by xEdit on the go. Tags can be defined using config files that point to particular data in records. For the first example with guards

{ACBS.Stats}

NPC_ \ ACBS \ Level

NPC_ \ ACBS \ Calc min level

NPC_ \ ACBS \ Calc max level

NPC_ \ DNAM \ Health

 

There tags for NPCs already exist, just showing as to how they can be defined and configured.

 

Second example for interior fog changing mod

{C.Fog}

CELL \ XCLL \ Inherits [Flags] Fog near, Fog far, Clip distance, Fog power, Fog max

CELL \ XCLL \ Fog Near

CELL \ XCLL \ Fog Far

CELL \ XCLL \ Fog Clip Distance

CELL \ XCLL \ Fog Power

CELL \ XCLL \ Fog Max

 

The first line could mean particular flags in Inherits flags since they affect from where values will be taken - XCLL subrecord or LTMP Lighting template.

If mod is already tagged by the author in TES4 header description, DBash can easily use those settings and don't try to detect other changes itself. It would also be possible to skip tags completely and allow mod author to explicitly add specific datas he wants to be bashed from his mod in the mod's description field, using this syntax for example

 

{

NPC_ \ HCLF - Hair Color

NPC_ \ NAM9 - Face morph

}

 

As you can see, the possibilities are unlimited and could range from full changes autodetection to bashing of specific fields for any mod if desired. This is my idea of how bashing should work, and how DBash can work if I ever get time to write it. I believe that blind "smashing" of any changes is not the right approach to the problem. And while it can be customized too, I doubt a lot of people will bother with it. We already have a working list of tags definitions that comes from years of modding experience, better to use it as a base and expand instead of creating something new.

 

 

Already planned:

 

 

Because of exclusion you can just use the default setting for most mods without having problems.  There will also be more setting presets packaged with Mator Smash in the future for users to use right out of the box.

 

I can see how bashing/smashing certain subrecords can lead to undesirable results, however I still don't fully understand how bash tags allow a program to gain an understanding of when to combine records and when not to, unless your solution is to not traverse into certain elements and copy them entirely (or perhaps even with groups of elements).

 

I'm not against tagging and it is planned, though I still believe that general traversal should work in 90% of situations.

[Leonardo]

I wanted to post my ideas on how bashing should work long time ago, but always getting distracted.

We all know that PBash stands for Python Bash, CBash for C/C++ Bash, so DBash is Delphi Bash according the programming language it uses. Sounds kind of dumb, but hey, it is not me who started it!

 

<snip>

 

The 6(?) years history of Wrye Bash showed it is unlikely to happen and I really don't know why. Python and C++ are far more popular languages than Delphi (object pascal), and still noone from programmers is willing to even maintain them regulary, not even talking about improving or rewriting bashing code. I admire Sharlikran persistence, but he obviously lacks the knowledge, at least for now. Maybe he will learn enough some day

 

<snip>

I have said it before I am not a programmer, but I do know that Pascal is a lot faster than C++ and some test shows that Delphi is about 10 times faster than C++.

 

One explanation would be is that Pascal has already translated the source code into machine code once the source code has been compiled hence for larger files and don't you have to rely on the runtime libraries (which is the reason why you see so many files in Windows and that's written in C++ IIRC) in C++ or in Visual Basic such as the old ones e.g vbrun100.dll, vbrun200.dll, vbrun300.dll in Windows 3.x or in Windows 95/98.

 

So should you miss any of these runtime libraries the program stops working, but that's not the case for either Delphi or Pascal and yet C++, VB are far more popular than Delphi or Pascal thanks to M$.

 

I really wish I could learn how to become a programmer I do, but I think I'm born in the wrong decade.

 

 

Anyway, I appreciate anything that could improve the tools the community already have not only for Oblivion and Skyrim, but also for Morrowind despite for having a different handling of any objects in-game.

[Mator]

I have said it before I am not a programmer, but I do know that Pascal is a lot faster than C++ and some test shows that Delphi is about 10 times faster than C++.

 

One explanation would be is that Pascal has already translated the source code into machine code once the source code has been compiled hence for larger files and don't you have to rely on the runtime libraries (which is the reason why you see so many files in Windows and that's written in C++ IIRC) in C++ or in Visual Basic such as the old ones e.g vbrun100.dll, vbrun200.dll, vbrun300.dll in Windows 3.x or in Windows 95/98.

 

So should you miss any of these runtime libraries the program stops working, but that's not the case for either Delphi or Pascal and yet C++, VB are far more popular than Delphi or Pascal thanks to M$.

 

I really wish I could learn how to become a programmer I do, but I think I'm born in the wrong decade.

 

 

Anyway, I appreciate anything that could improve the tools the community already have not only for Oblivion and Skyrim, but also for Morrowind despite for having a different handling of any objects in-game.

 

What?  Can you please tell me where you came across this information?  My experiences (as a Computer Science major) are in direct contrast with what you're saying here.

[zilav]

I can see how bashing/smashing certain subrecords can lead to undesirable results, however I still don't fully understand how bash tags allow a program to gain an understanding of when to combine records and when not to, unless your solution is to not traverse into certain elements and copy them entirely (or perhaps even with groups of elements).

 

I'm not against tagging and it is planned, though I still believe that general traversal should work in 90% of situations.

I'm not for tagging mods, actually vice versa. I want to get rid of them in most cases when possible. I'm just saying that "tags" should serve as a logical grouping of different fields that must always be copied together even when only a single value has changed out of dozens assigned to a tag. And those groups must be statically defined in some config file, similar to how tags are hardcoded in Wrye Bash code.

 

Look at the cell's fog example. Even when lets say only a XCLL \ Fog Near value was changed in some mod, you must still copy all other tied fog fields too to get a consistent and desirable effect in game. Recursively going through all fields in a record and "smashing" all changed fields together in a bashed patch is undesirable.

[alt3rn1ty]

The idea behind LOOT ( or originally with BOSS ) and paring up with Wrye Bash as I see it was this :

 

You organise your conflicting mods ( forget the rest which do not conflict, they can be ordered anywhere ) - And the plugin which includes the most conflicts loads first.

 

Then logically the plugins with less changes to the game get a chance to be in game ( for people in the community who have no idea on a deeper level what any particular mod contains, and have no clue how best to manually arrange their load order, at least they do not have the rest of their load order overridden by one overhaul mod loading last )

 

Then earlier loading mods can have tagged conflicts .. at the users choice ( Bashed patch patchers ) .. pulled into the bashed patch loading last

 

 

The whole provides less complexity / knowledge required for users modding the game, and a simple idea for making parts of mods win over others at the users choice.

 

This worked really well for Oblivion http://static-1.nexusmods.com/15/mods/101/images/35230-1-1343151466.png

 

 

So .. to my question : ( Which I realise depends upon how smash goes with its planned tags implementation, however comprehensive that may or may not be .. I have read everything written so far )

 

Tags if not in the mods description, can also be sourced in LOOTs masterlist.yaml.

 

Would Smash also be able to utilise that masterlist for users of LOOT, for the mods where the author does not provide them in the description field of the plugin ?

 

( Wrye Bash also refers to the same, if no tags are provided in the plugin, it updates its own tag list from the more up to date masterlist when BOSS or LOOT is installed and being used )

 

 

Edit : Trying out 0.8.4 tonight unless another version comes along see if I can find anything ( doubt if I will, more experienced eyes than mine have already been there - But will give it a go anyway )

[alt3rn1ty]

You mean when those mods are comprised of override records only? I can work on developing a notification or log message that could alert the user when a mod is comprised entirely of override records and has been/can be entirely included in a smashed patch.

 

^^ Forgot to answer that bit - Yes, like Multiple Floors Sandboxing ( its just a couple of GMST .. I think a more recent version of Sharlikrans Wrye Bash updates have these as "Tweak Settings" Patchers ), the plugin is green text and appears in the Merge Patches ( also see Didacts Clothing and Armor fixes WIP on this forum for another example ) ..

 

De-selected ( + instead of tick ) in Wrye Bash, due to being merged in the bashed patch

 

Also ...

 

----------------------

 

Confirmation scroll bar working

 

 

.. But I noticed that after choosing Apply Script, then selecting the script from the list of Scripts and running it - The Apply Script dialogue is still showing behind the Smash Selection dialogue ( its like the graphics do not refresh underneath the Smash window )

 

If you also drag the Smash window over the top of TES5Edit window to the left ( list of plugins ), when you then drag it to the right the TES5Edit list of plugins becomes graphically blanked .. Which also feels like a graphical refresh on overlapping windows is getting lost