[WIPz] TES5LODGen

[Sheson]

Are you working on something like Grieche's Skyrim Distance Overhaul?

 

Yes and then some. It requires LODGen to produce static object LOD.

 

It doesn't support non square source images unoirtunately.

 

Hmmm then imagemagick montage comes to mind. Since you planted the seed I will probably keep pondering over this in the next months

[zilav]

Hmmm then imagemagick montage comes to mind. Since you planted the seed I will probably keep pondering over this in the next months

I have code ready for dynamic creation of atlases (and decomposing existing ones into separate sprites), so will probably add TES5TreeLODGen in near future that will make CK obsolete for generating lod for trees. I'm now thinking on the format of source tree billboards, their names and location. Something like plugin\formid.png (or bmp, tga, dds).

textures\terrain\lodgen\Skyrim.esm\0001306D.png

 

If I can take width and height in game units from object bounds field in TREE records, then I won't need anything else. If not, then probably get them from existing LST files and store in a text file with the same name. I really don't want to read tree nif models for that.

[Jon]

Trees LOD has randon angles to avoid repetitiveness and different shading compared to objects. Making them as objects lod causes a big drop in quality afaik.

 

Sheson was talking about rotation about X and Y which is not possible with the tree LOD system.  I talked about this a bit over PM with you already,  but to reiterate, some trees need to be rejected if they deviate too far from being vertical.  For example, there are many trees that lay down on the ground which are not currently ignored during BTT generation.

 

It looked something like this:

 

          xRot := GetElementNativeValues(r, 'DATA\Rotation\X');
          yRot := GetElementNativeValues(r, 'DATA\Rotation\Y');
          if ( (xRot > 30.0 ) and (xRot < 330.0)  ) or ( (yRot > 30.0) and (yRot < 330.0) ) then begin
            z := -30000.0;
          end; 

 

Making each tree as an object would not be a big drop in quality, but in performance.  A massive one.  The trees are most certainly batched in some way to reduce draw calls.  Each tree that is treated as its own object will not be batched.

 

Like I said in PM, the BTT patcher should reject such trees, and let something like Sheson's mod handle more "dynamic" trees like driftwood, fell trees, etc.   Also disabling (i.e. setting to -30000.0 Z) billboards for anything with enable parents.

 

 

I have code ready for dynamic creation of atlases (and decomposing existing ones into separate sprites), so will probably add TES5TreeLODGen in near future that will make CK obsolete for generating lod for trees. I'm now thinking on the format of source tree billboards, their names and location. Something like plugin\formid.png (or bmp, tga, dds).

textures\terrain\lodgen\Skyrim.esm\0001306D.png

 

If I can take width and height in game units from object bounds field in TREE records, then I won't need anything else. If not, then probably get them from existing LST files and store in a text file with the same name. I really don't want to read tree nif models for that.

 

I'm curious how you plan to do this.  _lod_flat files must point to one texture and one texture only,  that of the worldspace tree lod file.  For Tamriel that'd be Terrain/Tamriel/Trees/TamrielTreeLOD.dds

 

_lod_flat files will simply not work if they try to point to more than one DDS file.  You said something about bypassing the UVs of the NIF files..  You can't do this because the vanilla lod_flat NIFs (and custom ones for that matter) can use slightly different UVs for each side of the X-shaped billboards, that is they don't adhere strictly to the bounds of the tree mesh, nor to each other.  It's not strictly that of the min/max of the tree texture or the NIF bounds, but the min/max of whatever shapes you create in the _lod_flat file.  A tree billboard may be arbitrarily stretched horizontally or vertically to better fill the area, or scaled up by 10% for example to appear more "full" from afar.  If you're going to make some kind of atlas generator you need to read the UVs from the _lod_flat NIF file first, and then use them to piece together the atlas.

 

If you somehow do trick the Tree LOD system into referencing more than one DDS for one worldspace, you're kind of undoing the point of the tree LOD system,  which is to batch all the trees into as few draw calls as possible by using an atlas.

 

There are also the positioning issues we discussed via PM...  The Reach trees and the "Tundra Driftwood" trees have extremely poorly placed origins in the full LOD NIF files.  To place proper tree LOD for these I had to offset the billboard position manually in the BTT patcher for those trees.   This could possibly be done via the LST file instead, since Sheson provided the decoded format.   A user friendly way would be to have a kind of crosshair/icon over top of the atlas generator UI for the individual tree files so that you could manually place an origin for the tree, which the program would then take into account when patching the LST file.

 

For example the origin on TundraDriftwood01 is halfway up the tree.  The tree LOD generation currently will give you no way of compensating for this (it ignores any kind of transform offsets in the _lod_flat file).   If instead there was UI for each individual tree texture, you could have something where you click where you want the origin/offset to be.  It would use the bounds of the texture for the atlas generation (for optimum packing) but use the offset data when generating the LST/BTTs.

 

 

---

 

While writing I think I realized specifically why they "messed up" the origin on TundraDriftwood01.  I'm guessing they originally meant for it to never stand upright, hence the name.  The origin being halfway up the tree would make it easier to orient assuming you want it to be laying down on the ground, since it'd rotate from the middle of the tree.  They then started using TundraDriftwood01 everywhere like a normal Reach tree and not necessarily just as driftwood.

[Sheson]

I have code ready for dynamic creation of atlases (and decomposing existing ones into separate sprites), so will probably add TES5TreeLODGen in near future that will make CK obsolete for generating lod for trees. I'm now thinking on the format of source tree billboards, their names and location. Something like plugin\formid.png (or bmp, tga, dds).

textures\terrain\lodgen\Skyrim.esm\0001306D.png

 

If I can take width and height in game units from object bounds field in TREE records, then I won't need anything else. If not, then probably get them from existing LST files and store in a text file with the same name. I really don't want to read tree nif models for that.

 

 

Woah

 

The flat trees are not in the master files. I guess CK just looks for the filename when creating tree LOD. You would have to create a mod with them and recalc bounds.

 

The LST file gives no clue what source tree the data belongs to.

 

Providing the dimensions is one time job, just like the textures. Use an ini file /  options window where mod authors could add their data?

 

A "simple" TES5Edit script nif parser to calculate the objects bounds would be a nice addition to TES5Edit, hint hint

 

 

 

 

Making each tree as an object would not be a big drop in quality, but in performance.  A massive one.  The trees are most certainly batched in some way to reduce draw calls.  Each tree that is treated as its own object will not be batched.

 

I tested this. A modern graphics card with a few GBs  and decent CPU are not the bottleneck for huge and busy object LOD bto. It generated BTO files well over 20MB in size. Due to the fact  I created a mixture of flat trunks and tree branches. No point in just using the same flat trees is there

The objects lod data all goes into main memory and requires my mem patch to be used to the brim at highest settings the game would start with. There won't be much room for mods anymore. It may be interesting for screen archers. There will be a (hidden) option for this in the TES5Edit script that I plan to release together with the LOD object resources.

 

For better understanding: I have created lots of new LOD meshes for LODGen. I wrote a script that among a few other things updates the base elements adding the new LOD data.

 

You are correct about the rotation and the angle filter and the origin points for trees. Using a mixture of tree lod and object lod for the trees is the way to go if you after the "perfect" LOD. It works well, imho. So far it would have required using CK for generating tree LOD for a users load order. A step I do not intent enthusiast modders to do. Once there is TES5TreeLODGen it will be easy to add the logic and resources to what I already have

[zilav]

I think there is misunderstanding of what I try to achieve.

I want TES5LODGen to generate LST, BTT files and atlas DDS file dynamically based on trees data in plugin and a separate billboard textures for each tree (like in Oblivion). Atlas will be created dynamically so I don't need any UVs or flat lod files, I will be creating all UVs myself. It is like objects lod in CK where atlas and UVs are generated dynamically based on lod meshes included in final lod.

To create a full lod for trees, I need:

1) Position and scale, have it in REFR records

2) Width and height of a tree lod in game units, either read them for some config files or get from Object Bounds field in TREE records. If object bounds don't match real size or you think it is better to make this configurable, I'll read it from somewhere.

3) UVs, will create myself.

 

If some trees are half burried or have other pivot shifts, all you need is to add an empty space in billboard texture.

Also there is something strange with dead reach trees. Tamriel.LST has 34 entries iirc, and when I split vanilla atlas into separate textures, turned out that only pines and aspens are used.

[zilav]

The LST file gives no clue what source tree the data belongs to.

I can read LST, then scan all BTTs to get any REFR FormIDs for LST index, then get a base TREE object for it and associate it with lod index in LST. So I can split existing atlases into separate billboards and give them proper TREE FormID name. That's why I want to store billboards using a combination of plugin name and FormID like Skyrim does with tints or voices. That way trees lod generation and how it will look can be managed by adjusting packages with billboard textures in any mod manager, the last installed will win.

[Jon]

I tested this. A modern graphics card with a few GBs  and decent CPU are not the bottleneck for huge and busy object LOD bto. It generated BTO files well over 20MB in size. Due to the fact  I created a mixture of flat trunks and tree branches. No point in just using the same flat trees is there

 

I took benchmarks using the built in tree LOD system and I was surprised at how intensive just the tree LOD system actually is.   I can't locate the exact figures right now, but if I took off VSync, the performance impact depending on the number of billboards could be quite severe, in terms of % drop.  But you would never know the difference with VSync on.  However, if you're running a system that can't actually keep the game locked at 60FPS, I'm not sure what % impact it would have.

 

I believe it was something like ~240FPS with the default vanilla tree billboards in my reference frame (a view of the swamps).  I added the Reach trees via the tree LOD system and it was about ~230FPS.   I then added a couple dozen other things using the tree LOD system (bushes, shrubs) and the FPS went all the way down to ~200.

 

Vanilla

Vanilla + Reach

Vanilla + Reach + Shrubs

 

In this particular case the addition of the shrubs and such is not actually worth the FPS impact, percentage wise.  Overall it does make quite a worthwhile addition, but that's for another topic.

 

Anyway my point is that even the highly optimized tree LOD system is not "free".  So I can't imagine the normal LOD system being very feasible for tree LOD.  Again, except for the trees that need to be dynamic or laying on the ground, etc.  I had attempted to compare my tree LOD swamp to one generated by your system to actually see the performance difference,  but I couldn't actually get yours working after many hours of attempting (always reported the file as corrupt after going into game).

 

3) UVs, will create myself.

 

If some trees are half burried or have other pivot shifts, all you need is to add an empty space in billboard texture.

No, it's that they're floating.  You can't add "negative" space via some empty space in the atlas.  The billboard will start halfway up the tree no matter what.. adding more space will just offset it further.  You actually need to tell the texture to sink lower in this case which requires manual compensation for the origin.  Besides, adding a bunch of padding would necessitate including that in the atlas, which wouldn't exactly be optimal.  It'd be best to have the bounds + some padding (for mip mapping issues) + some configurable offsets independent of the billboard texture.

 

Re: UVs ...  I agree it's optimal to have the UVs perfectly match the exact bounds of the billboard texture, but realistically they are going to differ.   For example in my project with adding tons of bushes and stuff,  I specifically have a shrub texture which is not at a 1:1 XY ratio to the actual mesh.  I did this in order to make better use of some of my atlas space,  by squashing it horizontally and then using the UVs and the shape of the _lod_flat to stretch it back out to normal.   And for at least two bushes, I have them using the same UV space in the atlas, but the _lod_flat NIFs are non uniformly scaled relative to one another.

 

So the _lod_flat shape and the UV shape very much need to remain separate.

[zilav]

I won't be using _lod_flat files and hence premade atlas, that's the whole point of creating a custom lodgen system to get rid of the awful Bethesda one.
By stretching lod_flat file you are actually adjusting size values stored in LST, right? In that case this can be configured separately. Shift values can also be configured if needed.
I don't know much about mipmapping and alignment, are there any rules for sprites positions? Right now all sprites are sorted by size and stacked starting from the top left corner, so there is no alingment at all. Here is an example of dynamically generated atlas from vanilla lods, with 4 pixes padding between sprites.

http://s217773845.onlinehome.us/files/atlas.dds
 

I can also calculate CRC values of sprites when adding to atlas, and reuse UVs for already existing ones.

[Sheson]

I can read LST, then scan all BTTs to get any REFR FormIDs for LST index, then get a base TREE object for it and associate it with lod index in LST. So I can split existing atlases into separate billboards and give them proper TREE FormID name. That's why I want to store billboards using a combination of plugin name and FormID like Skyrim does with tints or voices. That way trees lod generation and how it will look can be managed by adjusting packages with billboard textures in any mod manager, the last installed will win.

 

Oh that is tricky But you won't be able to do this for new trees added by mods that don't have CK tree load generated. So a manual way of supplying textures and data would be needed either way I think. 

 

There should a "simple" way to add resources like http://www.nexusmods.com/skyrim/mods/38651/

If two different mods each add their own TREE record for the same tree the only common ground is the full model nif file name. So, I believe the best way would be that the data for TreeLODGen should come from the resource not mods using it, right?

 

 

I took benchmarks using the built in tree LOD system and I was surprised at how intensive just the tree LOD system actually is.   I can't locate the exact figures right now, but if I took off VSync, the performance impact depending on the number of billboards could be quite severe, in terms of % drop.  But you would never know the difference with VSync on.  However, if you're running a system that can't actually keep the game locked at 60FPS, I'm not sure what % impact it would have.

 

Theoretically I would expect performance impact GPU wise to be no big difference if the flat trees are done by tree lod or object lod if the amount of flat trees / draw distance is the same. That is academical though For me it is clear to use both systems as much as one can for their intended tasks, because the tree lod system using less main memory is most important of all.

 

 

Actually, the most important visual flaw of doing trees as object LOD is the time it takes for LOD switching. LOD trees are replaced first and fast by full tree models when a cell attaches. Object LOD seems to be later in the order and waits for almost all objects to load before the LOD is turned off. This can cause quite a few seconds of flicker while both models are active. No idea if this timing can be influenced or not.

 

 

All that aside. It should be clear that LODGen, a future TreeLODGen and adding more LOD objects in general and whatever else is intended for high end PCs. It is technical 32bit engine limitations that holding us back, not hardware power.

[Jon]

By stretching lod_flat file you are actually adjusting size values stored in LST, right? In that case this can be configured separately. Shift values can also be configured if needed.

Right.  There are two shapes.  One that represents the size/scale of the tree mesh (the _lod_flat vertices), and one that maps that shape to the texture (the UVs).

 

Theoretically I would expect performance impact GPU wise to be no big difference if the flat trees are done by tree lod or object lod if the amount of flat trees / draw distance is the same. That is academical though For me it is clear to use both systems as much as one can for their intended tasks, because the tree lod system using less main memory is most important of all.

 

I've not been talking about GPU whatsoever.  Draw calls are a CPU issue.  The tree LOD system is optimized to minimize draw calls.  Adding more stress to the CPU will still lower the FPS, and again my only point in all of this is that the Morthal swamps and the Reach shouldn't need your object LOD system, and they should be using the tree LOD system,  because without a doubt the object LOD system is going to be worse for performance than the tree LOD system.   If you have several thousand Reach trees in the game, all covered with Object LOD instead of Tree LOD, this is going to be more taxing.  I would do the benchmarks myself to show you, if I could get your system working.. I believe ENB exposes the draw calls for the scene in its UI and I'd easily be able to compare the performance difference between Object LOD and Tree LOD.

 

The only trees that need to be covered by Object LOD are again the ones that are laying on the ground or that have enable parents.  So my suggestion to you is a hybrid approach.  Tree LOD for all the Reach and Driftwood trees that are close to upright.  Object LOD for the ones the Tree LOD system cannot cover.

 

I do agree that there is (theoretically) minimal GPU difference between Tree/Object LOD, if they were still all atlased to one texture and still only represented with two quads.

 

If I were to use Object LOD for the Reach trees, I'd probably just go ahead and make them slightly more detailed...  The trees are highly asymmetrical, and two quads in an X-shape can lead to a really weird mirrored/kaleidoscope effect from certain rotations.   The less symmetrical trees would ideally have 1 texture from the side and 1 texture from the front, and the two quads would be mapped to the different sides.  That is one awful limitation of the Tree LOD system..  You are given the ability to UV map the two quads in the _lod_flat NIFs differently,  but this doesn't actually work in game.  You end up with each UV area mirrored in an X-shape with itself.  So if you have UV quad A and UV quad B, and the _lod_flat is an X-shape of A-B... the billboards in game actually look like two trees,  one an X-Shape of A-A and two an X-Shape of B-B. **

 

** This is if you have the two sides adjacent to each other in the atlas.  As you showed with your LST decoding it's actually just taking the min/max of the UV vertices on both sides of the quad.  A pretty horrible limitation if you ask me. I'm pretty sure Bethesda could have figured out how to allow for different UVs on each quad.

[Sheson]

 

The only trees that need to be covered by Object LOD are again the ones that are laying on the ground or that have enable parents.  So my suggestion to you is a hybrid approach.  Tree LOD for all the Reach and Driftwood trees that are close to upright.  Object LOD for the ones the Tree LOD system cannot cover.

 

 

I agree This will be really easy to do when there is a TreeLODGen. It just needs to filter by the angles rules you posted earlier.

 

You will have to wait for my new beta that uses LODGen/bto for all these static, non animated non changing objects instead of doing them in papyrus

[zilav]

Trees TES5LODGen
http://www.afkmods.com/index.php?/topic/4051-wipz-tes5lodgen-for-trees-lod/

Better to discuss it in a separate thread since it doesn't have anything in common with objects lod.

[Sheson]

Hey, I have a couple spots where the MultiBound settings seems to be off for some reason. The LOD disappears a cell too early before the full model is loaded. See attached screenshot.

 

Just generate LOD as normal with LODGen to test. This is Tamriel.4.32.8.bto

 

I have no idea why or how to fix atm.

[zilav]

Yes, this is a known bug.

[Sheson]

You going to chuckle at this one:

 

Turns out LODGen exe only combines nifs into the same BSMultiBoundNode if the textures filenames are identical, meaning lower/upper case matters.

 

Not really a bug, just thought I mention it

 

Obviously in my own endeavors I'll simple make sure all paths and filenames to identical textures are exactly the same ...

[zilav]

You going to chuckle at this one:

 

Turns out LODGen exe only combines nifs into the same BSMultiBoundNode if the textures filenames are identical, meaning lower/upper case matters.

 

Not really a bug, just thought I mention it

 

Obviously in my own endeavors I'll simple make sure all paths and filenames to identical textures are exactly the same ...

Interesting

Well, I tried to contact with the author recently, he is aware of lodgen bugs but too busy with other projects to fix them right now, so we can only wait and hope.

[Sheson]

I released a beta of my project over at http://forum.step-project.com/topic/5011-beta-dynamic-distant-objects-lod-dyndolod-0425/

 

There is also pretty videos to see what it does

[zilav]

You should warn that people with graphics cards lower than GTX770 or similar better not use it, otherwise ILS and CTDs are highly increased.

I thought about making atlas for lod objects similar to CK before calling lodgen (to get the same performance), and again have code ready for that, but that will be some big time investment into programming and debugging, and I don't want to dedicate that time until lodgen is fixed. At least disappearing multibound issue which is the most serious one.

[Sheson]

All existing LOD files were updated and all new LOD files I created both use atlas textures

Thy are precomputed so to speak.

[zilav]

All existing LOD files were updated and all new LOD files I created both use atlas textures

Thy are precomputed so to speak.

You reUVed all lod models to atlases manually? Damn, I can imagine that was hell of a job, I'm speachless

By the way I noticed new lod models for trees which are combination of billboards and low-poly models, do they look better in game from your point of view? Or flickering when close is too much to compensate better visuals?

[Sheson]

You reUVed all lod models to atlases manually? Damn, I can imagine that was hell of a job, I'm speachless

By the way I noticed new lod models for trees which are combination of billboards and low-poly models, do they look better in game from your point of view? Or flickering when close is too much to compensate better visuals?

 

I wrote a niftoaster spell to scale and move the UVs and change texture. It is not *that* much of work  

Since the dynamic LOD switcher uses LOD models directly in the game the pre-done atlas benefits twice and was kind of needed anyways.

 

Regarding those hybrid trees, yes they do look better medium range, but they can flicker quite a lot when being switched, pines more than others. For me, the reach/tundra trees benefit the most of it, because of lighting on the branches.

I am sure most people won't even notice the extra branches in the distance, unless you constantly looking

 

If you put an empty .lst file and run process from scratch with no cache files you can have it do all trees. That is only for screenshots maybe. Make sure to increase default heap memory ...

 

I guess you can spend quite some time trying different settings and meshes to see what is possible and works for you...

[Deathbydestiny]

Does this handle the map lod too? If I recall it uses a similar system as the world

[zilav]

Map lod consists of landscape lod (*.btr files generated by OScape) and level 16 objects lod (Worldspace.16.x.x.bto files generated by TES5LODGen). So yes, TES5LODGen handles map partially, basically it is the mountaines meshes and big buildings you see on the map.

[Deathbydestiny]

Sheson, I'm using the latest version of your dyndolod script, it's pooping out whenever it gets to the point of generating lod for any dawnguard worldspace. Tes5edit just throws an exception with the formid of the worldspace

 

Edit: I found the problem, it for some reason didn't add dawnguard as a master

Edit 2: I added it as a master but when it gets to using lodgen, it crashes that too

[Sheson]

Wromg gorum

 

Sheson, I'm using the latest version of your dyndolod script, it's pooping out whenever it gets to the point of generating lod for any dawnguard worldspace. Tes5edit just throws an exception with the formid of the worldspace

 

Edit: I found the problem, it for some reason didn't add dawnguard as a master

Edit 2: I added it as a master but when it gets to using lodgen, it crashes that too

 

Wrong forum Please post over at http://forum.step-project.com/topic/5011-beta-dynamic-distant-objects-lod-dyndolod-0426/

 

I uploaded a new version 0.42.7 to fix the missing addmaster.

The LODGen.exe. crashes are a new issue. Uncheck the 'remove unseen faces' for each LOD level and see which one works.

 

 

 

@Zilav

LODGen.exe crashes when creating LOD with LODGen.pas for DLC01FalmerValley and setting 'remove unseen face' with the terrain LOD unpacked. It only happens in LOD level 8. 

 

If I had to hazard a guess it probably has to do with the y coordinates origin not being 0 ... dlc01falmervalley.4.0.-1.btr ... and because of this LODGen exe looking for a btr at the edges that doesn't exist