Jump to content

Recommended Posts

Due to Bethesda forums going to a different forum ( Bethesda.net ), reproducing a very useful topic here on AFK Mods just in case all old topics fall off the face of the internet as Bethesda move on, and the old locked forums are no longer maintained.
 
Original research and credit to Martigen ( and all participants in the linked topic who helped ), I am just copying the info to preserve it on AFK Mods
 
---------------------------------------------------------------------------
 
 
 
[!] If you've released a texture mod, or plan to, please read

 

 

Martigen wrote :
 
I've noticed something about some of the texture mods uploaded to Nexus. This post is to draw attention to good texture mod packaging practices, and generate discussion on techniques artists can use to create such mods.

Disclaimer: I'm not actually a texture artist myself, though I have some limited experience manipulating textures (re: MMM), so those more knowledgeable than me please feel free to chime in!


The Problem
```````````````

There are so many texture mods of all makes and models that have been released so far, some of them fantastic and very professionally made. However I noticed, delving into the files, that some of them have textures saved in the wrong format, while others often forget mipmaps. And given Skyrim is already more demanding memory wise than its predecessors, it's more important than ever to keep an eye on memory usage when using texture mods. Unfortunately texture mods, especially large ones, if saved using the wrong format can waste a lot of valuable video memory real estate.

When video memory reaches its limit, textures are streamed from main memory causing a huge performance loss. There may be users playing now with texture mods that don't realise the sudden FPS jumps and stuttering they're experiencing are due to taxed video memory on their cards.

But it's not just performance that takes a hit. Visual fidelity can be sacrificed in order to stay within the limits of video memory. Users are told to run at a lower resolution, or turn down their AA, to recover performance -- but if the problem is a poorly packaged texture mod pushing you past your card's limit, then there may not be a need to do this at all. And if you're an author, keep in mind that if your mod is too demanding people may elect not to use it -- and if you made it to be shared and used, you're losing out from having your work bringing joy to the denizens of Skyrim (read: everyone!)


The Example
```````````````

How important is the correct format? By way of example I'm running a good 30 odd texture mods -- some are large replacers (landscape, NPCs, clothing, armour, weapons for example) while others focus on particular areas (objects, architecture etc). I've selectively chosen them to keep within my video card's memory limit -- 1536MB (Nvidia GTX 580).

Now there are a lot of amazing texture mods on Nexus, but a fair few I've seen so far don't always use the correct texture formats. A high-profile and popular mod (and I don't mean to single you out Chris!) that's done this is Chris2102's Whiterun HQ Texture Pack.

By default this weighs in at 226M zipped, 487M uncompressed. However for whatever reason, while Chris has saved his textures containing an alpha layer and normalmaps correctly as DXT5, there's a large number of textures without alpha also saved as DXT5 when they can be saved as DXT1 for a significant memory saving. Re-compressing Chris's mod using DTX1 where needed gives us:
  • Default -- 487M
  • Recompressed -- 362M
A saving of 125M -- and remember no image quality is lost. DXT5 and DXT1 look the same, DXT1 just doesn't store alpha. If we presume the majority of these textures are loaded while walking around Whiterun, and if you're already at your video card's memory limit, not only would this prevent stuttering caused by taxed video memory, but it would also leave enough video memory to go from using no AA at all to 8xAA, or enabling SSAO, or using a mod like ENB. Or simply having more video memory to be able to use more texture mods!

And that's just with one mod.

Two other examples I've come across inlcude HQ Towns and Villages where recompressing DXT5 textures without alpha to DXT1 netted a 17M reduction in size (which, given all these textures are used at once, is a saving of 17M in-game) and the Blunt Weapon Enhancement Pack which goes from 90M to a tiny 12M, saving 78M -- here, you might not have all the blunt weapons loaded at once but each one gets a saving of around 3M, so if four different blunt weapons are loaded in a cell that's 12M saved. And, of course, the savings from each of mod add up.

Now before you get too excited, some of the major overhauls -- including the gigantic Skyrim HD and Vurt's Skyrim Flora are both compressed correctly, and there's no savings to be made. But I have, at least in my list, about 18 texture mods -- of just those that I've chosen to download -- that I found I could recompress to gain valuable video memory back.

As an aside, these are all factors that affect video memory usage in addition to textures being loaded:
  • Resolution
  • Anti-aliasing
  • Transparency anti-aliasing
  • Triple buffering
  • SSAO
  • Skyrim: uGrids and extended range tweaks
  • Post-processing mods like ENB
Look at this list as a trade off to what you can and can't have if your video memory is approaching its limit -- let alone the use of more texture mods as well. Thus, there's tremendous benefit for texture mod authors to correctly compress and package their mods -- it not only allow users to use your mod in the first place, but as an author the smaller file sizes means it's quicker to upload too. Win!


The Solution
```````````````

So if you're a texture modder, what are the guidelines to follow?
  • Remember to generate mipmaps (these help reduce GPU load)
  • Save textures as DXT1
  • Save textures with alphas as DXT3 or DXT5
  • Save normalmaps as DXT5
And
  • If you've uploaded a texture mod already, and you're not sure about the formats you've used, please check it and re-upload the mod if necessary.

Deathb0rn has conveniently linked a table of common texture volumes with the resulting file size after compression at various DXT levels -- this way you can usually tell what compression a texture has by its size.

 

gallery_1652_58_32189.jpg

 

The only thing to add to this is that when a texture is lacking mipmaps, it's usually 33% smaller than the sizes you see in the table.

Again my forays into texture creation are limited, so any input from the experts is welcome here -- for example what about bump maps? Specular? Can normal maps be compressed as DXT1 if they have no alpha? Please share your expertise!

Incidentally, I've been using AMD's The Compressonator to examine textures, generate mipmaps, compare textures and of course recompress them. It's a brilliantly easy tool to use, and its image comparison feature especially is fantastic (you can see at a glance what impact different compression levels will have). While I can't confirm this, from what I've read it apparently also produces better quality compressed textures than Nvidia's DDS tools, but I'll leave that for the experts to debate.


The Solution Ver 2 (kinda)
``````````````````````````````
Ethatron has released his DDSOpt tool which does a great job at picking up and fixing (by re-compressing a texture or a normal map correctly) some of the issues discussed above. It also has an excellent mipmap generator that adds higher levels of details in the mipmaps, leading to even better texture quality at mid to long ranges.

So -- for users and especially texture authors -- you can run DDSOpt to optimise textures to be correctly saved in the right format, and thereby reduce VRAM usage. It is still in development however, so if encounter any odd results please post in the DDSOpt topic on the STEP Forum here

 

Alt3rn1ty Edit : Changed the link above to the STEP forum due to Bethesda forum soon to close, and Ethatron has been most active on the new topic linked : Also since Ethatrons own website and documentation has been down for more than a year, the best practices and most recent notes are at the STEP forum and WIKI guide. ( Original topic and more links at the end of this post )

 

 

-----> However, there is a large caveat <-----

DDSopt can correctly re-compress textures and can sometimes pick up minor issues such as unnecessary alpha channels, but it is not a silver bullet to magically make all your textures perfect. It cannot fix errors in textures and alpha channels created (however inadventently) by the texture author. Things like left-over borders, stray pixels in alpha channels and a whole bunch of stuff that actual texture artists could probably mention that I can't.

In other words, if you make and upload texture mods DDSOpt can help you ensure your textures are properly saved with the most optimal DXT compression format, but it cannot fix problems with textures, alpha channels and normal map generation. This is up to you to get right first time before you upload the mod -- and unfortunately, while testing DDSOpt with Ethatron, there are a lot of mods that have these sorts of issues and as a result DDSOpt can't optimise them. So if you're a texture author, please check your work smile.png
 

 

Thankyou for reading.

isoku chimes in:

'isoku', on 08 Jan 2012 - 7:16 PM, said:snapback.png

Normal maps are treated the same as everything else. Depending on the alpha, save as DXT3 or DXT5. No alpha = save as DXT1. DXT1 - 1 bit alpha is also a choice if you have an alpha channel with only black and white (aka no grays).

There is also another problem I've seen where textures that shouldn't have an alpha channel have an empty white one saved with it further bloating file size. Solution: delete the alpha channel and save as DXT1.

 

MadCat221 adds:

'MadCat221', on 08 Jan 2012 - 7:34 PM, said:snapback.png

Another thing to add.... Uncompressed textures.

It's been determined in the Oblivion period that DXT compression does Bad Things™ to normal maps. Channels aren't compressed evenly, DXT compression artifacting causes oddities when the normal map is rendered.

To get a similar filesize savings, instead, reduce the dimensions to 1/4 its size (or half squared basically, 1024x1024 to 512x512) and save as ARGB8 Unsigned. It's uncompressed, and thus has no DXT artifacting, and the size reduction far more evenly distributes the reduction artifacting for a better looking normal map.

...

Throttlekitty released some nVidia Texture Tools DDS Photoshop Plugin profiles with improved mipmap settings back in '07 for Oblivion modding. They're still applicable for both FO3/FONV and Skyrim textures. Check it out here: http://www.nexusmods.com/oblivion/mods/11817/?

 

And as a final note.... Do not upsize the texture unless you're actually adding detail to it. Larger dimensions by themselves != more detailed textures. It cannot undo size reduction detail loss.

 

teanandcigarettes says:

'teaandcigarettes', on 08 Jan 2012 - 9:14 PM, said:snapback.png

Mip maps are necessary for almost every texture. Removing mipmaps from normal maps would have a horrible effect on the clarity of the image. Not using mipmaps would result in a "shimmering" effect as seen on the image here: http://www.shinvisio...mapping-egz.png

If mipmaps are removed, the pixels on the left example would be constantly moving. This would be very noticeable on normal maps; from what I've seen normal maps included in most texture packs are done rather badly and are incredibly noisy.

...

Unfortunately, creating texture replacers for objects that had their normal maps baked from highpoly models is pretty much pointless. Their diffuse textures can be tweaked, but modyfing their normal maps in Photoshop will often produce technically incorrect results. Objects like that rely on normal maps NOT to add surface detail, but to define the shading of a lowpoly object and bring it closer to its highpoly version. Upscaling normal maps and overlaying noisy detail on top of it can make it appear slightly sharper, but in most cases it will produce worse results than rendering the normal map at a higher resolution. Here's a massive article about normal mapping in current generation video games: http://wiki.polycount.com/wiki/Normal_map

 

The article also has links to tutorials that deal with the subject in more detail.

 

Ethatron adds:

'Ethatron', on 09 Jan 2012 - 02:43 AM, said:snapback.png

Oblivion did switch shaders based on the DXT-format and for Oblivion DXT1 means "no alpha", not "1-bit alpha", and it uses the shader which doesn't use the alpha-information. The exception is the UI, there you indeed can use "1-bit alpha", but not for assets used in the game itself. Also, the meaning of "no alpha" changes is you have a color-map or if you have a normal-map. For color-maps the alpha-channel may contain transparency, so neutral alpha is white. For normal-maps the alpha-channel can contain specularity, so neutral alpha is black. For color-maps the alpha-channel may also contain a heightfield, so neutral alpha is whichever solid grey (it's a difference-map, no differences, no effect, only waste of GPU-cycles).


Huleed gives the straight dope:

'Huleed', on 09 Jan 2012 - 06:37 AM, said:snapback.png

Normal maps should be saved at quarter resolution (eg, 1024x1024 -> 512x512) as 8.8.8.8 uncompressed. This results in the same file size, and avoids the DXT artifacts which can mess with normal maps.

The thing to keep in mind is that normal maps, essentially, aren't color. They're actually 3-dimensional vectors that are stored using color values, that special shaders read and convert back to 3D vectors to do directional lighting calculations with. For instance, say you have the vector:
{ -0.707, 0.0, +0.707 }
which points a bit towards the side. This gets converted to color as:
{ R: 90, G: 128, B: 218 }
Now, remember that DXT compression is lossy, which means the color that's read will be slightly different than what it originally was. So, for example, you may end up getting back something like:
{ R: 88, G: 125, B: 225 }
which is close, and other neighboring texels (texture pixels) may also be modified to hide the discrepancy. But, this is to hide the difference ofperceived color, and again, this is actually a 3D vector. So when you convert it back to a 3D vector, you get:
{ -0.69, -0.98, 0.765 }
Rather different from where it started, no? It's pointing off-kilter, which would cause the texel to be lit incorrectly. Add little bits of "noise" like this all across the face of the surface, and it's not hard to see this is suboptimal when dealing with non-color information.

If it's instead saved uncompressed, the color that's read back will be the same as it was originally stored with, which when converted back, is:
{ -0.706, 0.003, 0.710 }
Much closer to the original. The resolution of the normal map may be quartered, but it will provide a relatively smooth transition between neighboring texels that each better represent their original values, thus has fewer anomalies.

PS. There may be some inaccuracies in the above post, but the idea is there. In essence: DXT compression doesn't play well with normal maps. Using uncompressed formats is better, even if it means quartering the resolution.

 

And last but not least Throttlekitty brings news

Quality post! I needed a refresher, thanks for putting that all together.

 

Just so I have something to add, we have a fine wiki at Polycount with many nuggets of wisdom.

 

The latest fad in getting good normal maps is to bake in 16 bit then convert to 8bit, here's some info by the excellent Earthquake. Note this is only valid for baking info from a high poly mesh. The idea is to get rid of the "banding" that tends to appear with slight gradients by introducing dithering, while not disturbing the normal data too much. It does result in a slightly noisy result, so it's not always ideal for say "Clean Room Sci-Fi", but great for "Rusted Iron Sword".

 

 

Resources
````````````
The Compressonator -- Download here
The above tool has now been superceeded by AMD Compress

 

Oblivion DDS guide -- Read me!

Wikipedia on DXT modes -- Click me!

Generating normal maps (thanks to teaandcigarettes) -- Click here
More on DXT formats (thanks to teaandcigarettes) -- Read me!
Another good read on DXT formats (thanks to Phitt) -- Click me or the kitten gets it!

How to choose the right DXT compression (thanks to Thottlekitty) -- I am sexy and I know it
Normal map workflow (thanks to Thottlekitty) -- If you build it they will come
Throttlekitty's Keep Details profiles for Photoshop -- That's not a mipmap, this is a mipmap!

DDSopt by Ethatron - Original forum starter topic -- It's life Jim, but not as we know it 

DDSOpt topic on the STEP Forum here

DDSOpt Guide on the STEP WIKI here

DDSOpt can be downloaded from Nexus here. ( DDSOpt Pre-Release Update 4 is recommended )

 

Oh and the STEP Team have already done all the vanilla textures using the above DDSOpt guide :

From the Optional files here ..

http://www.nexusmods...kyrim/mods/11/?

.. Grab the file "STEP Optimized Vanilla Textures - Standard Version"

"All vanilla DDSopt-imized textures (no resolution caps, compression and mip-map optimized) not covered by STEP mods. Dawnguard, Hearthfires, and Dragonborn required".

 

Edited by Martigen, 15 January 2012 - 01:41 PM.

Edits by Alt3rn1ty started 08 Oct 2015 - Typos / A few more appropriate links / Old broken links fixed

Note : Rodgreen PS Action, website linked in old post for combining normal maps cannot be found

  • Like 2
Link to comment
Share on other sites

Tools for Texturing;

The best software to use if you can afford it ( or get it as a student ) is Adobe Photoshop

If you have Adobe Photoshop you will be needing the Intel Texture Works plugin

https://software.intel.com/en-us/articles/intel-texture-works-plugin

 

If you are working with Skyrim Special Edition textures - Recommend you use the Intel Texture Works plugin instead of NVidia plugins

( At time of writing there is not a version of this plugin for G.I.M.P .. So SSE Modders need Photoshop currently, or Paint.NET )

 

Reason : See this topic on Reddit

 

 

 

 
 

In short, SE can use textures with newer forms of DDS compression. Why is this note worthy? Some of you may or may not know that in original Skyrim compressed textures could either be DXT1, DXT3, DXT5, or 5.6.5. These are all problematic in some way because DXT1,3,5, reduce file size, but really destroy the quality on some textures especially normal maps. For comparison, a mod author would create a normal map that looks like this, but saving it with DXT compression would make it look like this. To combat this, some authors would just leave their textures uncompressed or use 5.6.5. This fixes the artifacting problem, but file sizes becomes a problem with even a 1k texture being 4mb rather than .6mb when compressed, this adds up fast when your game loads lots of textures.

 

So on to the new stuff. From my testing, SE will use textures saved with BC7 compression. The Intel Texture WorksPlugin will do this for Photoshop users. GIMP and Paint.net I haven't found any alternatives for you but they might be out there. BC7 offers very nearly the same quality as uncompressed while still being the same size as DXT5 compressed textures. It does this so well, I can say that in SE, there's almost no reason to use uncompressed textures when BC7 can be applied. This is pretty cool. Those textures we 'must not compress for fear of blockiness' can now be compressed without fear of loss in quality. Here's that same normal map from above with BC7 now.. (Compare it with the first one I linked, they're very nearly identical.)

 

Here's another example using body normal maps from Mature Skin which up until now should have always been saved uncompressed for the best look. Here's femalehead_msn.dds uncompressed. With DXT1. With BC7. I also put together an animated .mp4 to show a quick side by side comparison. And Since imgur compresses the crap out of things, here's an original tga download with the screenshots.

 

Another way to visualize the difference between our uncompressed and compressed textures is the 'difference' blend mode. If you don't understand or know what this is, it's simply just a way of showing which pixels are different between two images. (Pretty straight forward). Black means theres no different between the two images and colored pixels represent a different. The images are probably explanation enough though. So we'll take the uncompressed femalehead_msn place it on the bottom layer and then take the DXT and place it on the top layer and switch its blend mode to difference. This is the result. As you can see all the non-black pixels are what's difference between our uncompressed and DXT compressed texture. It's not too bad I guess, but there's quite a lot. This is the result for BC7. Far less colored pixels which is good because it shows us that very little has changed going from uncompressed to BC7.

 

So where does this leave us? Well going forward, I'd recommend that all mod authors use BC7 when they can for SE textures. Normal maps especially since the artifacts are very apparent with DXT compression. For diffuse maps without alpha DXT1 is still 'fine' in most cases, it still yields a smaller file size than BC7 and it's okay to use unless certain patterns or designs fail to look good using this method. For Diffuse maps with alpha layers, there's no reason not to use BC7 since it gives the same file size as DXT5 at a higher quality.

 

I hope this was helpful and my droning on about texture quality didn't bore people to sleep. I listed some general FAQs people might have about this new finding.

 
  •  

    Q Wait, does this mean I should go out re-save everything using BC7 compression?


     
  •  

    A Well, probably not. If you have a texture that's already compressed, saving it with BC7 won't take away the artifacts that are already there. You'd need the original uncompressed textures and then you can re-save them as BC7 to see any improvements. It's up to mod authors to re-save their stuff with BC7. But like I said most diffuse maps are okay with DXT1/DXT5 compression. It's mostly normal maps that would benefit from BC7.


     

You should also consider the fact that if you've never noticed compression artifacts, you probably won't see any benefits to using BC7 to begin with.

 
  •  

    Q: 'BC'?


     
  •  

    A: Block Compression. it's in reference to the method in which the algorithm compresses the images. Read more about it here.


     
  •  

    Q: What's with 'BC7' not DXT7?


     
  •  

    A: Well technically 'DXT' naming for compression was dropped a long time ago, but Skyrim being 'old school' it's still a useful term for us. But technically DXT1 is BC1, DXT5 is BC3. DXT3 just sorta got dropped, because it's practically useless anyway.


     
  •  

    Q: What about the other newer forms of BC compression?


     
  •  

    A: I haven't tested them extensively, but some just can't be used because of the way they save the information to the RGBA channels. BC6H i've ruled out already, but that's mostly for cubemaps or HDR images. BC5 might be usable for normal maps, but it yields the same file size as BC7 anyway, so i'm not sure of the point. I haven't tested it in game however. BC4 Might be able to be used for saving parallax maps, if parallax ever makes it to SE.


     
  •  

    Q: What about fast vs. fine compression in the Intel Texture works plugin?


     
  •  

    A: The fast preset just uses their fast compression method when saving the texture. Which is much quicker than the fine preset. However, I don't really notice a huge different in quality between the two presets. If you don't mind waiting an extra 3 or 4 seconds then just use the fine preset if you're in that big of a rush, then the fast preset will do.


     
  •  

    Q: So textures work the same as they do in Fallout 4 then?


     
  •  

    A: Honestly I have no clue about FO4, I know then engine can load BC7 and BC5 textures and that's about all. The game hasn't held very much interest for me so maybe someone with more expertise can give a comparison.


     

I'd like to give a shout out to /u/zilav for originally bring up the idea. It would have never occurred to me on my own. Also to /u/eejoseph for finding the comment by zilav and sharing it with me.

 

TL;DR : Skyrim SE now supports BC7 texture compression. This is far superior to the older styles of texture compression original Skyrim uses. BC7 is so good, it can be used to save textures we previously had to leave uncompressed while keeping the file size the same as the old textures for Skyrim but maintaining nearly perfect quality compared to the uncompressed textures.

 

Edit: I've been told that BC5 can also be used for specular and normal maps and is preferable over BC7 in terms of quality while still being the same file size. I haven't tested this myself, just passing along what I hear.

 

Edit2: Tested BC5 on normal maps in game. Would not recommend it. It makes objects appear much darker than they should be. Likely due to the fact that BC5 makes the blue channel completely black.

 
 

 

 

There is an Open Source alternative for those who do not have Photoshop ...

 

G.I.M.P ( Gnu Image Manipulation Program ) 2.10.12 - http://www.gimp.org/

GIMP now has its own Built in plugin for DDS textures, but it does not yet support the newer DirectXTex DDS formats (BC6h / BC7)

 

 

The following is not as advanced as Photoshop or G.I.M.P, but is very capable of producing superb graphics, and can load / save all dds formats

 

Paint.NET ( Now supports all DirectXTex variations of dds textures ) - https://www.getpaint.net/

Variety of plugins pinned on the paint.net forums - http://forums.getpai...ublishing-only/

 

 

Best used as an intermediary tool for some jobs and saving out the work in lossless png format, then you can load the work up in GIMP or PS afterwards without any loss to quality, before eventually saving out to a lossy dds texture.

 

Windows Snipping Toolhttp://windows.microsoft.com/en-gb/windows/use-snipping-tool-capture-screen-shots#1TC=windows-8

I dont know how well known this is now, but Microsoft snook this into Windows in Win Vista onwards.

It was included with no real fuss, and not obvious unless you went digging down into the Start menu, All Programs menus.

 

Its a screen grabber for windows taking snapshots of areas you encompass with your mouse by Drag n Dropping a rectangle around the area, really useful for making tutorials and screens of other tools in use, or even games if they are in window mode and you have some kind of strange keyboard setup which prevents PrtScn working.

 

Also as noted at the link - Can be used to capture menus while they are open.

 

Worth looking into if you have never used it before, it comes in handy quite often for me.

 

------------------

 

IrfanViewhttp://www.irfanview.com/, ( with the plugins installed ), allows you to view dds files (currently not all DirectXTex formats) quickly without loading up a full blown paint package.
 
 
You can right click a dds and choose 'Open with' from windows context menus, and once opened, go to the image menu, and choose information ..
  
 
Gives information about the format the dds file was saved in.
 
 
A tip for doing those close up crop images en-masse ..
 
 
 
If you have a bunch of screenshots taken in game so you end up with a heap of screenshots, but only want the cropped part 
 
 

 

 
gallery_1652_58_5281.png
 
 
 
 
 
Use say Paint.NET to note the upper left coordinates of the box
 
Then also note as you drag a rectangle the lower right coordinates.
 
 
 
Then - IrfanView can perform a little magic. Put all your un-edited copies of the screenshots in a working folder
 
 
 
Now the games default .bmp format screenshots are ideal because IrfanView may have to perform converting the files to a new format ( say .png ) at the same time as batch cropping, otherwise the cropping will not be performed <- This is only if you do not set the advanced options to overwrite existing files ( and I think the default is not to overwrite ). Personally I prefer not to overwrite so that the original screenshots are preserved if anything is not quite right with the process, and the batch process then outputs separate files ...
 
 
 
In IrfanViews 'File' menu, choose 'Batch Conversion / rename'
 
 
 
In the batch dialogue, browse to your folder, and shift select all images, then choose 'Add'
 
 

 

 
gallery_1652_58_688.png
 
 
 
 
 
Then change the 'Output format' to a different format than the source images, tick 'Use Advanced', click the advanced button, tick 'Crop' and enter your upper left x and y pos, and the lower right width and height coordinates, then click 'Okay', and lastly 'Start Batch'
 
 

 

 
gallery_1652_58_111905.png
 
 
 
 
 
You will end up with a folder full of the cropped images - I changed them all to .jpg before packing them all into the final thing to reduce their file size greatly
 
 

 

 
gallery_1652_58_101267.png
 
 
 
 
 
Sounds like a lot of steps, but if you have ever wanted to manually crop a load of images the same, doing them individually in G.I.M.P or Paint.NET can take ages, you can load all images into layers in those tools, and crop them all like a cookie cutter through all layers .. But IrfanView does the same trick on multiple files a lot quicker, because you dont have to manually setup a load of layers for each image

 

-------------------------

Someone on Nexus gave me a good tip :

RenderDoc designed to handle GPU formats, can be used as a Texture Viewer for dds

It gives the necessary info we need about what format a texture is

From the documentation https://renderdoc.org/docs/getting_started/tips_tricks.html

  1. RenderDoc can be used as an image viewer! If you drag in or use file → open, you can open images in a variety of formats - .dds, .hdr, .exr, .bmp, .jpg, .png, .tga, .gif, .psd. The image will load up in RenderDoc’s texture viewer and you can use the normal controls to view it as if it were the only texture in a capture. Note that .dds files support all DXGI formats, compressed formats, arrays and mips - all of which will display as expected. If the file is modified, RenderDoc will reload it and display it. Note that changing the image’s dimensions or format will likely cause problems.

  2. ~snip~

  3. ~snip~

  4. Right clicking on one of the channel buttons in the texture viewer (R, G, B, A) will either select only that channel, or if it’s already the only one selected it will select all of the others. This is useful e.g. to toggle between viewing RGB and alpha, or for looking at individual channels in a packed texture or render target.

-------------------------

I noticed a couple of issues with RenderDoc, though its something that can be worked around.

Issue 1. If you dont have enough horizontal Screen space, and the file path for the loaded texture is long, the Format of the texture goes off the end of the available space to show it.

 

To solve issue 1, I decided I did not need the "Event Browser" box open all the time, so closed it.

 

Issue 2. Currently in RenderDoc v1.5, if you have closed the Event Browser, and try to load a texture into RenderDoc (by whichever method), RenderDoc crashes.

 

Issue 1 is going to be addressed in a future version, I contacted Baldur

Issue 2 is fixed in a private build last night, so will be in the next version whenever that occurs.

 

Meanwhile, Baldur gave me a tip - You can get around Issue 2 by dragging and docking the "Event Browser" as another tab alongside the Texture Viewer Tab etc. Which free's up more horizontal space to show the full path and Format at the bottom of the interface (well that is so long as the file path is not so long as to hide the format again).

 

Anyway, those minor issues aside ..

[/quote]

 

I very much like RenderDoc (even if I dont have much if any use for its full potential), aswell as being able to select all Channels including the Alpha individually for viewing, I like also that you can use a drop down menu to see how many MipMaps the texture has, and I also like that among the Formats displayed RenderDoc also includes un-compressed Formats like A8R8G8B8 .. Whereas IrfanView would just not show any format info for such a texture because it is not compressed, so distinguishing un-compressed formats is a breeze with RenderDoc.

 

Its a bit slower to load than Irfanview, but its a lot more comprehensive for dds support than IrfanView.

 

I think RenderDoc has just become my favourite viewer for investigating dds files :)

 
 

 

Link to comment
Share on other sites

Types of Textures in Skyrim :

 

  • in a "menu"-folder: User-interface element, alpha is element-mask
  • in a "landscapelod"- or "terrain"-folder: World-space normal-map for terrain, no alpha
  • "_n"-suffix: Tangent-space normal-map, alpha is specularity
  • "_msn"-suffix: Model-space normal-map, no alpha
  • "_g"-, "_glow"- or "_emit"-suffix: Glow-map, alpha is a mask
  • "_hh"-suffix: Gloss-map for hair, no alpha
  • "_hl"-suffix: Detail-map for hair, alpha is opacity
  • "_m"-suffix: Reflectivity-map for light-sources, no alpha
  • "_em"- or "_envmap"-suffix: Reflectivity-map for environment-maps, no alpha
  • "_e"-suffix: Environment-map (some are planar, some are cube-maps), no alpha
  • "_b"- or "_bl"-suffix: Backlight-map, no alpha
  • "_s"-suffix: Specularity-map for skins, no alpha
  • "_sk"-suffix: Tone-map for skins, no alpha
  • "_p"-suffix: Parallax-map, no alpha
  • "_d"-suffix: Diffuse-map, alpha is opacity
  • "_h"-suffix: Haze-map, alpha is unknown

All others are color-map, alpha is opacity or parallax

 

 

Thanks to Ethatron who originally collated this list while developing DDSOpt.

Link to comment
Share on other sites

Some Notes

 

Texture Sets

Skyrim brings a new trick to the table, with older games ( Fallout / Oblivion ), if you wanted an item to have a different colour, you would need a mesh with a path to a different texture with its unique colour.

Skyrim has done away with that, you no longer need multiple meshes and textures, now you just need different coloured textures for the same mesh : And a little work in the Creation Kit.

Hana has done a tutorial guide on the subject here - Skyrim Texture Sets

Note : Texture Sets have the potential to greatly increase VRAM use.

 

Compressed or Uncompressed, that is the question

As mentioned in the first post ( see also the quotes ), some normal maps are best saved as uncompressed ( 8.8.8 format ), even though they are less optimal :

 

Typically body mods Face normals. For anyone who remembers the vanilla Skyrim normal maps and not using Hi Res replacers, you will know well the awful looking blocky noses. These are artifacts introduced with compression.

 

I think there are some water mod textures also have artifacting which at a distance creates an unwanted effect, what I call the distant polarised matrix effect ( its not a technical term anyone else has used, in my mind it just describes what I think is going on there :), as you move slightly side to side as you walk forward, that distant shimmering effect you see with some textures installed and when its raining - Similar to what you can see in optical illusions with a layer of lines on transparent cell overlaying another cell with lines at a different angle .. into infinity ) on the surface of the water which I think is a result of compression artifacting on those layered textures.

Edit : The unwanted "distant polarised matrix" effect is called a Moire Pattern

 

Anyway, uncompressed normals for special cases can have the downside to them ( large size ) offset by reducing them to a quarter size as mentioned in the OP. But that may not be viable if it affects the look drastically of large textures at long distance, or in the case of faces where people like to get close to them and the objective is to have beautiful skin texture.

 

BUT : Dont take this idea to the extreme and decompress every texture in the game, compression is good for loading performance, and the efficiency of the game engines overall balance. The key is having good quality texture sources, and only saving the texture ONCE in dds format ( any work / edits in between should be saved as a lossless format image until ready to finalise the texture as a dds with mipmaps ).

 

People using for example DDSOpt to decompress all the games textures, you are making your game performance less optimal, see another guide in this forum reference BSAs and You - Compression and its advantages are also discussed there.

 

 

Texture size

Also mentioned in the first post, only make textures as large as they need to be. A body mod which has 4096 body textures does not need 4096 hand textures, a bit of proportion in texture sizing would be good, even though limited to quarter sizing the next step down to 2048 is still a bit big imho for hands or heads. Compare hands textures with vanilla armour gloves texture sizes for example.

As Nuska once said after she had acquired a huge screen and tablet to work on graphics professionally in great detail, increases in texture size on careful examination have diminishing returns the bigger they go, there is no need for any texture to be larger than 4096 in games, and a lot of the ones mod makers produce which are that size or above are just unnecessarily large for how they are employed within the game. Anything bigger than the original game texture is probably just a waste of VRAM, and adding to slowing down the execution of loading all resources.

 

 

Power of 2 Textures ( 16x16 / 32 / 64 / 128 / 256 / 512 / 1024 / 2048 / 4096 ) - Why ?

It relates to graphics card optimizations. They've been designed to process them in that way. Most cards these days will allow you to load textures with dimensions that are not powers of two, but it will likely have a negative impact on performance. It also depends on how old the graphics card is, best bet to ensure compatability across the board is to stick with textures sized in Powers of 2

 

You will notice when you generate mipmaps saving a dds texture, all the mipmap layers ( using either Adobe Photoshop or GIMP with associated plugins for DDS ) will also be in decreasing sizes in powers of two.

 

Non power of two textures when loaded need converting costing you load time and not saving any memory. Non-square textures are generally OK, as long as both their dimensions are powers of two.

 

Graphics hardware is optimized for matrix operations, fragment operations and vector operations. Simply put square matrices are easier to deal with, as calculations may be done in blocks ( called fragments ), hardware is optimized for block operations, which is why there are things like file buffers, the RAM blit does not blit to disk until a block has been populated. The same is true of graphics memory.
 
The frame buffer is composed of fragments which are square. For example in a screen with a resolution 800x600 and an RGB color space (0-255) there are 800x600 points with 3 bytes each channel there are a grand total of 3x800x600 = 1,440,000 bytes to address in the frame buffer. That means there are 1,875 addressable fragments that are 256x256x3 bytes. Because the texture data is square it makes it significantly easier to map from the GRAM matrix to the screen buffer matrix using bicubic scaling, where as if it wasn't square the bias for the longer side would take more time to calculate when it needed to be scaled.
 
Many graphics APIs will accept non square texture data, because they accept UV mapping coordinates as floating point data, however once it is sent to the GPU, padding is added to the texture data, because the actual proportions of the image do not change the mapping appears unaffected, however padding is added to the texture data, because the GPU likes addressing it as a perfect square.
 
So if a 100x1024 image is used, and image that is 1024x1024 is used which means 946,176 bytes are wasted. Even more so if compositing is to be done, because an alpha channel will need to be added in order to indicate that the padding data should not effect the composited texture.
 
See also KatsBits article on the subject :

And various comments on Stackexchange ( where some of the above explanation was sourced )
Why are textures always square powers of two, what if they arent

 

 

Well thats me done for this topic I think for now, please feel free to post any tips / knowledge which add to it.

 

Edit : Anyone has any useful knowledge not covered, feel free to give your post a bold title and blab away - Anyone expert on the subject of LODs for example would be a good addition.

Link to comment
Share on other sites

  • 3 weeks later...

Edits to OP :

 

Added more links reference DDSOpt in the Resources section

Added a link to the STEP Nexus page, which has an optional file including all Vanilla Skyrim Textures DDSOptimised in accordance with their own guide.

Link to comment
Share on other sites

  • 2 weeks later...

Quality post! I needed a refresher, thanks for putting that all together.

 

Just so I have something to add, we have a fine wiki at Polycount with man nuggest of wisdom.

 

The latest fad in getting good normal maps is to bake in 16 bit then convert to 8bit, here's some info by the excellent Earthquake. Note this is only valid for baking info from a high poly mesh. The idea is to get rid of the "banding" that tends to appear with slight gradients by introducing dithering, while not disturbing the normal data too much. It does result in a slightly noisy result, so it's not always ideal for say "Clean Room Sci-Fi", but great for "Rusted Iron Sword".

Link to comment
Share on other sites

:cool: Tk Tk is in da house, thank you will add your words of knowledge to one of the posts later ...  :whistle: I shouldnt be here right now

Link to comment
Share on other sites

  • 11 months later...

Hey alt3rn1ty, thanks for preserving this here.  I have a related question about the packages available online of optimized vanilla high res textures.  I have been using the Bethesda High-Res DLC Optimized textures by vano89.  But now, everyone seems to be recommending the newer Optimized Vanilla Textures by tony971 and the STEP team.  

 

Tony is crediting you with helping get his archive set up for Wrye Bash, so I was wondering if  you knew why tony is suggesting an installation script for Wrye Bash users to install a texture replacer (his instructions say to run the "install.vbs" even for Wrye Bash users).   I'd rather just install it through Wrye Bash and manage my installation order in the installers tab.  Is there any reason I can't just do that?

 

I was also wondering if you had any thoughts about tony's set of texture replacers vs the set I have been using by vano89. From what I can tell, tony's set completely replaces all of the official high-res DLC, while vano's set only replaces some of them.  Other than that, do you know if tony's method of optimizing the textures is better than what vano did?

Link to comment
Share on other sites

I have not used them personally

I think Tony's objective was to try and save a few plugin slots, but he also wanted his textures in BSAs to :

 

a ) Override vanilla textures, and be inside an efficiently packed BSA for fast loading

 

and

 

b ) Be overriden by mods BSAs with their own texture replacers, and by loose texture replacers

 

 

My method of achieving that was to use dummy plugins for Vanilla Reduced Textures, and have them load early in the Load Order

 

Tony wanted also to cut out the dummy plugins to save slots, but his problem with that was the average user could come unstuck at the editing the archivelist part of his method

 

I have never looked at Tony's finished thing in great detail, but I think thats what the vbs script is all about, doing the editing of the archivelist for users who may have got that part wrong, and possibly also disabling the High Res DLC plugins ...

 

If you have a machine capable of running the Bethesda Hi-Res texture DLCs, then use his High definition version ( and I believe you do not have to have the Hi Res DLC installed if you have that .. so even more plugins eradicated from the load order ).

 

This set of textures has had a lot more research into optimising them than VAN0's, it was the culmination of a lot of research throughout a few really long topics on the STEP forum, and followed a restructure of the STEP texture optimising guide, assisted by help from Ethatron on the best use of his DDSopt for every texture type concerned, and every special case textures were treated separately in batch groups.

 

So you end up with more optimal mipmaps ( DDSOpt excels at redoing texture mipmaps which are better than the originals in detail, aswell as being more optimal ) throughout all the games textures, the Hi Res dlcs textures also optimised, a few less plugins and your graphics card less bogged down with VRAM taken unnecessarily for no visual benefit.

 

Follow that up with USLEEP being installed and loading later than his BSAs ( because USLEEP is plugin loaded in its Load Order position, and archivelist BSAs load before them ), then you have all the USLEEP fixed textures in game aswell

 

 

IMHO he did a good job of it, I dont think he has Photoshop, and is not familiar with G.I.M.P. enough to make a fair few textures detail ever so slightly better .. But he did do his best to only try and process every texture once ( to minimise detail degradation saving to some DDS lossy formats, you dont want to be saving the same texture multiple times after various processes - Do all your changes in one pass and save the finished product once ) .. Thats just me being nit-picky though, what I noticed nobody else will, and especially not in game.

 

For the objective of optimising and replacing all the games textures including DLCs and Hi Res Texture DLCs, it is better than VAN0s

 

How you install it .. I have not ventured to try it out fully installed as per Tony's method.

 

 

Personally now I have a nice laptop, I dont use any optimised textures, just the Hi Res DLCs, USLEEP and the Unofficial High Res Patch, the game runs superb on this machine

 

I'm also using Beth INI and its Ultra settings

 

Edit : Very much looking forward to Skyrim Special Edition remastered, which has gone 64 bit

 

 

Edit 2 : Forgot to mention .. If you use STEP, then Z did his own couple of sets of optimised textures using a similar routine as Tony did, but did not include any textures already covered by the mods which STEP utilises - Have a look at the Optional files here http://www.nexusmods.com/skyrim/mods/11/?

So STEP users would probably be better off using those

Link to comment
Share on other sites

Hey, thanks for the info!  Makes sense.  That's a nice laptop you got there.  My laptop only has a 750M card so it probably won't be able to handle the high res textures too well, but my desktop can.  I too am looking forward to SSE.  A lot of people seem to be complaining that it doesn't include new content and that mods requiring SKSE will need for SKSE to get recompiled in 64 bit, but I am just happy that we are getting a 64 bit/dx11 engine.

Link to comment
Share on other sites

  • 1 month later...

Adding a recommendation to Post #2

 

If you are working with Skyrim Special Edition textures - Recommend you use the Intel Texture Works plugin instead of NVidia plugins
( At time of writing there is not a version of this plugin for G.I.M.P .. So SSE Modders need Photoshop currently )
Reason : See this topic on Reddit

In short, SE can use textures with newer forms of DDS compression. Why is this note worthy? Some of you may or may not know that in original Skyrim compressed textures could either be DXT1, DXT3, DXT5, or 5.6.5. These are all problematic in some way because DXT1,3,5, reduce file size, but really destroy the quality on some textures especially normal maps. For comparison, a mod author would create a normal map that looks like this, but saving it with DXT compression would make it look like this. To combat this, some authors would just leave their textures uncompressed or use 5.6.5. This fixes the artifacting problem, but file sizes becomes a problem with even a 1k texture being 4mb rather than .6mb when compressed, this adds up fast when your game loads lots of textures.

So on to the new stuff. From my testing, SE will use textures saved with BC7 compression. The Intel Texture WorksPlugin will do this for Photoshop users. GIMP and Paint.net I haven't found any alternatives for you but they might be out there. BC7 offers very nearly the same quality as uncompressed while still being the same size as DXT5 compressed textures. It does this so well, I can say that in SE, there's almost no reason to use uncompressed textures when BC7 can be applied. This is pretty cool. Those textures we 'must not compress for fear of blockiness' can now be compressed without fear of loss in quality. Here's that same normal map from above with BC7 now.. (Compare it with the first one I linked, they're very nearly identical.)

Here's another example using body normal maps from Mature Skin which up until now should have always been saved uncompressed for the best look. Here's femalehead_msn.dds uncompressed. With DXT1. With BC7. I also put together an animated .mp4 to show a quick side by side comparison. And Since imgur compresses the crap out of things, here's an original tga download with the screenshots.

Another way to visualize the difference between our uncompressed and compressed textures is the 'difference' blend mode. If you don't understand or know what this is, it's simply just a way of showing which pixels are different between two images. (Pretty straight forward). Black means theres no different between the two images and colored pixels represent a different. The images are probably explanation enough though. So we'll take the uncompressed femalehead_msn place it on the bottom layer and then take the DXT and place it on the top layer and switch its blend mode to difference. This is the result. As you can see all the non-black pixels are what's difference between our uncompressed and DXT compressed texture. It's not too bad I guess, but there's quite a lot. This is the result for BC7. Far less colored pixels which is good because it shows us that very little has changed going from uncompressed to BC7.

So where does this leave us? Well going forward, I'd recommend that all mod authors use BC7 when they can for SE textures. Normal maps especially since the artifacts are very apparent with DXT compression. For diffuse maps without alpha DXT1 is still 'fine' in most cases, it still yields a smaller file size than BC7 and it's okay to use unless certain patterns or designs fail to look good using this method. For Diffuse maps with alpha layers, there's no reason not to use BC7 since it gives the same file size as DXT5 at a higher quality.

I hope this was helpful and my droning on about texture quality didn't bore people to sleep. I listed some general FAQs people might have about this new finding.

  • Q Wait, does this mean I should go out re-save everything using BC7 compression?

  • A Well, probably not. If you have a texture that's already compressed, saving it with BC7 won't take away the artifacts that are already there. You'd need the original uncompressed textures and then you can re-save them as BC7 to see any improvements. It's up to mod authors to re-save their stuff with BC7. But like I said most diffuse maps are okay with DXT1/DXT5 compression. It's mostly normal maps that would benefit from BC7.

You should also consider the fact that if you've never noticed compression artifacts, you probably won't see any benefits to using BC7 to begin with.

  • Q: 'BC'?

  • A: Block Compression. it's in reference to the method in which the algorithm compresses the images. Read more about it here.

  • Q: What's with 'BC7' not DXT7?

  • A: Well technically 'DXT' naming for compression was dropped a long time ago, but Skyrim being 'old school' it's still a useful term for us. But technically DXT1 is BC1, DXT5 is BC3. DXT3 just sorta got dropped, because it's practically useless anyway.

  • Q: What about the other newer forms of BC compression?

  • A: I haven't tested them extensively, but some just can't be used because of the way they save the information to the RGBA channels. BC6H i've ruled out already, but that's mostly for cubemaps or HDR images. BC5 might be usable for normal maps, but it yields the same file size as BC7 anyway, so i'm not sure of the point. I haven't tested it in game however. BC4 Might be able to be used for saving parallax maps, if parallax ever makes it to SE.

  • Q: What about fast vs. fine compression in the Intel Texture works plugin?

  • A: The fast preset just uses their fast compression method when saving the texture. Which is much quicker than the fine preset. However, I don't really notice a huge different in quality between the two presets. If you don't mind waiting an extra 3 or 4 seconds then just use the fine preset if you're in that big of a rush, then the fast preset will do.

  • Q: So textures work the same as they do in Fallout 4 then?

  • A: Honestly I have no clue about FO4, I know then engine can load BC7 and BC5 textures and that's about all. The game hasn't held very much interest for me so maybe someone with more expertise can give a comparison.

I'd like to give a shout out to /u/zilav for originally bring up the idea. It would have never occurred to me on my own. Also to /u/eejoseph for finding the comment by zilav and sharing it with me.

TL;DR : Skyrim SE now supports BC7 texture compression. This is far superior to the older styles of texture compression original Skyrim uses. BC7 is so good, it can be used to save textures we previously had to leave uncompressed while keeping the file size the same as the old textures for Skyrim but maintaining nearly perfect quality compared to the uncompressed textures.

Edit: I've been told that BC5 can also be used for specular and normal maps and is preferable over BC7 in terms of quality while still being the same file size. I haven't tested this myself, just passing along what I hear.

Edit2: Tested BC5 on normal maps in game. Would not recommend it. It makes objects appear much darker than they should be. Likely due to the fact that BC5 makes the blue channel completely black.

 
You will also notice Skyrim SE face normal maps if you load them using the old plugins, some of the channels are swapped - BUT in game it makes no difference .. So whats going on there then ?
 
Load the same files up with the new Intel Texture Works plugin and the channels are not swapped. So the new normal maps for faces are fine, and the old NVidia plugins / G.I.M.P open source plugins are loading these files wrong.
 
See the quote from Jon in this post
Link to comment
Share on other sites

  • 1 year later...

I has news, very nice news.

I am no longer keeping Adobe Photoshop and its damned subscription going .. Step asside Adobe and make way for :

Affinity Photo, currently just shy of 50 of your english pounds - https://affinity.serif.com/en-gb/photo/

It was only available for Mac, but now is available for Windows, and its superb - Dont just take my word for it, here are some reviews ..

http://www.techradar.com/reviews/pc-mac/software/graphics-and-media-software/image-editing-software/serif-affinity-photo-1295010/review

https://www.softwarehow.com/affinity-photo-review/

 

And check this getting started video out, it has some features that are frankly amazing

https://vimeo.com/channels/affinityphoto

 

It imports .psd files

 

Plugins : They are coming, a few are supported already but the main one we are interested in is ..

Intel Texture Works plugin for all the new DDS texture formats ..

Sadly at this time its on the developers to-do list, so could be a while.

 

----------------------------------------

 

So what to use to export those BC7 Linear fine textures in the meantime ?

Save your texture image out as a lossless format ( say .png ) from Affinity

Now please do not laugh, Paint.Net now supports all the new DirectXTex DDS formats

Seriously, when you go to Paint.Net "Save As..", use the drop down menu to select Direct Draw Surface (DDS)

Now you can save textures and generate mipmaps in all the new Intel Texture Works formats, from Paint.Net.

If the image you load is Lossless format, and you dont do any editing in Paint.Net, just save it out as whatever dds format you need and the whole procedure is as good as if you used Photoshop to edit and export your dds texture.

 

So good resource file for your texture image, edit it with Affinity instead of Photoshop, save your image in a lossless format.

Load it up in Paint.Net, then "Save as" DDS, select your format, generate MipMaps etc.

 

 

Y38cz3j.png

 

@MadCat221 @Nico coiN @Arthmoor @Hana @Lorelai @Metallicow @Sheson @Jon @Jeir - I think I have everyone I know of who will be interested in this and may wish to get Adobe's claws out of your bank, please ping any others you know of on this board I have not mentioned.

Link to comment
Share on other sites

Quote

...Note, when Paint.Net saves the dds file, it will give it a file extension of .dds2, just rename it to .dds and it will work fine...

:facepalm:

...But also note, if you try to load up say a BC7 texture into Paint.Net, rename it to have a .dds2 extension again, and Paint.Net will now Load it...

:facepalm::facepalm:

Link to comment
Share on other sites

  • 1 year later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...