[Tutorial] How to edit Normal & Spec maps for Fallout 4

[XunAmarox]

Normal Maps

 

Someone named justice123 over at Nexus found out a workaround to edit normal maps for textures in Fallout 4. Here's what he said:
 

How do i edit normal map?
In fact the way i used to make this textures, was not perfect.
it's little different for original normal maps (just little bit brighter, but not that much)
i tried google it for proper way, but i just can't find any

well this is quick tutorial for my improvised way

1. when you open up the vanilla normal map, you will notice that it's brighter then ordinary normal map(it because Fo4 uses '3Dc' format for normal maps)
2. open up the 'level' adjust window by pressing Ctrl + L
3. in 'input level', you may find tree values, adjust very left of it into '128'
4. and now you got ordinary looks of normal map! Edit what ever you want!
5. save it as DDS, and select the format '3Dc' you don't need to change level again, when you save it as 3Dc, it will automatically changing
6. done! but as i said, it's not proper way to make normal map

I've tested this method and it does work, and it's easy to do. It's by far the easiest method. Ironically enough, it also produces the best results.
 
For another method using NVIDIA's GPU Accelerated Texture Compression Tools, see inside:
 

For this one you'll need NVIDIA's GPU Accelerated Texture Compression Tools. If you have the DDS tools for Photoshop, it's not the same thing, so you'll still need it.
 
Also, before I continue... this is a command line tool. If you want to use it for compression you'll have to make *.bat files. Here's the command line documentation. We are unfortunately going to have to use for compression. You'll see why later.
 
If you don't know what format the original textures are in you can view them with NVIDIA's Windows Texture Viewer (at the bottom). Once you open a texture with that program the format will be at the bottom left. Of course, to get the textures in the first place you'll need B.A.E. - Bethesda Archive Extractor if you don't already have it. If you want to edit materials then cell here uploaded a Material Editor to the Nexus.
 
Alright. So, this isn't nearly as complicated as it sounds. Once you have NVIDIA's GPU Accelerated Texture Commpression Tools navigate to its install directory, which for me is "C:\Program Files\NVIDIA Corporation\NVIDIA Texture Tools 2\bin" and that's the folder that you're going to need to be in to work with this stuff.
 
Before we continue, extract a norm (_n) or spec (_s) from one of Fallout 4's *.ba2 archives. If you want to follow along, when you open BAE click "file" then navigate to your FO4 Data directory and select Fallout4 - Textures1.ba2 then navigate to Textures > Actors > Bloatfly, click Select None up at the top and then just put a checkmark in the Bloatfly box which will select all of its textures. Extract that to whatever directory you want as your BSA extraction directory. You can, of course, use whatever textures you want for this as long as they're norm or specs.
 
For the purposes of these instructions we have three files extracted:
Bloatfly_d.dds
Bloatfly_n.dds
Bloatfly_s.dds
 
I'll assume you know what those are. Just go ahead and copy them over to the NVIDIA Texture Tools 2/bin folder mentioned earlier. What we're going to do next is drag Bloatfly_n.dds onto nvdecompress.exe - you don't need to open it, just drag the file onto it and drop it. Once you do you should have a new file created called Bloatfly_n.tga. Go ahead and open that up in Photoshop.
 
If everything went right it should look like this:

 
At this point if you're familiar with normal maps you're probably wondering "wtf is this crap?! It's yellow!" but that's easy to fix. Just invert it. In Photoshop that is Ctrl+I. We now have this:

 
We're now working with the normal maps that we know how to work with and you can make whatever changes you need to make.
 
If you had opened the normal map previously you'll recall that it looked like this:

 
Don't worry about that. Do the changes you need to make and when it comes time to export you can export as DDS in 3Dc format, but it's not ideal. The results aren't perfect unfortunately. You can get a lot closer to the original if you use the tool itself via command line (or creating a *.bat file). It's pretty simple, just create a new text file. You can name it whatever you want, I named mine compress. Change the extension from *.txt to *.bat and edit it in the text editor of your choice. For the purposes of this texture, it would read as follows:
 
 

nvcompress -normal -bc5 Bloatfly_n.tga Bloatfly_n.DDS

 
 
Here's some comparisons  (click to enlarge), and why we aren't just exporting directly from Photoshop in 3Dc format:

 
So for that, you'll want to just save the tga file and then run the command line *.bat file you made on it.

 
To bottom line it, here are the results (click to enlarge):

 
The easiest method is practically 100% indiscernible from the original and the results are significantly better than the other methods. I am still including it however as the methods described will be useful when dealing with specular maps, as I'll cover below.

 

Specular Maps
 
As for the spec maps... I'm not even going to pretend that I fully understand how they work. They "can" be replaced with the old DXT greyscale ones for some things but it's not really that simple. The red channel has the metallic properties and reflections. The green channel controls specularity. The blue channel controls subsurface scattering. They're in 3Dc format (ATI2N / BC5).

 

I know some of you are completely in the dark though so I'll share what I do know about the spec maps... First of all, the Levels "easy" method above for normal maps that produces the best results, it doesn't work with spec maps. They're... way more complex. But okay, here's how I know that...

 

If I open up a spec map directly these are the Channels that I'm presented with:

 

Everything is way too bright and there's subsurface scattering (blue channel). That's just flat out wrong. Adjusting the levels does not fix this.

 

Using GPU Accelerated method (drag the dds texture onto nvdecompress.exe) in the spoiler above to decompress the texture first however produces the following channels:

 

Now, this looks right. There's nothing at all in the blue channel (pure black). The red & the green channels are both greyscale, and you can sort of treat them like spec maps that you're used to working with in previous Bethesda games, but keeping in mind the caveats mentioned above (e.g. what these channels are actually for).

 

You can edit them directly but if you're a fan of layers you can just go to the red or green channels and select everything then copy it, then go to the RGB channel and paste it in as a new layer and edit as you normally would. Do ctrl+shift+c (copy all visible layers) to copy your modifications when you're ready to add them and go back to the red or green channels and paste it in there. It'll replace it a lot like working with alpha channels. Obviously it would be best to have a PSD for your work and then copy/paste from that into the channels on the unmodified image that you're actually going to be saving.

 

We can't export from Photoshop directly as a 3Dc because Photoshop royally screws up the output, swapping the Red & Green channels for some bizarre reason. We're going to have to use command line from the GPU Accelerated method in the spoiler above. It's a little different for spec maps though. You'll use this line in a *.bat file (though obviously with the input/output filenames matching what you're working with):

nvcompress -bc5 Bloatfly_s.tga Bloatfly_s.DDS

We're doing this because with the Windows Texture Viewer opening the original it displays as an ATI2N texture and according to the documentation that uses the -bc5 flag with the tool. We aren't however using the -normal flag because we aren't working with a normal map. The result is identical to the original and once decompressed you can see the channel information is also correct.

 

So, hopefully that helps. If anyone has more information to add, or methods that produce better results, then by all means please share.

[XunAmarox]

There's a new tool released by Intel called Intel ® Texture Works Plugin it seems to work without the need of using any external programs other than Photoshop.

I can't actually tell which method is better because Photoshop won't open DDS files saved with the Intel plugin using NVIDIA's plugin. They just look black and messed up (and crash NVIDIA's Windows Texture Viewer), but open fine using the Intel program. I'm guessing it's because NVIDIA's plugin isn't updated to work with the new headers... and I assume that B.A.E. is exporting with the incorrect headers. So, I don't know. But you can use it if you want. It's a lot easier.

[Z0mBieP00Nani]

How do you create really detailed normal maps from scratch that look like the Bloat Fly normal maps in your pictures? I know You can use a plug in made by NVIDIA for Photshop (a normal map filter), but it doesn't produce that kind of depth and detail.

[Vioxsis]

^ By baking a high poly sculpt to the low poly mesh.

[Jeffglobal]

^ By baking a high poly sculpt to the low poly mesh.

 

Yeah, what exactly to you mean by low poly mesh?  Say for example, the HD hands.obj from the modders resources Xun and Caliente released for their CBBE model?

 

All I want to do is create better effects for my manicure mod.  It would definitely help me make stuff in substance designer and painter, but from my perspective, having the hd obj of the hands and not the low poly model they used for the UV unwrap, I really can't do anything with it.  I could make my own UVW map, but I still have no tools to make a nif that the game reads UNLESS the FO4 reads the "old" nif exports from 3ds max.

 

I just learned 10 mins ago, the NifSkope 2.0 dev4 2015 12-14 WORKS IF you shut off Avast.  Hot damn, that should be on the actual download page man!

 

Idk.

 

Thanks for any clues to point me in the right direction.

[Vioxsis]

Yeah, what exactly to you mean by low poly mesh?  Say for example, the HD hands.obj from the modders resources Xun and Caliente released for their CBBE model?

 

A low poly mesh will have its tri count between one and tens of thousands, while a high poly will be well into its millions/billions.

 

The HD hands.obj is a 'low' poly 'game ready' mesh. They are the models used in the CBBE mod in game.

The reason they are there is due to a lack of mod tools particularly nif importers/exporters at the time.

So Caliente built fbx importing in to OS (it already had obj importing) to get skin data in and OS exports working F4 nifs, and sheared the obj/fbx so that others can get a jump start on making mods.

 

All I want to do is create better effects for my manicure mod.  It would definitely help me make stuff in substance designer and painter,

Substance painter can bake high poly meshes to low poly meshes.

 

but from my perspective, having the hd obj of the hands and not the low poly model they used for the UV unwrap, I really can't do anything with it. I could make my own UVW map,

As i said above the HD hand.obj is the low poly, and its UVs are done already, if you change them it will break compatibility with other mods so best to avoid that if possible.

 

but I still have no tools to make a nif that the game reads UNLESS the FO4 reads the "old" nif exports from 3ds max.

Why would you need to make a nif for a texture? If you are making a manicure mod then would you not just need to edit the albedo/diffuse?

[Jeffglobal]

 

 

Yeah, what exactly to you mean by low poly mesh?  Say for example, the HD hands.obj from the modders resources Xun and Caliente released for their CBBE model?

 

A low poly mesh will have its tri count between one and tens of thousands, while a high poly will be well into its millions/billions.

 

The HD hands.obj is a 'low' poly 'game ready' mesh. They are the models used in the CBBE mod in game.

The reason they are there is due to a lack of mod tools particularly nif importers/exporters at the time.

So Caliente built fbx importing in to OS (it already had obj importing) to get skin data in and OS exports working F4 nifs, and sheared the obj/fbx so that others can get a jump start on making mods.

 

All I want to do is create better effects for my manicure mod.  It would definitely help me make stuff in substance designer and painter,

Substance painter can bake high poly meshes to low poly meshes.

 

but from my perspective, having the hd obj of the hands and not the low poly model they used for the UV unwrap, I really can't do anything with it. I could make my own UVW map,

As i said above the HD hand.obj is the low poly, and its UVs are done already, if you change them it will break compatibility with other mods so best to avoid that if possible.

 

but I still have no tools to make a nif that the game reads UNLESS the FO4 reads the "old" nif exports from 3ds max.

Why would you need to make a nif for a texture? If you are making a manicure mod then would you not just need to edit the albedo/diffuse?

 

 

Ty for your detailed reply.  I'm still learning how to get back to replies to my own posts...I had given up forums in the dial up days...I literally had to hear those stupid phone sounds.

 

The reason I considered making new uv maps was:

I have a manicure mod and I wanted to do a couple of things for my customers as per their suggestions:

 

-Have longer and different shaped finger nails:  I can do that now with 3ds max given the new nif import/export made by Jon, but idk how to handle the material ids and how that affects the skin modifier and the BS modifier that looks like the material ids list for a nif file that I can't manipulate and the fbx file loses if I export that way (at least given my knowledge)

 

-Have special effects for the nails.  One woman wanted super glossy nails and I thought I could get that with the specular map manipulation, but I ran into the material file...wtf is a material file...so I tried to learn to make my nails glow like the glowing bobbleheads or magazines or something...AIN'T so simple (for me) it's NOT just glowmap true, glowmap femalehands_g.dds...  Too many parameters, too little info.

 

-Have mani-pedis.  Now this is where I thought about changing the unwrap.  Given the toes/nails were so little on the texture map, I could mimic and redo the map to increase the pixels used for textures there as per best practices I've just learned for optimizing uv maps.

 

I've been going from finger painting to PS to Substance Designer and Painter, to 3ds max learning, in an ad hoc manner, continuously not knowing what I don't know I don't know.  I've been watching a lot of videos about the programs and have made a lot of progress in 3 weeks, but it's 3 weeks...

 

Thank you for your time and any help.

 

 

[Vioxsis]

The reason I considered making new uv maps was:

I have a manicure mod and I wanted to do a couple of things for my customers as per their suggestions:

 

-Have longer and different shaped finger nails:  I can do that now with 3ds max given the new nif import/export made by Jon, but idk how to handle the material ids and how that affects the skin modifier and the BS modifier that looks like the material ids list for a nif file that I can't manipulate and the fbx file loses if I export that way (at least given my knowledge)

You have a working (fallout 4) .nif exporter for max?

Material IDs don't affect the skin modifier at all.

BS modifier? do you mean BSDismemberSkin? If so then thats how you setup dismember partitions (so the game knows what body parts it needs to mask when they get destroid).

 

-Have special effects for the nails.  One woman wanted super glossy nails and I thought I could get that with the specular map manipulation, but I ran into the material file...wtf is a material file...so I tried to learn to make my nails glow like the glowing bobbleheads or magazines or something...AIN'T so simple (for me) it's NOT just glowmap true, glowmap femalehands_g.dds...  Too many parameters, too little info.

To make something 'super glossy' i would have the roughness map (the green channel in the "****_s.dds") practically white, that would get you the effect you want.

 

A material file (.bgsm) looks to be a way of editing settings in the .nif (probably to reduce having duplicate meshes just to edit simple effects).

http://www.nexusmods.com/fallout4/mods/3635/? <- an editor to make your own from scratch.

http://www.nexusmods.com/fallout4/mods/1124/? <- command line tools, the above was based off (if you don't like/need a gui)

 

To make something glow you need to have a viable glow map and to have it set properly in the nif as well as turning on some flags.

If you open the glowing bobbleheads/magazines .bgsm in a text editor you will see all the changes being made to the nif to make it glow.

 

-Have mani-pedis.  Now this is where I thought about changing the unwrap.  Given the toes/nails were so little on the texture map, I could mimic and redo the map to increase the pixels used for textures there as per best practices I've just learned for optimizing uv maps.

Given what you plan to do it might be better to model new nails on the hand (over the top of the old ones kind of like false nails) and unwrap them to their own texture map, that way they can have as much pixel density as you need. Then you can either set them up as equipable in game or have them simply replace the default hand nif.

 

I've been going from finger painting to PS to Substance Designer and Painter, to 3ds max learning, in an ad hoc manner, continuously not knowing what I don't know I don't know.  I've been watching a lot of videos about the programs and have made a lot of progress in 3 weeks, but it's 3 weeks...

 

Thank you for your time and any help.

Well you can't know what you don't know..

Looks like you've jumped in at the deep end, so good luck with your learning.

 

[Jeffglobal]

 

 

The reason I considered making new uv maps was:

I have a manicure mod and I wanted to do a couple of things for my customers as per their suggestions:

 

-Have longer and different shaped finger nails:  I can do that now with 3ds max given the new nif import/export made by Jon, but idk how to handle the material ids and how that affects the skin modifier and the BS modifier that looks like the material ids list for a nif file that I can't manipulate and the fbx file loses if I export that way (at least given my knowledge)

You have a working (fallout 4) .nif exporter for max?

Material IDs don't affect the skin modifier at all.

BS modifier? do you mean BSDismemberSkin? If so then thats how you setup dismember partitions (so the game knows what body parts it needs to mask when they get destroid).

 

-Have special effects for the nails.  One woman wanted super glossy nails and I thought I could get that with the specular map manipulation, but I ran into the material file...wtf is a material file...so I tried to learn to make my nails glow like the glowing bobbleheads or magazines or something...AIN'T so simple (for me) it's NOT just glowmap true, glowmap femalehands_g.dds...  Too many parameters, too little info.

To make something 'super glossy' i would have the roughness map (the green channel in the "****_s.dds") practically white, that would get you the effect you want.

 

A material file (.bgsm) looks to be a way of editing settings in the .nif (probably to reduce having duplicate meshes just to edit simple effects).

http://www.nexusmods.com/fallout4/mods/3635/? <- an editor to make your own from scratch.

http://www.nexusmods.com/fallout4/mods/1124/? <- command line tools, the above was based off (if you don't like/need a gui)

 

To make something glow you need to have a viable glow map and to have it set properly in the nif as well as turning on some flags.

If you open the glowing bobbleheads/magazines .bgsm in a text editor you will see all the changes being made to the nif to make it glow.

 

-Have mani-pedis.  Now this is where I thought about changing the unwrap.  Given the toes/nails were so little on the texture map, I could mimic and redo the map to increase the pixels used for textures there as per best practices I've just learned for optimizing uv maps.

Given what you plan to do it might be better to model new nails on the hand (over the top of the old ones kind of like false nails) and unwrap them to their own texture map, that way they can have as much pixel density as you need. Then you can either set them up as equipable in game or have them simply replace the default hand nif.

 

I've been going from finger painting to PS to Substance Designer and Painter, to 3ds max learning, in an ad hoc manner, continuously not knowing what I don't know I don't know.  I've been watching a lot of videos about the programs and have made a lot of progress in 3 weeks, but it's 3 weeks...

 

Thank you for your time and any help.

Well you can't know what you don't know..

Looks like you've jumped in at the deep end, so good luck with your learning.

 

 

 

The "BS SubIndex Modifer" is above the skin and editable mesh... sounds VERY NifSkopy to me, and NifSkope is inscrutable to me without a manual, or any real videos.  I know, I don't know that.

 

Hey, I have not heard the specular map green channel being called roughness before, what is the red and blue channels, and I know there's something in the blue channel for both the normal and specular cause if I open a normal map with the nVidia plugin I see a faint stain of information past.  I have no working knowledge of the specular map channels.  Only yesterday could I make corrections with smudge and stamp, and additions to normal maps like a pro (5 dolla).

 

The first thing I tried was opening the bobblehead mod bgsms, but nay, they do not open, the glowing eyes mod DOES open, but idk wtf was going on there.  Then I got a material editor by ousnius and that didn't open the bobblehead mod files either.  I couldn't make jsons and look at them or directly with ousnius' program.  I wonder if you can read protect them?  He did for me!

 

I don't understand the paste on nails/equipable idea.  So I lock Caliente's mod hands in place and then I make my nail replacements...[way too close to irl according to feedback I'm getting about nails and nail polish today.  fungus ew] ok, but then idk how the game will read them floating there in 3d space all alone, no skin or bones...

 

Well this is what I have today, after just messing with the original nif, without anything fancy, AND after fk'ing the nif up once somehow, idk, so I had to do it all over again:

 

My brain hurts, cause while I was playing with 3ds max I FORGOT how I used to orbit and manipulate the viewport fast and easy to see what I was doing (I used to use the masta screen and just use shortcuts BUT 8 years ago!)...after like 4 hours, it was ok.  oh boy. 

 

The problem with the way I did it, is I stretched one area about 3/4 down the nail creating areas of distortion, so if I try to add things like skull decals to the black nails for example not really close to the nail bed I'm screwed...  I could fix that, but I first wanted to see if it was possible to manipulate the nif mesh and then put it back ingame and I didn't fk it up.  I did do that today. yay, but I really want to have an almond shaped nail (don't ask me how I know) cause that is the most sexy and it's actually trending now, but I doubt I can get reasonable results with such a low poly count in that area.  Interestingly, every finger nail was modeled in a different way, making the extension of the nails, a unique wtf did I mess up problem, but it got trivial after a while.  I thought that the modelling methods so extremely different bizarre, or it's just my idiot savant inability not to count edges...

 

Again ty for you time to reply and info you shared.

 

 

 

 

 

[Vioxsis]

The "BS SubIndex Modifer" is above the skin and editable mesh... sounds VERY NifSkopy to me, and NifSkope is inscrutable to me without a manual, or any real videos.  I know, I don't know that.

I don't have the plugin your using but your right in that its something NifSkopy (the BS at the beginning stands for bethesda studios/BethSoft).

I've seen SubIndex used in nifs replacing the BSTriShape, i can't say for sure but that modifier is probably how the exporter configures the exported nif for use in fallout 4.

 

Hey, I have not heard the specular map green channel being called roughness before, what is the red and blue channels, and I know there's something in the blue channel for both the normal and specular cause if I open a normal map with the nVidia plugin I see a faint stain of information past.  I have no working knowledge of the specular map channels.  Only yesterday could I make corrections with smudge and stamp, and additions to normal maps like a pro (5 dolla).

 

Its not a specular map its a combined texture.

 

Red = metalness map

green = roughness

blue = ??? I have not seen it used for anything so..

 

Fallout 4 uses PBR and so uses a metalness, roughness map to replace specular.

 

And you will need intels beta plugin to open the new version .dds fallout uses.

https://gametechdev.github.io/ITW-Beta-Test/

 

And the blue channel in the normal is not used, so there is no data in it.

 

The first thing I tried was opening the bobblehead mod bgsms, but nay, they do not open, the glowing eyes mod DOES open, but idk wtf was going on there.  Then I got a material editor by ousnius and that didn't open the bobblehead mod files either.  I couldn't make jsons and look at them or directly with ousnius' program.  I wonder if you can read protect them?  He did for me!

 

you can simply open the bobblehead .bgsm in notepad because they are simply .json with the file type renamed.

If they are vanilla bgsms then you will need to convert them first, or use the GUI editor.

 

I don't understand the paste on nails/equipable idea.  So I lock Caliente's mod hands in place and then I make my nail replacements...[way too close to irl according to feedback I'm getting about nails and nail polish today.  fungus ew] ok, but then idk how the game will read them floating there in 3d space all alone, no skin or bones...

 

Well this is what I have today, after just messing with the original nif, without anything fancy, AND after fk'ing the nif up once somehow, idk, so I had to do it all over again:

 

My brain hurts, cause while I was playing with 3ds max I FORGOT how I used to orbit and manipulate the viewport fast and easy to see what I was doing (I used to use the masta screen and just use shortcuts BUT 8 years ago!)...after like 4 hours, it was ok.  oh boy. 

 

The problem with the way I did it, is I stretched one area about 3/4 down the nail creating areas of distortion, so if I try to add things like skull decals to the black nails for example not really close to the nail bed I'm screwed...  I could fix that, but I first wanted to see if it was possible to manipulate the nif mesh and then put it back ingame and I didn't fk it up.  I did do that today. yay, but I really want to have an almond shaped nail (don't ask me how I know) cause that is the most sexy and it's actually trending now, but I doubt I can get reasonable results with such a low poly count in that area.  Interestingly, every finger nail was modeled in a different way, making the extension of the nails, a unique wtf did I mess up problem, but it got trivial after a while.  I thought that the modelling methods so extremely different bizarre, or it's just my idiot savant inability not to count edges...

 

Again ty for you time to reply and info you shared.

What i mean is simply creating a new mesh that sits just on top of the nails like so.

 

 

 

 

 

Do one for each nail (on one hand and mirror it to the other) and then unwrap the UVs so they have their own texture set (and not the hand one).

That way you don't inadvertently create any incompatibility by changing the hands UVs.

 

Then you have them share a nif with the hands (2 separate trishapes) and either have them as replacers to the default hands or (and this would be much easier when the GECK comes out) set them up as equipable in game (like armour is).

 

 

But nevertheless looks like your making good progress, more then me even i've not exported any meshes yet.