Creature tangent normal maps - wrong lighting direction?

[Grafenstein]

So I've been working on some creature mesh remesh/retextures, and I've hit a snag (and not for the first time).

Situation: In Blender, everything looks fine in all modes (even Render mode, from what I can tell).

I export the OBJ (or NIF). I look at the new mesh in BodySlide to see how it'll look in-game. I use the same normal map that was used in Blender.

It's all screwed up.

That's essentially the situation.

 

As the following images show, somewhere between "Everything looks good in Blender" and "Now it's in BodySlide," something goes wrong. It looks like some segments of the normal maps don't have the same lighting directions or something.

Again, these are tangent space normal maps, and the creature here uses mirrored UVs. That could be a factor. Also, as the creature has numerous different segments, that could be a factor too.

 

But still, I know people have done this before. I'm sure there's a solution.

So for my sake and the sake of everyone else in this situation:

what is the trick for going from Blender to Skyrim using tangent space normal maps for complex creature meshes?

 

Spoiler

Example: here is the shoulder in Blender

 

 

Here is the same shoulder in Bodyslide.

 

 

Here is the BodySlide view of the mesh only - no textures, no normals. Doesn't look like there's any problems here...right?

 

 

Things that have not worked:

- recalculate normals in BodySlide

- update tangents in nifskope

- face normals + update tangents in nifskope

 

[Kendo 2]

Why are you using bodyslide if you know how to navigate in Blender and Nifscope?

[Grafenstein]

2 minutes ago, Kendo 2 said:

Why are you using bodyslide if you know how to navigate in Blender and Nifscope?

Primarily for copying bone weights from reference, other weight-painting related tasks, and generally finalizing the look and characteristics of the nif before sticking it into the game.

[Andy14]

UV is unchanged?

Edit Mod -> Faces -> A -> CTRL + N in Blender for alignment of the normals already used?

[Kendo 2]

4 hours ago, Grafenstein said:

Primarily for copying bone weights from reference, other weight-painting related tasks, and generally finalizing the look and characteristics of the nif before sticking it into the game.

Yeah, bone weight copying and painting weights by hand are hit-and-miss sometimes.

Q: Are you using a vanilla normal map or one you made?

[Grafenstein]

37 minutes ago, Kendo 2 said:

Yeah, bone weight copying and painting weights by hand are hit-and-miss sometimes.

Q: Are you using a vanilla normal map or one you made?

Newly made, baked from a sculpt.

I took the low-res mesh from the game, subdivided, and used that as my base mesh. Then I used multiresolution to sculpt on top of it, then baked new tangent normals from the result. So the UV unwrapping is essentially unchanged from the game mesh.

 

59 minutes ago, Andy14 said:

UV is unchanged?

Edit Mod -> Faces -> A -> CTRL + N in Blender for alignment of the normals already used?

Thanks for the attempt, but that didn't do the trick either. I've already recalculated the mesh normals, and they don't look obviously wrong.

[Kendo 2]

Try this normal map and see if the seam is still there.  If it is; your UV map might be scaled wrong for that section of the mesh.  If is isn't; your normal map is borked.

 

flat_n.dds

[Grafenstein]

15 minutes ago, Kendo 2 said:

Try this normal map and see if the seam is still there.  If it is; your UV map might be scaled wrong for that section of the mesh.  If is isn't; your normal map is borked.

 

flat_n.dds 5.48 kB · 0 downloads

Thanks for the attempt. As you can see in the picture, there's no problem with the flat normal map.

Thus, as you say, my normal maps are somehow indeed borked. My hope is that someone will have encountered this issue before and will be able to lead me in the path of deborkification.

 

Spoiler

 

[gtarocks123]

This is likely not relevant to you anymore as this post is two years old (maybe it will help with future mods), but the issue here was that you needed to invert the green channel of your normal map. Blender uses OpenGL normal maps while Skyrim and Fallout 4 use DirectX normal maps, hence why it looked normal in Blender but messed up in-game. The only difference between these two types of normal maps is that the green channels are inverses of one another, so you have to invert the green channel of the normal map when switching between Blender and any game/program that uses DirectX normal maps.

 

Games typically use DirectX maps (although Unity Engine games use OpenGL), so if you are porting a game asset to Blender the normal map will likely need its green channel inverted to display correctly in Blender (and then inverted back when you port it to another game). If you don't want to be constantly editing textures, the green channel inversion can be done in Blender through Material Nodes so you can leave the normal map file as DirectX compatible.

 

I'm posting this in case anybody comes across this thread in the future and may find this information useful.