FPS Drops from too much high res texture?

[endersgame]

I've been noticing that when looking directly at a group of 6 followers, my FPS drops from a solid 60 to around 52~54 in certain spots. These include the bridge near Riverwood (looking at Riverwood), Whiterun gate near the Khajit camp (while looking towards the farms).

 

While I realize 50+ FPS is more than playable, I'd like to figure out what's causing the drop - CPU or GPU.

 

I am using HDT with 3BBB (CPBC only) and in limited mode, and Skyrim Monitor tells me my CPU usage is around 10~16% (Ryzen 3900x), so it doesn't seem like a CPU issue.

 

However, it appears my VRAM usage is around 7~8 GB (1080TI with 11GB) - which seems... excessive (went a little overboard with high texture mods)

 

My question - could the high VRAM usage actually contribute to a FPS drop, or is the FPS drop just Skyrim being Skyrim?

 

Edit: I'm also running Fill Her Up, Hentai Pregnancy, and Apropos (for the SlaveTat feature) - not sure if those could be contributing to the drop or not.