Two Questions Regarding BSAs

[Poontank]

At the moment I am working on a plugin which is intended to use BSA(s) for an easy install/uninstall and to prevent 'Data' folder cluttering with quite a large amount of files. Since I have no real experience with the BSA format besides unpacking such archives to get to their resources, two questions come up that need clarification:

 

Is there a size limit to BSAs? Since I have read using compression can be detrimental (because the game needs to uncompress the archive everytime it accesses its files) I'd rather not compress the mod's resource collections. Which in turn means those BSA(s) might become the largest ones in the game's history... Way larger than the game's very own BSAs, probably around the 5 GB mark for the textures!

 

The other question is about folder structure. Ever since I started working on the mod I threw all texture files only into a few subfolders - one for armors, one for clothes, one folder for weapons. Over time they have become quite large even though many meshes are sharing the same set of base textures wherever possible. Is this a feasible system or would it be more advantageous to create a lot of subfolders?

 

For the meshes I do have subfolders to categorize the equipment, it's just the textures that are stored in a few ever-increasing directories. Maybe it was a mistake not to organize the textures like that as well from the beginning... Can having large amounts of files in one single folder be detrimental to loading times or the general flow of the game?

[Jayce Dimmer]

For BSAs: 2.1GB is the limit. Dun go over that. You can have multiple BSA files, by the way.

 

I'd suggest using a butt-load of subfolders, but no more than is necessary. Occam's Razor 'n' shiz.

 

Having too many files in one folder affect loading and whatnot? Not that specifically. Can having too many loose files affect loading and the like? Yes. BSAs should be used whenever possible.

 

Unless you've got some insane SSD raid system going.

 

That being said, you can remove the bsa-decompressing penalty by setting the compression in BSAOpt to "0(off)". I do not know if this is any better than just having loose files, though. Oblivion refuses to invalidate archives when I'm using MO, so I haven't had the urge to investigate in-depth.

[gregathit]

bsa archives are handy to help with compatibility issues.  While it won't solve them, it will help keep one mod's files from overwriting another.  

The downside of bsa files is that ANY loose file that is the same name and in the same file structure, will overwrite the bsa when you archive invalidate via bsa redirection.  So you still have to be careful there.

 

I also recommend using the zero compression on the bsa files to keep your game running as fast as possible.

 

Cheers,

Greg

[Poontank]

Ah, thank you very much. This is valuable input, especially about the compression.

 

So I probably will need to create several BSAs for the textures. I reckon the filenames won't pose a problem because I have renamed everything to be more easy to find and assign to a multitude of meshes.

 

I'm still on the fence about the subfolders. Very probably I will create them now for any new additions and then over time put the existing files in there in the refining phase of the project. Reassinging a few texture paths in the process...

 

Although my questions in fact are answered I will wait a little longer if someone else still has anything else to add regarding BSAs in general.

[varenne]

Prior to BSA creation for any meshes, I highly recommend running a mesh optimization process. I've done this with all the mesh BSAs I have, that I couldn't find on the 'net.

 

IMO, the impact to FPS is significant enough to warrant the level of effort involved. If you need help with that just let me know. I started doing that a short while back to many of Krista's & Colourwheel's MODs, and even Anvil Bay Mistery House has benefited to where it is now significantly more playable. The end goal there would be to have BSAs for some of Krista's and maybe Colourwheel's MODs that currently don't use them.

 

My next suggestion would be to maybe post your file/folder structure, along with the final file names, or just post BSA alphas for us to test. I have a rather extensive individual file system, even though it is supposed to be slower than using BSAs. I could test to see if your intended BSA has file and/or folder name duplication in my setup, where archive invalidation would override you BSA. and end up using the loose files.

 

 

[Poontank]

Thank you for your offer, Varenne! This is exactly what I will need at a certain point in the near future - testers to point out the inevitable flaws and shortcomings. On a mere aesthetical level as well as on a technical one. The mod's dependencies are MPB++ and x117race.esm. Stuff about everyone should have active in the load order.

 

It's also a good idea to optimize the files even further than I already do. Although I am using every single one of Symon's recommended optimization techniques when saving the Nifs, I have no clue about PyFFI at all. In general I am coming more and more to the conclusion this actually shouldn't be my big modding project - it could become a community effort instead for which as many of the talented LL modders as possible can contribute content according to each and everyone's personal skills. Makes for a much richer and more varied product. So I certainly am going to take you up on your offer of PyFFI optimizing.

[varenne]

The mod's dependencies are MPB++ and x117race.esm. Stuff about everyone should have active in the load order.

 

Actually no, not everyone uses those two, the Tamago/Hiyoko thread is just one example of that. There certainly is less difficulty in implementing with the .omod version being available but some simply don't like the 'look' of MBP++ NPCs. So you may want to consider a non-MBP++ version if that is possible with your intended MOD.

 

PyFFI is fantastic with mass optimization of files, gerra6 has identified just one feature that is not working correctly and that needs to be addressed via Nifskope spells. One thing to add into your BSA calculations is that PyFFIed files may end up ever so slightly larger. Not much but it can add to the overall total. Just a early heads up so we don't exceed the BSA file size limit.

 

I have a fairly stable pre-process using NifSkope and PyFFI together. I now do a cleanup of NIFs prior to running gerra6's tools on them like Mesh Rigger. You'd be surprised how dirty or poorly structured NIFs of old can be, and how much an improvement to FPS can be achieved.

 

Another personal optimization quest for me these days is editing EIDs of older MODs; those with numbers (integers) in the beginning of the EID. That can cause all kinds of scripting issues, not to mention CS/E instability when building MODs. Again I have noticed a significant improvement in FPS stability with doing this. One does have to take care when building MODs with these though, if they are in any way used as a master. More on that later.

 

Just ping me when you're ready. I'm back on a new employment contract, but this one is 100% from home so I have time to spare and work on Oblivion stuff again 

[Symon]

As I was summoned.

 

Firstly, I agree with Varenne when it comes to MPB++ and x117race. I have neither and always remove any cosmetic dependencies from mods. Some of us like women to look like women!

 

As for BSAs I'm going to say that I always unpack these for convenience and improved performance! Breaking open the Beth mesh and texture BSAs noticeable speeded up my rig.

 

Think about it.

With loose assets, your system makes file system calls to locate and read them. Simple in NTFs.

With BSAs it has to make system calls to locate the BSA and start reading it. Then make 'game' calls to locate the asset you need in the BSA, offset to the right place in this massive file, which will require more system calls to find, then load the asset into memory.

 

I do not know where the idea that BSAs are somehow faster than NTFs came from. Are 7z or zip files faster to locate things in than a simple disk read? Ever spotted a txt file and decided to read it without extraction? Notice in can take many seconds to open. All a BSA is another archive format. A compressed BSA is even slower of course as you have to decompress parts of it.

 

Beth (and other companies) use a compressed archive because it is easier for them to distribute, not because it offers better/faster asset access.

 

With that out of the way, I much prefer manual Nif cleaning. After all, I may notice a Nif is still oddly large, remove doubled vertices and save a ton of space. I may also notice the Nif uses a huge 2048x2048, monochromatic texture (or worse, an even larger fairly featureless normal map), more space to save. I may notice the dds file has no mip maps, so can spare my processor having to create them on the fly.

 

Don't get me started on how there a lot of broken keyframes.

 

[Poontank]

The input from LL members I consider the most tech-savvy keeps on coming! Much appreciated, Varenne and Sy.

 

I'm afraid that master file dependency is most crucial to the mod... Because this is a slightly revamped and expanded version of MoDem's old 'City Life for x117'. To me the 'main star' of the mod are its now more than seventy-five x117 characters, as well as a handful of Miqote NPCs (a race introduced with the ++ version of MBP). Deleting those NPCs would take out a large chunk of the mod's attractivity.

 

Movomo also stated using BSAs is neither advantageous nor necessary. Now I'm convinced to refrain from using archives for the mod. I was just fascinated with the idea of a neat and tidy mod set-up - dropping a few BSA archives into the 'Data' folder and people are good to go. But in all actuality, with just one single mesh and texture folder, installing the mod will be just as easy. A thought which didn't occur to me in the first place! Oblivious to the obvious.

 

So the initial plan of using BSAs and my questions on that topic brought about unrelated but important points regarding file optimization.

 

Ever since I started working on the files I have deleted truckloads of bogus nodes and duplicate vertices. Combined thousands of material properties and updated tangent spaces like a madman. To think that even material from high-profile modelers oftentimes contains lots of junk...

[Symon]

Ah yes. If it's an x117 mod, then that cosmetic mod is a requirement, and anything that depends on.

Hopefully, you can assume a decent skeleton too. Which one is another matter. The more I inspect animations, the more obvious it becomes that there are a great many skeletons with unique bones out there. Think op1-3 is standard? Not when you find a great many animations that refer to brest1 and 2 instead. (Love the miss-spelling as well). Easy enough to change in Nifskope but......

 

There are a lot of strange things in some mods. In part this is due to the massive breadth of skills required. A breadth almost no one has.

 

The perfect modder would be:-

Creative, good with Photoshop, NifSkope, Blender or 3DSMax (both meshes and animations), the CS (level design, dialogue and especially scripting) and more besides. Not everyone can manage this of course.

 

Edit: If you want an example, search your ~\data\meshes directory for nif files containing "program files". You should find none but will probably find tens or even hundreds containing badly hard-coded texture paths.

[Poontank]

Now that you mention skeletons: Recently I stumbled upon TalkieToaster's 'Skeletal Variation' system. Is this a feasible approach for making a few particular characters more unique? From the readme and TT's pictures it seems legit, easy-to-use and without any drawbacks apart from the occasional slight distortions of armor and clothing parts. It should be way more advantageous than Growlf's body resizing system or even VORB for a similar effect.

 

There might be a potential incompatibility/conflict with combat systems like DR and/or UV? A lot of people are using them...

 

Now this idea is slowly coming up: With a bit of extra work the MBP dependency can easily be scrapped. Implementing the relevant MPB/x117 races as standalones and changing the characters to these race versions, getting rid of the master file dependency in Gecko, adding the race asset folders to the mod's files. Cowabunga! Everybody will be able to use the mod. Well worth the effort, takes only a short time.

[varenne]

There might be a potential incompatibility/conflict with combat systems like DR and/or UV? A lot of people are using them...

 

Again an assumption. I have both DLed, have kept up with updates and watched both evolve, but have never yet installed either. I would say "a lot of people" is highly subjective. Many? Maybe. I think it's a select group that have them installed and active; adrenaline first-person shooter junkies come to mind.

 

I've also had debates with at least one stellar Sr. member on BSAs vs. loose files, and I'm still not convinced that BSAs load any where noticeably faster than loose files. Bottom line as I see it is both have to be accessed and read into memory to be run in-game. If I want to organize and simplify a MODs structure I may go in the direction of a BSA, simply to contain everything. But I would argue it's more about containment and structure for say distribution purposes than speed. And If I want to go .omod I still feel loose files are just as good as BSA, especially if I am needing to alter anything during subsequent releases; alpha, beta, final.

 

Any how, looking forward to your MOD as I am fully MBP++ and x117 enabled. I've had various versions of MoDem's old 'City Life for x117' enabled at numerous points, but ended up disabling it for a variety of reasons. It may be another MOD entirely but I recall a certain Nord or maybe it was an imperial that caused me issues as she would warp in right on top of my PC, causing CTDs. I'll be keeping an eye out for her during your alpha release.

[Poontank]

Do you mean Gertrude Eisenbrech? I'm totally fond of her. She has a find player package, probably to spook out people with her stalking. With the right equipment she can be an attractive and competent support character. There's also her sister Beatrice in Leyawiin.

 

Those combat mod systems I spot frequently in people's load orders, that's why I generalized their potential widespread use. Back when I was only playing console games I enjoyed my fair share of gory mangling ultra-violence. But nowadays I'm not that interested in this type of gameplay anymore so I never felt compelled to install any UV or DR. Mainly using archery instead. Attempting third-person trick shots.

 

It's just that there might be the pitfall of skeleton issues when using such systems on characters with a TalkieToaster special skeleton. I'd better not assign those to hostiles. No fat marauders or emaciated necromancers then.

[varenne]

Gertrude Eisenbrech? That just might be her! I liked her too, just not the game lockups she was causing me.

 

I recall having to disable that find player package at one point as it would often lock me up. Tends to be really bad if you save game just as she's warping in. I finally found out which was causing it as she would sometimes partially warp in, sort of like a ghost image of her. That find player package may need some additional limits (AI tweaks) added.

 

I don't recall her sister Beatrice causing any issues.

 

I personally would avoid anything Growlf's body resizing system or even VORB. I don't consider either that mainstream, but especially VORB, which is way too challenging for the casual Oblivion player to set up.

 

As for TalkieToaster's 'Skeletal Variation' system, I think you'd have to pay careful attention to meshes and animations those would use. Some of Emily's aka Triratna of the Monstergirl series used (borrowed from other MOD authors?) some really bizarre skeleton/mesh/animation combos, to less than spectacular results. Having a skeleton as part of a mesh has always baffled me the most, not to mention causing me major angst whilst trying to optimize stuff.

[movomo]

Oh don't worry about x117 and MBP. There are those who like women to look like cartoon characters. Myself for example.

 

Personally I'd prefer x117 and MBP dependency because it's space efficient. But it's also true some folks don't like to install that huge cosmetic mess.

It would probably be better to duplicate the race records and ship them together.

 

There is an additional advantage to this...

You may had hesitated adding other races other than included in MBP and x117 because every other race will increase dependency, but now you are free of it. You can include x110, Evy mini, FF mini, you name it.

 

Hmm.

About hard disk space problem, perhaps making symlinks might work. So, if they don't want MBP and x117, they have to install the real resource files. Or if they already have the requirements, they can replace them with zero size symlinks that are pointing to the actual race resources.

Just perhaps.

[varenne]

symlinks? First I've heard of this. What are they exactly and how do you go about implementing something like this?

 

I recently reworked a MOD but didn't care for the Evy race resources they used as mine were already tweaked and looked way better IMO. So it sounds like this would have been an approach I could have used instead.

[movomo]

hmm.

Links are similar to shortcuts(.lnk), but slightly different. It looks like a real file, unless some program specifically wants to distinguish them (such as security programs).

 

for example, windows xp had a folder named 'documents and settings'. Recent windows don't have them, but old programs still ask for it.

Then Microsoft dudes can make a link to the 'users', and the link looks like as if it's the 'documents and settings'. So the program thinks it's accessing the 'documents and settings' folder, but actually it's just looking at the 'users' folder disguised as a winxp folder.

 

Links can also work with relpaths.

[varenne]

Ah, okay a Windows construct vs. a specific Oblivion one. Got it.

[Symon]

I've often meant to try symlinks in Windows. Comes for my Unix background I suppose.

If only there were as simple in Windows as

ln -s /usr/local/oblivion/data/meshes/armour/xxx/armour.nif !!!

 

http://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/

http://en.wikipedia.org/wiki/NTFS_symbolic_link

 

and others. Note you must have an NTFS file system. These days, you probably do have but note that FAT32 of any version does NOT support symlinks.

 

An important note, ALWAYS use soft links unless hard links are mandated. Then be very careful to always delete a hard link before you delete the file it points to!

 

[movomo]

Thanks, Symon, that was incredibly helpful.

[varenne]

Thanks, Symon, that was incredibly helpful.

 

Yes, thank you. Surprised I've never tripped over that in Windows or Ubuntu. Learn something new every day.

 

Apologies for hijacking or derailing your thread Poontank!

[Poontank]

A few new questions arose in the meantime.

Now that I have abandoned the idea to use BSAs altogether this thread can be considered my general tech question ask-around place instead.

 

Magic effects: I had a neat idea that kinda got shot down when I took a look at the magic effects in the CS. They are hard-coded and seem to be impossible to expand upon? If I was to add, say, new bound armors - how could I go about that? Is there a workaround maybe via scripting that allows me to add a whole bunch of completely new bound armor spells together with their associated armors?

 

Special animations: If I was to add a totally new type of weapon with attack and movement animations unique to this weapon type - how would I add these animations to the NPCs wielding these weapons? And even more importantly - this I imagine to be the really hard part - if the player picks up such a weapon: How to make these weapons work even for the player? With the correct animations, that is.

 

If there's no way to make these weapons work correctly for the player I would have to resort to some trick making those weapons dropped by defeated enemies unusable... Something along the lines of the Arena items script that even forbids merely picking up such dropped gear.

[Satyrist]

A few new questions arose in the meantime.

Now that I have abandoned the idea to use BSAs altogether this thread can be considered my general tech question ask-around place instead.

 

Magic effects: I had a neat idea that kinda got shot down when I took a look at the magic effects in the CS. They are hard-coded and seem to be impossible to expand upon? If I was to add, say, new bound armors - how could I go about that? Is there a workaround maybe via scripting that allows me to add a whole bunch of completely new bound armor spells together with their associated armors?

Actually there are two ways of doing this;

Either by creating a scripted spell, as described here: http://cs.elderscrolls.com/index.php?title=Scripting_Tutorial:_Custom_Bound_Items

Or by using Oblivion Magic Extender from here: http://www.nexusmods.com/oblivion/mods/31981/? this plug-in allows you to create your own magic effects.

[Poontank]

Sounds good! Exactly what I need. Bookmarked this CS wiki page for future reference. This seems to be a more simple approach than using the 'Magic Extender' thingy. After all, it's just a bunch of additional bound armors. Nothing too fancy (besides the armors themselves).

[varenne]

Special animations: If I was to add a totally new type of weapon with attack and movement animations unique to this weapon type - how would I add these animations to the NPCs wielding these weapons? And even more importantly - this I imagine to be the really hard part - if the player picks up such a weapon: How to make these weapons work even for the player? With the correct animations, that is.

 

If there's no way to make these weapons work correctly for the player I would have to resort to some trick making those weapons dropped by defeated enemies unusable... Something along the lines of the Arena items script that even forbids merely picking up such dropped gear.

 

Special animations for weapons are weapon specific. A mace animation would be built around the NPC/PC holding a mace, a sword animation built around a sword, etc. So I would look to see if your intended new weapons are near to the same as existing ones, and test those Special animations in-game to see if they behave as desired.

 

Besides the vanilla weapons Special animations, there are some custom ones around, even ones for FLY. So you may, with a little bit of search work, be able to find an existing animation that matches a new weapon.

 

I don't know if you then need to go into CS/E and add that to a weapon animation group or not, I've honestly never tried it. It may require a new weapon animation group first be added then assign the Special animation to it. Maybe take a look at the FLY animations and see what they did there?