PyFFI

[sen4mi]

Does anyone here know how to work with amorilia's PyFFI?

 

Apparently, pyffi can work with bsa files, just fine: http://www.gameskyrim.com/pyffi-python-file-format-interface-t84727.html

 

But when I try (in python)

 

bsadata= pyffi.formats.bsa.BsaFormat.Data()

bsaStream= open(filename, 'rb')

bsadata.inspect(bsaStream)

 

I get a data structure that only has a few integers. I do not understand how I can read the bsa directory.

 

I suspect that the issue must be bsa format changes, between Oblivion and FalloutNV, which suggests I need an updated bsa.xml. But if someone knows how to use this system, and has time to explain some basics to me, I would greatly appreciate the help.

[Mickster]

Why not extract stuff from the BSA, run the meshes through PyFFI and then repack?

[Docfmf84]

Does anyone here know how to work with amorilia's PyFFI?

 

Apparently' date=' pyffi can work with bsa files, just fine: http://www.gameskyrim.com/pyffi-python-file-format-interface-t84727.html

 

But when I try (in python)

 

bsadata= pyffi.formats.bsa.BsaFormat.Data()

bsaStream= open(filename, 'rb')

bsadata.inspect(bsaStream)

 

I get a data structure that only has a few integers. I do not understand how I can read the bsa directory.

 

I suspect that the issue must be bsa format changes, between Oblivion and FalloutNV, which suggests I need an updated bsa.xml. But if someone knows how to use this system, and has time to explain some basics to me, I would greatly appreciate the help.

[/quote']

 

Ok have you gone here it helped me quite a bit http://www.tesnexus.com/downloads/file.php?id=37548

 

 

[sen4mi]

Why not extract stuff from the BSA' date=' run the meshes through PyFFI and then repack?

[/quote']

 

I am not trying to create a bsa file, I am trying to inspect bsa files.

 

Specifically, I am working on a script to check game configuration integrity. I have, in the past, unpacked all my bsa files and used that directory structure to check for missing files. But this is tedious. And every time I change my game configuration I would have to throw away my unpacked directory structure and create another one, if I want to check my new configuration for completeness. And, I am hoping that I can turn this script into something I can give to other people to use. And the people that most need this kind of script are the kind of people that get confused during complex installations. (Which, of course, includes me.)

 

Anyways, I have something that is approaching usefulness, but there's no point in reporting missing files needed by a plugin when that file is in a bsa.

 

But I am now thinking that PyFFI is a dead end, for my purposes. I think it will be to be easier for me to write my own bsa reader than it will be for me to get PyFFI working and useful for my checking script. (It does not help that the nif.xml included in PyFFI is so old that it will not read FalloutNV nifs.)

[Klamath]

You may want to check out BSAOpt and DDSOpt. IIRC they're both based on a library designed to handle all the files in Beths games.

[sen4mi]

I have written the code I need.

 

If anyone else needs something to just read bsa directories, here is a copy of my test instance:

 

(EDIT: requires python 2.7)

 

import collections, re, struct, sys

def unpack(spec, data, offset):
keys= [pair.split(':')[0].strip() for pair in spec.split('\n') if ':' in pair]
fmt= ''.join([pair.split(':')[1].strip() for pair in spec.split('\n') if ':' in pair])
return collections.OrderedDict(zip(keys, struct.unpack_from(fmt, data, offset)))


data= open(sys.argv[1]).read()

header= unpack('''
start:4s
version:I
folders_offset:I
archive_flags:I
num_folders:I
num_files:I
folder_name_length:I
files_names_length:I
file_flags:I
''', data, 0)

assert header['start'] == 'BSA\x00'

folders= []
offset= header['folders_offset']
for n in range(header['num_folders']):
folder= unpack('''
	sort_key:Q
	file_count:I
	data_offset:I
	''', data, offset+n*16)
folders.append(folder)

offset= header['folders_offset']+16*header['num_folders']
allfiles= []
for folder in folders:
name_len= struct.unpack_from('B', data, offset)[0]
folder_name= data[1+offset:offset+name_len]
offset= offset+1+name_len
folder['name']= folder_name
folder['files']= files= []
for n in range(folder['file_count']):
	file= unpack('''
		sort_key:Q
		file_size:I
		file_offset:I
		''', data, offset+n*16)
	files.append(file)
	allfiles.append(file)
offset= offset+folder['file_count']*16


filenames= data[offset:offset+header['files_names_length']].split('\x00')[:-1]
for name, file in zip(filenames, allfiles):
file['name']= name

 

I made a few minor changes in my working copy but this is good enough to show the basic structure of the code.

 

I think converting this to something that also extracts contents from the file should be easy, but I do not care about that right now. I have no intention of building code that builds bsa files -- we already have plenty of that. This is for building scripts that can reference bsa contents.

 

(EDIT: I can supply the cleaned up code that I am currently using where I added a few actual comments and where I have structured the thing to be useful to me, if anyone feels like they want that.)

[prideslayer]

FYI, and to sound like a broken record, FOMM already does this. Start with a clean install an don't install anything without using FOMM (except those things that it won't install) and it provides all the management of your data directory that you need.

[sen4mi]

FYI' date=' and to sound like a broken record, FOMM already does this. Start with a clean install an don't install anything without using FOMM (except those things that it won't install) and it provides all the management of your data directory that you need.

[/quote']

 

How does FOMM tell me which files are missing?

[Guest Loogie]

I don't know, but in theory no files should be missing if you use FOMM.

[sen4mi]

I don't know' date=' but in theory no files should be missing if you use FOMM.

[/quote']

 

... and if I have installed everything needed for all of my mods.

 

I do know that FOMM has this information available to it: FNVEdit can determine which files are referenced by which mods, and FOMM could hypothetically determine which of those are currently missing from my install. But its UI does not expose that information to me (unless I have overlooked something major -- and I really hope that I have overlooked something major).

 

[Docfmf84]

I don't know' date=' but in theory no files should be missing if you use FOMM.

[/quote']

 

... and if I have installed everything needed for all of my mods.

 

I do know that FOMM has this information available to it: FNVEdit can determine which files are referenced by which mods, and FOMM could hypothetically determine which of those are currently missing from my install. But its UI does not expose that information to me (unless I have overlooked something major -- and I really hope that I have overlooked something major).

 

 

Sounds like you are wanting a form of wrye bash for Fallout NewVegas.

all I can say as i look at all the Code you all do... "you awsome moders" Is be humbled by your abilitys to write code from scratch AND MAKE IT WORK!!!!!!!! I try to write a little myself but it seems 90% of the time it just dosent work the way I intend it to.

 

 

[prideslayer]

FYI' date=' and to sound like a broken record, FOMM already does this. Start with a clean install an don't install anything without using FOMM (except those things that it won't install) and it provides all the management of your data directory that you need.

[/quote']

 

How does FOMM tell me which files are missing?

 

It won't tell you if files are missing, but if you let it manage everything, no files will ever be missing,

[Mickster]

Same can be said about Wrye Bash. But what if the moder makes a mistake and forgets to pack some of the meshes/textures or he points the ESP entry to the wrong place.

[prideslayer]

Same can be said about Wrye Bash. But what if the moder makes a mistake and forgets to pack some of the meshes/textures or he points the ESP entry to the wrong place.

 

Not sure what you mean by that. "What if the modder screws up?" can only be answered by "The modder needs to fix it." In either case, it won't corrupt the data directory.

[Mickster]

My point is that such a tool could be very useful for avoiding that scenario. Kinda like getting rid of dirty edits with ElMinster's tools.

[prideslayer]

My point is that such a tool could be very useful for avoiding that scenario. Kinda like getting rid of dirty edits with ElMinster's tools.

 

I still don't "get it" I guess.

 

If I release a mod and somehow forget to include a file or texture, or (more real world) you're using SCR which requires a bunch of models and textures that aren't included in it, the "right way" to install those missing files is to simply use FOMM to install them as well.

 

If the files are included but in the wrong directory, and the mod author isn't awol, a fix from them will resolve it. If they are awol, then somebody else will take over and fix the issue. In the mean time, anyone can just make a copy of those files (outside their FONV directory), zip them up with the right structure, and install that with FOMM as well.

 

I'm just not understanding how such an issue requires (or desires) a new tool that, in the end, will only serve to confuse other tools (like FOMM) about what is vanilla vs. what isn't.

[Klamath]

I may be misunderstanding things, but it sounds like you want something similar to a Fallout version of TES4Files. It reads though esps/esms and pulls all referenced data files into a separate folder. Handy for packaging mods and can let a user know if anything's missing.

[sen4mi]

Sounds like you are wanting a form of wrye bash for Fallout NewVegas.

all I can say as i look at all the Code you all do... "you awsome moders" Is be humbled by your abilitys to write code from scratch AND MAKE IT WORK!!!!!!!! I try to write a little myself but it seems 90% of the time it just dosent work the way I intend it to.

 

I do not think my end goal is very much like wrye bash' date=' though they do share a features in common.

 

Also, if things fail for you 90% of the time, that just needs you need to try 10 times before they work!

 

Anyways, mostly I am using a lot of google searches, a lot of testing and a small amount of common sense.

 

And, I am trying to build a FalloutNV version of "Lint" http://en.wikipedia.org/wiki/Lint_(software)

 

This tool would only generate a report on an installation, and would not modify any part of the game, nor would it build any files which would be used by the game.

 

So far, I have a partially working tool. It tells me, for example, that Sexout.esm has the following quirk:

 

Checking: Sexout.esm

PROBLEM: sound\fx\phy\human\bodyfall_h\ ( The system cannot find the path specified ) from Sexout.esm

FormID: SOUN - Sound [010075A3']

EDID - Editor ID: SexoutThud

 

That error message is misleading -- it's python's error message when I try to open the file. The real issue is that the named file is a directory rather than a sound file.

 

And, I am not certain if this is a problem -- for now, it is just information. But Sexout.esm is not a mod that uses a large supply of resource files. So this tool in its current form does not have much usefulness in the context of Sexout.esm. (I can add a check for the nvse extender plugin, when Sexout.esm is installed, but before that can be useful I need to reduce the "noise" from the tool, so that high priority issues are obvious. And, ideally, when I implement that warning, I would like to associate that warning with the presence of the bytecode range the plugin supports, rather than a file name -- but I do not know yet if I will be able to make that work.)

 

This tool might be useful in the context of SexoutCommonResources.esm which requires a dozen other packages be installed properly.

 

FOMM currently will not tell the user "SexoutCommonResources has references to 100 files which are missing", nor will it tell the user "you have animations which use the bones breast.L and breast.R but your skeleton does not contain these bones".

 

The issue I am trying to address is: people make mistakes. And I think I can write a tool that will catch many of the common mistakes.

 

If people did not make mistakes, FalloutNV would not crash.

 

(However, if it turns out that my tool is useless, I will still have gained knowledge and experience from my attempt.)

 

Anyways, Prideslayer: I do not see how a tool that only generates a report on an installation could possibly confuse FOMM. Am I missing something? (I so often do )

 

 

[prideslayer]

 

FOMM currently will not tell the user "SexoutCommonResources has references to 100 files which are missing"' date=' nor will it tell the user "you have animations which use the bones breast.L and breast.R but your skeleton does not contain these bones".

 

The issue I am trying to address is: people make mistakes. And I think I can write a tool that will catch many of the common mistakes.

 

If people did not make mistakes, FalloutNV would not crash.

 

(However, if it turns out that my tool is useless, I will still have gained knowledge and experience from my attempt.)

 

Anyways, Prideslayer: I do not see how a tool that only generates a report on an installation could possibly confuse FOMM. Am I missing something? (I so often do )

 

 

[/quote']

 

This all sounds good to me, just doesn't sound the same as how you explained it earlier, when you were talking about unpacked/unpacking BSAs. There's no reason to do that, especially for vanilla assets, when you're talking about missing mod files.

 

I doubt "shipping" FONV mod files by packing them into BSAs is ever going to be very popular, and checking vanilla BSAs is a needless step.

[sen4mi]

 

FOMM currently will not tell the user "SexoutCommonResources has references to 100 files which are missing"' date=' nor will it tell the user "you have animations which use the bones breast.L and breast.R but your skeleton does not contain these bones".

 

The issue I am trying to address is: people make mistakes. And I think I can write a tool that will catch many of the common mistakes.

 

If people did not make mistakes, FalloutNV would not crash.

 

(However, if it turns out that my tool is useless, I will still have gained knowledge and experience from my attempt.)

 

Anyways, Prideslayer: I do not see how a tool that only generates a report on an installation could possibly confuse FOMM. Am I missing something? (I so often do )

 

 

[/quote']

 

This all sounds good to me, just doesn't sound the same as how you explained it earlier, when you were talking about unpacked/unpacking BSAs. There's no reason to do that, especially for vanilla assets, when you're talking about missing mod files.

 

I doubt "shipping" FONV mod files by packing them into BSAs is ever going to be very popular, and checking vanilla BSAs is a needless step.

 

SexoutCommonResources requires some packages which have nif and dds files supplied in bsa.

 

 

[prideslayer]

SexoutCommonResources requires some packages which have nif and dds files supplied in bsa.

 

Mikoto Beauty is the only mod I have installed right now that has a BSA. I don't doubt you, as I said, just "not very popular," not "you will never see." Good luck with the project in any case.

[sen4mi]

SexoutCommonResources requires some packages which have nif and dds files supplied in bsa.

 

Mikoto Beauty is the only mod I have installed right now that has a BSA. I don't doubt you' date=' as I said, just "not very popular," not "you will never see." Good luck with the project in any case.

[/quote']

 

Thank you!

 

And, I just now checked, and I mis-stated: SexoutCommonResources only requires one mod with bsa content (which is VegasChokers). However, maternity clothes requires TribalPack.

 

(And, to get this content into the game, a person must either include the entire mod or must manually extract the files from the bsa and build an fomod from this extracted content. I think VegasChokers is safe, but TribalPack worries me since it reminds me of Bodies by Race.)

[prideslayer]

That last part isn't right. You don't need to install the entire mod, or "any" of it.

 

To "deal with" a BSA with FOMM:

 

1. Don't install the BSA.

 

2. Open the BSA with FOMM (BSA Browser).

 

3. "Extract All" to a new directory of your choosing (outside the DATA folder).

 

4. Close the BSA Browser.

 

5. Go to the directory you extracted to and zip it up.

 

6. Install the zip as normal with FOMM.

 

7. Never bother with or worry about a BSA again.

 

You only need to install the BSAs, not the "entire mod". FOMM can manage both just fine in either case though, and can even extract the BSA to a directory -- which you can then just zip up and install as a normal MOD.

 

It keeps the correct directory structure for you.

[prideslayer]

After the above steps, you manage the individual files in your "BSA ZIP" from within the FOMM file manager as normal -- you don't have to dig through the directory by hand deleting or otherwise inspecting stuff. This is the kind of thing the FOMM file manager is really good at.

[sen4mi]

Yes, your detailed steps here were what I meant to convey when I said "or must manually extract the files from the bsa and build an fomod from this extracted content."

 

But your steps of course go into much greater detail.