To merge, or not to merge...

[SouthernGorilla]

EDIT: I ramble a lot and made a procedural error in my testing. The good info starts in my fifth post. Tried to link it here, but couldn't get it to work.

 

 

One of the first bits of advice people give to folks who are beginning to dabble in Sims CC is to merge the files for better performance. I've been experimenting almost since I got the game and recent results have given me reason to question the worth of merging packages.

 

For one, people rarely mention the problems caused by merging. Mainly, you can't easily track what you have and avoid duplicates. It also makes it more difficult to remove or repair broken CC. Seems like what you save in boot time you lose in maintenance time.

 

I'd read about special characters in file names impacting performance and tinkering with that seemed a good way to learn Powershell. Sure enough, pruning the cruft from the file names made a very noticeable difference in boot times even with the little bit of CC we had at the time.

 

So yesterday I decided to really put it to the test. I downloaded a few CC folders and actually unmerged everything to get a nice mess. I threw my handful of Powershell snippets at the resulting 8,300 packages to get rid of duplicates and excise the junk from the file names. There were 750-odd duplicates hidden in the merged packages. That couldn't be good for performance. By the time I got done cutting out everything I already had, or just didn't want more of, there were about 5,200 packages. That gave me a total of 10,165 packages... 32.39 GB of CC. That's far from a record, but it's definitely an indication of an addiction. But the punchline to this long-winded monolog is that I get to the world-management menu in just 2.5 minutes from the time I click the launcher. I  read people all the time talking about 10-15 minute load times with less CC.

 

I still have some experimenting to do. I need to see the load time with no CC and with the "raw" mess I downloaded to see where my current setup falls on the spectrum. But I do know it loads faster now with 10,000 packages than it did a few weeks ago with a quarter of the CC.

 

I should also note that I run a really old rig. It's an Intel Q9650 on a mobo that had its last BIOS update 11 years ago. So my results can't really be compared directly to anyone else's. 

 

Anyhow, I'll update this thread when I get some more data if anyone is interested. I can tidy up the snippets I use to manipulate the files and share them as well. This is much less effort than merging and seems to produce equivalent results with no discernible downside.

 

P.S. I really wish CC creators would show some consideration for the folks who choose to use their stuff. I found one package with a name that was 79 characters long. That doesn't include the path or the extension, just the basename. That's just absurd. The average name length was just over 30 characters.

[SouthernGorilla]

I just went back and started the experiments in earnest. The folders I downloaded came with 5,155 packages. I unmerged 115 merged packages and now have 9,950 packages. So 4,800 individual files had been merged into just 115 packages, pretty impressive. Before unmerging, however, I only found 70 sets of duplicate files. After unmerging I had 616 sets with almost 1,900 total files. So an average of three files per file "name". I put "name" in quotes because I check for duplicates by hash, the actual file name is irrelevant. That's close to 1,300 duplicates, 13% of the 9,950 packages are duplicates. I don't know exactly how the game handles duplicate CC. But I can't imagine that having 10-15% of your content being duplicates would not have an adverse effect either on load times or on general performance.

[SouthernGorilla]

Well, this is an interesting twist.

 

I pulled everything out of my mods folder and fired up the game. It took 2 minutes 11 seconds to get to the world menu. That's only 13 seconds faster than it took with 10,000 CC in the folder.

 

I then put the raw junk I downloaded into the mods folder and it took 4 minutes 32 seconds to load. Which still isn't a ridiculous time. But at least it showed that a messy folder with lots of duplicates effects loading time. But I had forgotten to put my actual mods back in the folder, so I copied them back in to see if mods made a difference compared to just CC. Loaded the game and it only took 2 minutes 41 seconds. 9,950 packages loaded just 30 seconds slower than no packages at all.

 

Figuring this was due to caching, I went and did other stuff to clear out the RAM and whatnot. I also cleared out the various cache folders. Tried again and the game loaded in just 2 minutes and 6 seconds... even faster than it had with an empty mods folder.

 

The only conclusion I can draw is that either my game loads slow by default so the CC doesn't really add much time OR that CC doesn't effect load time like people think. I lean towards the latter. My bet is that folks with slow load times don't do any of the regular cleanup required to keep things smooth.

 

It's also possible the system is caching the game somewhere I don't know about. That would explain why changing the mods folder caused a significant slowdown at first but made no difference the second and third times I loaded the game. But I'm not rebooting my rig every time I load the game to test that, not feeling that energetic today. It may be as simple as the fact that I manually set up a massive pagefile and the data stays stored in it. The only way to test that is to set the system to clear the pagefile on shutdown then reboot. So we'll see.

[SouthernGorilla]

Okay, here's what I've learned to this point; content makes a difference, but only on the first load after starting the computer.

 

1) Reboot the computer, start the game with an empty mod folder... 1 minute 44 to the world menu.

2) Reboot the computer, start the game with 9,950 packages full of messy names and doubles... 4 minutes 52 to the world menu.

3) Reboot the computer, start the game with 10,365 packages with "clean" names and no double... 5 minutes 02 to the world menu.

4) Start the game again with the 10,365 package mod folder... 1 minute 44 to the world menu.

 

So the second loading of my full mods folder was just as quick as loading with an empty mods folder. I rarely shut down or reboot. Which means the amount of content, or the tidiness of it, is irrelevant to me.

 

I'd have to run these tests a few times each to come up with a statistically significant average. But I don't see the need. It seems pretty clear that, yes, content makes a difference. But only on the first load after booting the computer. Without rebooting my load times are all within 30 seconds of each other regardless whether the mods folder is empty or loaded with rubble. The only thing I haven't tested yet is whether merging all this stuff would really make a difference. Guess I'll have to see if S4S will cooperate with me and merge a bunch of stuff at once.

 

At this point, my "official" recommendation to people is to manually set your pagefile and make it pretty big. Windows does not, by default, clear it at any point. It's basically persistent RAM. So even if you reboot your rig regularly your game will stay in "memory" just like you hadn't shut down. That's the only explanation for why my load times have been so quick and consistent.

[SouthernGorilla]

It occurred to me that I was measuring the wrong thing. I forgot there was time between the world menu and the game still.

 

Desktop to in-game, no mods:                            01:50

Desktop to in-game, mods with "clean" names:   03:33

Desktop to in-game, mods with "messy" names:  05:30

 

So just cutting the special characters out of the file names and getting rid of duplicates cut two minutes off the load time. These times are all without reboots so they may change when I try each mode after restarting the computer.

[RayOhio]

On top of duplicates I find outdated game mods like Wicked Whims Tuning and Nisa's Wicked Perversions in the merged file, I unmerged a 227MB file tonight and the sim used 1 hat out of the entire file. I cut 900mb off of the size of the size of that sim when I saved it to a zip drive and it was all stuff that the sim didn't need. She looks the same at 161MB as she did at 1.1 Gigs.

[SouthernGorilla]

Yeah, as I get deeper into this I'm finding that merged packages are basically evil. Well, when you get them from someone else. People are just grotesquely incompetent and randomly merge any bit of junk they have. Why would you merge actual mods with CC? That's just bizarre. I found one file with over 700 packages in it, completely random, broken stuff, just a mess.

 

I haven't finished setting up the new testing protocol yet. But my guess, right now, is that any performance gain from merging comes from the fact that the file system has fewer names to sift through.

 

Here's how I plan the test;

 

1) 10,000 mods, completely raw, as downloaded. Except for being unmerged.

2) 10,000 mods, no duplicates.

3) 10,000 mods, no duplicates, no subfolders

4) 10,000 mods, no duplicates, no subfolders, remove special characters from names.

5) 10,000 mods, as above, but renamed so that every file has a 4-character name. Just to see if short names improve speed.

 

Finally, I'll merge the fastest group and the slowest group to see if merging improves either group's performance. I'm thinking I'll merge each into 100 files of 100 packages. I genuinely don't expect merging to help the fast group.

[Scorpio]

Good experiment.

Hardware and files sizes play a big part of load times too rather than the game itself.

And pagefile sizes too as you mentioned.

People with older computers or laptops will likely have far slower load times and different results of course.

My game loads much slower after I delete the localthumbcache which I do quite often due to adding/removing mods/cc frequently.

 

Somehow I recently got hold of a unknown merged file containing tons of eyebrows which I have to look for now.

Added another custom sim to my game recently.

EDIT: Found it: ARIANA MARIE downloaded from LL has a merged makeup file of 45K.

Also had roburky emotional inertia and other cc traits included which often break with game updates. 

 

I merge my hairstyles and clothing but it's a pain when I want to add more and have to check my un-merged backups first. 

 

 

[CanadianGhost]

Interesting - glad i already merge and clean the file names as best as I can.  Im pretty sure this is already done on your part but that stupid checklist that pops up in the beginning that shows all the cc content that you have loaded, disable that as well, thatll speed up your load time as well.

 

I would have figured the two load times that you can reduce the most are the start up and the game menu to your saved file.  The third load from world menu to your home I would imagine depends entirely on what you already got built, the lot, the number of sims you got, etc.  So all user created on that aspect.

 

If you come up with additional means to reduce the load time would love to hear about it.

[SouthernGorilla]

Yeah, the hardware makes a huge difference. I'm not going to worry about the absolute times so much as the relative times between the various sets of content. I have all the folders set up, I just need to make time to run them all. Well, first I need to make sure I can trick the game into thinking the files are in the Mods folder when they're actually on another drive. It will take a few hours to run the experiment probably since I'm going to reboot between each run to prevent caching from effecting the times. So... five folders, five runs each to get an average, with a reboot between each of those... that's fun. Hoping to find time tomorrow to get started.

 

In the meantime, Here's a list of the special characters and how many of each were in the filenames to start with. The first one is just a space. How much time does it take for the system to process all this?

"Name"	"Count"
" "		"22556"
"_"		"12703"
"-"		"6905"
"["		"2658"
"]"		"2652"
"("		"697"
")"		"695"
"."		"214"
"+"		"180"
"{"		"169"
"}"		"168"
"'"		"90"
"&"		"51"
"@"		"43"
"#"		"29"
","		"18"
"!"		"8"
"~"		"4"
";"		"1"
"™"		"1"

 

[Oops19]

A reboot in between? That will give you random start times. Also the initial build of localthumbnail.db-package will lead to strange results and it may be built differently each time.

 

If you want to know the time TS4 needs to process these files / data make sure you have enough memory to keep the files and TS4 in the file system cache. Otherwise the time may vary based on your ssd/hdd speed. Even though this overhead and the error is really small.

 

 

I'll keep my files with original file names manually sorted on NAS and create merged packs for TS4.

[SouthernGorilla]

I know from previous testing that load times are largely identical if you don't reboot in between. Having everything cached makes a huge difference. What I'm emulating is the common practice of people shutting off their computers when they aren't using it. 

[Oops19]

At least for me caching seemed to make only a small difference, reading 20 GB from SSD. Reading the files from a fragmented HDD could of course have an impact. I keep my computer in stand-by - only !$@##$ Windows updates require 1-3 reboots.

[Scorpio]

I'm well overdue for running defrag again, lol.

[somethingnew90]

I know it's necro, but considering I found the topic through google and OP is the only one actually experimenting rather than doing guesswork, I thought I'd add my case study on how not to be a CC addict:

 

With 50gbs worth of CC my start-up times averaged 32 minutes. With all of them but gameplay mods merged into about 30 packages, the game loaded in 5,2 minutes on average. I know it's a lot, but in my defense it's 3-4 years worth of CC

 

My problem with merging is the "maintenance" OP mentioned. Unmerged, I keep my CC organized by date, with gameplay mods and their objects obviously kept separate. When merging, I mixed all the non-gameplay mods into a single folder, divided them into roughly 2gb portions (as Sims 4 Studio can't reliably merge anything bigger ime.) This was a test to see what improvement I could get, resulting in the 5.2 minutes I mentioned.

 

Now, if I were to properly merge all this CC, I'd have to go by CC-type and creator-names to be able to unmerge and re-merge when I add something new. Three problems with that:

-CC creators giving non-descript names (XY_Maria, go figure), not using proper spacing in names, using special characters in names, naming their CC in cyrillic and other script, foreign languages etc.

-The sheer amount of mods I'd have to go through

-I'd need an unmerged, back-up CC folder to see if newly downloaded CC has any duplicates - that's 100gb for CC only, each new CC taking up double the space.

 

After spending about 8 hours organizing CC, I gave up, realizing it's not the best way to spend my quarantine. I've got an "unknown" folder for CC with non-descript names, and that itself had ~800 files I'd have to sort through via Sims 4 Studio just to find out what they are. This is in addition to trying to figure out what CC I don't need or want anymore. I've checked about 500 pieces of clothing and based on that, I'd say I'd ditch 25% of it all, but there is no tool to quickly see what's what and delete. Sims 4 Studio hangs 5-10 seconds between items, and that adds up quickly. All in all, I'd probably need a whole week of 9-5 to get everything in order. Sims 5 will probably drop by the time I get that time back in hours played.

 

Moral of the story: Get seriously organized as soon as possible. Figure out a system that allows you to pinpoint specific CC, and consider how you'll remove and add CC if necessary. A single custom sim's mods folder may contain hundreds of files, taking up space to 300mbs. Sort it out quickly, or you'll get to the point where merging is not a good option anymore.

[madmoper]

All good info here so far. Thanks SouthernGorilla for the explanation of your thought and testing process and results.

 

I'm in the latter stage of cleaning up my CC (though not the double-digit gigabytes others have). Deleted duplicates, renamed file names with only letters and numbers. For legibility I used underscores to close spaces and separate names from dates/versions.

 

A combo of a screenshot program to capture unwanted items in CAS and then using the screenshots as references while sifting through the CC packages with Sims 4 Studio has been my method for removing broken or unwanted CAS CC. CAS is a bigger display than what Sims 4 Studio has so that's why I screenshot CAS.

 

Things that helped me most to trim down my CC have been:

• To be very picky and delete low quality CAS/BB CC.
• To delete CAS CC that is too small to see the detail while playing the game. I'm usually zoomed far out enough to see and manage multiple sims in a household. Things like highly detailed rings, earrings, eyelashes, freckles, etc. are lost in zoom-out and the blur of moving around with the camera/view. If CAS were my main focus and enjoyment in Sims 4, then it would be a different matter.
• To delete any items or swatches that are similar enough to be redundant. This goes along with my previous point -- what level of detail do I actually see when playing the game? How many different varieties and shades of blue jeans do I really need if they all look the same when I'm zoomed-out to keep an eye on multiple sims? Again, if CAS were my main thing, then my choices would be just the opposite.
• To have a full backup of all CC, unaltered, unedited, unmerged, somewhere else so that I have the peace of mind that I can always retrieve and do-over. I've had plenty of do-overs as I fumble my way through this learning process.

Quote

 

 

[SouthernGorilla]

On 2/18/2022 at 9:06 AM, somethingnew90 said:

I know it's necro, but considering I found the topic through google and OP is the only one actually experimenting rather than doing guesswork, I thought I'd add my case study on how not to be a CC addict:

 

With 50gbs worth of CC my start-up times averaged 32 minutes. With all of them but gameplay mods merged into about 30 packages, the game loaded in 5,2 minutes on average. I know it's a lot, but in my defense it's 3-4 years worth of CC

 

My problem with merging is the "maintenance" OP mentioned. Unmerged, I keep my CC organized by date, with gameplay mods and their objects obviously kept separate. When merging, I mixed all the non-gameplay mods into a single folder, divided them into roughly 2gb portions (as Sims 4 Studio can't reliably merge anything bigger ime.) This was a test to see what improvement I could get, resulting in the 5.2 minutes I mentioned.

 

Now, if I were to properly merge all this CC, I'd have to go by CC-type and creator-names to be able to unmerge and re-merge when I add something new. Three problems with that:

-CC creators giving non-descript names (XY_Maria, go figure), not using proper spacing in names, using special characters in names, naming their CC in cyrillic and other script, foreign languages etc.

-The sheer amount of mods I'd have to go through

-I'd need an unmerged, back-up CC folder to see if newly downloaded CC has any duplicates - that's 100gb for CC only, each new CC taking up double the space.

 

After spending about 8 hours organizing CC, I gave up, realizing it's not the best way to spend my quarantine. I've got an "unknown" folder for CC with non-descript names, and that itself had ~800 files I'd have to sort through via Sims 4 Studio just to find out what they are. This is in addition to trying to figure out what CC I don't need or want anymore. I've checked about 500 pieces of clothing and based on that, I'd say I'd ditch 25% of it all, but there is no tool to quickly see what's what and delete. Sims 4 Studio hangs 5-10 seconds between items, and that adds up quickly. All in all, I'd probably need a whole week of 9-5 to get everything in order. Sims 5 will probably drop by the time I get that time back in hours played.

 

Moral of the story: Get seriously organized as soon as possible. Figure out a system that allows you to pinpoint specific CC, and consider how you'll remove and add CC if necessary. A single custom sim's mods folder may contain hundreds of files, taking up space to 300mbs. Sort it out quickly, or you'll get to the point where merging is not a good option anymore.

Your experience mirrors my own when I first started this process. I have another thread I started about a video I'm working on to show how to handle CC properly. There are tools and techniques which make it fairly painless, though still time-consuming. I've actually recorded about five different videos, but I've canned them all because I rambled too much. Seems I need to have a script to keep me on track, so I'm working on that at the moment.

 

But the main point is that, as you said, merging causes more trouble than it's worth. I have over 14,000 packages, some 54 GB, and my game loads in just over two minutes. Finding and deleting broken/unwanted stuff is fairly easy with Sims 4 Tray Importer, which also does a good job finding duplicates. The only thing I use S4S for now is fixing tags so I don't have cowboy boots in "swimwear".

 

Also, TwistedMexi's "Better Exceptions" mod is great for finding out-of-date stuff that may cause problems.