Snoopy911 Posted December 3, 2018 Posted December 3, 2018 Moved to download section After downloading a lot of mods i noticed i had a lot of duplicates. I tried ModConflictDetector which finds duplicates but didn't like deleting single files since i had a lot of duplicates. I checked the internet for tools too find file duplicates, found tools like ccleaner but didn't like what i read about spyware and other stuff included in many of this programs. Thus i downloaded the free visual studio 2017 version and made a tool myself. A warning if you decide to use it. It's about an hours work so don't expect anything fancy. I tested a little bit with my sims 4 installation and it was working well so far, but if you destroy your machine it's obviously your own fault, downloading strange stuff from the internet . If you decide to ignore my warning, 0) better make a backup of your mods folder. 1) copy the exe to your sims 4 mods folder, run it and press the big button. 2) you are done Any duplicate package files should be moved to a folder DuplicateBackup in your sims 4 base folder. If there is more than one of the same package file, only one will be moved to the backup folder, the rest will get deleted. The filelist is sorted by the full path length and it will keep only the file version with the shortest length. The idea was to rather remove files which are deep in your folder structure than the files from the main mod folder. At the end empty folders will be removed. I will also add the visual studio project and feel free to change it, if you want some more features or fix some terrible bugs i didn't notice Update: Reading TrueMayhem's remark about using the recycling bin, i checked the option to use it, but would have caused more problems than it's worth i think. Nevertheless Mayhem was right about that there should be a way to undo in case of a mistake. I tried to move the files including the folder structure and it's working quite well this way i think. There is no need to delete files anymore unless you keep copying the same duplicates to the same subfolder again and again. In this case the backup subfolder file will get replaced. While i was testing i also noticed a lot of other files like pictures, readmes and even zip's and thought i don't need this stuff. Thus i added a checkbox to allow also moving any stuff which is not required to run your mods (files except package, ts4script, cfg). If you leave it unchecked it will only move duplicate .package files. If you activate the checkbox it will move duplicate .package files and any file except package, ts4script and cfg One more warning, it only checks for filenames when checking for duplicates, not the content itself. If you have files with a simple name like ear.package which are all different content, located in different folders, it will move all of them except one. I didn't notice a single case where i have different cc with the same name in my collection, but can't tell about yours.
Dhexztah123 Posted December 4, 2018 Posted December 4, 2018 i like to try this since i have a lot of duplicated too. thanks for sharing but i got scared lol
Guy14T Posted December 4, 2018 Posted December 4, 2018 I just gave it a try. Looks like it worked just like you said it would.
Snoopy911 Posted December 4, 2018 Author Posted December 4, 2018 @Dhexztah123 yes that was the idea to make you scared. I could have written i do this kind of stuff for a living for a long time and it should be just fine, but especially since i do this kind of work for a living i know you might miss something with a spontaneous work like this one. I got really annoyed when i didn't find a suitable tool and broke a very old rule to never install something work related like the dev environment at home. Think of a dentist getting a teeth ache at home in the evening and grabbing a pipe wrench. It might work but will probably not be up to his usual quality standard :). All said i think it works just fine but you might want to wait for more brave souls like Guy14T to check it out.
TheTrueMayhem Posted December 4, 2018 Posted December 4, 2018 Hi ! Just a query. You wrote that the first duplicate will be moved to the backup folder and any others will be deleted. Do they get moved to the recycle bin ? or deleted plain and simple ? I ask because if they're shifted to the recycle bin it will be easy to restore them if there is a mistake ? Or is there a prompt asking for confirmation before deleting ?
Chipendale Posted December 4, 2018 Posted December 4, 2018 I've used WinMerge for the longest of time to get rid of any duplicates, so there's another suggestion for y'all ?
Snoopy911 Posted December 4, 2018 Author Posted December 4, 2018 11 hours ago, TheTrueMayhem said: Hi ! Just a query. You wrote that the first duplicate will be moved to the backup folder and any others will be deleted. Do they get moved to the recycle bin ? or deleted plain and simple ? I ask because if they're shifted to the recycle bin it will be easy to restore them if there is a mistake ? Or is there a prompt asking for confirmation before deleting ? That was a good remark and updated the post above with a new version making it easier to restore. I didn't use the recycle bin since this might not work on all machines. I'm recreating the folder structure in the backup folder and thus there is no more need to delete, if there is the same file in many different folders. Well unless someone insists to copy the same file to the same folder over and over again, than the version in the backup folder will get replaced. ps There is no confirmation because i'm using the tool myself and i'm lazy :). Confirmations were the reason i did't use conflict detector, because i would have spent hours confirming every single action. Would add a single one if there was an option to check all actions before starting the job, but this would require a lot more work.
TheTrueMayhem Posted December 5, 2018 Posted December 5, 2018 Thanks so much for letting me know - I greatly appreciate it and will try it later this week when I get the chance !! I love the 'lazy' comments ! Sounds like me ! haha
VikOliCyr Posted December 5, 2018 Posted December 5, 2018 I'm allways using duplicate file detective with a strong checksum to eliminate EXACT DUPLICATES (same content, same size, same extension, different names) you can also check for duplicate filenames.
angiebaby23 Posted December 9, 2018 Posted December 9, 2018 @Snoopy911 I don't scare easily, so I gave this a try and it worked like a charm. Thanks!
Snoopy911 Posted December 9, 2018 Author Posted December 9, 2018 1 hour ago, angiebaby23 said: @Snoopy911 I don't scare easily, so I gave this a try and it worked like a charm. Thanks! Think i could remove the scary warning. I'm using the tool for a week now and since i started only recently with the sims 4 i obviously had to download some stuff. The mod folder is now about 11 GB with 3 GB moved to the backup folder. Seems i didn't really check the stuff during my download orgy :), but it was some nice testing for this little tool and i didn't notice any problem.
rb888 Posted December 10, 2018 Posted December 10, 2018 On 12/3/2018 at 4:13 PM, Snoopy911 said: After downloading a lot of mods i noticed i had a lot of duplicates. I tried ModConflictDetector which finds duplicates but didn't like deleting single files since i had a lot of duplicates. I checked the internet for tools too find file duplicates, found tools like ccleaner but didn't like what i read about spyware and other stuff included in many of this programs. Thus i downloaded the free visual studio 2017 version and made a tool myself. A warning if you decide to use it. It's about an hours work so don't expect anything fancy. I tested a little bit with my sims 4 installation and it was working well so far, but if you destroy your machine it's obviously your own fault, downloading strange stuff from the internet . If you decide to ignore my warning, 0) better make a backup of your mods folder. 1) copy the exe to your sims 4 mods folder, run it and press the big button. 2) you are done Any duplicate package files should be moved to a folder DuplicateBackup in your sims 4 base folder. If there is more than one of the same package file, only one will be moved to the backup folder, the rest will get deleted. The filelist is sorted by the full path length and it will keep only the file version with the shortest length. The idea was to rather remove files which are deep in your folder structure than the files from the main mod folder. At the end empty folders will be removed. I will also add the visual studio project and feel free to change it, if you want some more features or fix some terrible bugs i didn't notice Update: Reading TrueMayhem's remark about using the recycling bin, i checked the option to use it, but would have caused more problems than it's worth i think. Nevertheless Mayhem was right about that there should be a way to undo in case of a mistake. I tried to move the files including the folder structure and it's working quite well this way i think. There is no need to delete files anymore unless you keep copying the same duplicates to the same subfolder again and again. In this case the backup subfolder file will get replaced. While i was testing i also noticed a lot of other files like pictures, readmes and even zip's and thought i don't need this stuff. Thus i added a checkbox to allow also moving any stuff which is not required to run your mods (files except package, ts4script, cfg). If you leave it unchecked it will only move duplicate .package files. If you activate the checkbox it will move duplicate .package files and any file except package, ts4script and cfg One more warning, it only checks for filenames when checking for duplicates, not the content itself. If you have files with a simple name like ear.package which are all different content, located in different folders, it will move all of them except one. I didn't notice a single case where i have different cc with the same name in my collection, but can't tell about yours. The tool CCMover.zip and the project in case you want to play with it yourself CCMover_Net4.5_Solution.zip Thanks any chance make one for skyrim too that be handie?
Snoopy911 Posted December 10, 2018 Author Posted December 10, 2018 54 minutes ago, rb888 said: Thanks any chance make one for skyrim too that be handie? It would be easy to change the file filter but don't think it's a good idea and also the mod managers are good enough to mostly avoid duplicates. While i don't have skyrim installed anymore i checked my fallout 4 folder and removing duplicate files would be a problem. Many files have generic names like 1,2,3,... but completely different content. For sims i only saw unique file names up to now. Also Bodyslide for example has several dozen language files all named the same but in different folders. I think it would rather cause a lot of problems than help. Searching for duplicates is no option i think. The second option of my little tool, moving out non essential content might work and the option to configure file types would be very easy to implement. Checking the FO4 files i also think it would be no good idea. Tried a file filter on jpg with windows explorer and it looks like some files might be used by mods, tried .txt, and also think some might be used by mods. I'm not sure if there is a file type where you can be sure it won't be used at all.
Recommended Posts
Archived
This topic is now archived and is closed to further replies.