DupFileManager: Ver 1.1.6 beta (By David Maisonave)

DupFileManager is a Stash plugin which manages duplicate files in the Stash system. It has both task and tools-UI components.

Features

Creates a duplicate file report which can be accessed from the Stash->Settings->Tools menu options. The report is created as an HTML file and stored in local path under plugins\DupFileManager\report\DuplicateTagScenes.html.
- See screenshot at the bottom of this page for example report.
- Items on the left side of the report are the primary duplicates designated for deletion. By default, these duplicates are given a special _duplicate tag.
- Items on the right side of the report are designated as primary duplicates to keep. They usually have higher resolution, duration and/or preferred paths.
- The report has the following options:
  - Delete: Delete file and remove from Stash library.
  - Remove: Remove from Stash library.
  - Rename: Rename file.
  - Copy: Copy file from left (source) to right (to-keep).
  - Move: Copy file and metadata left to right.
  - Cpy-Name: Copy file name left to right.
  - Add-Exclude: Add exclude tag to scene, so that scene is excluded from deletion.
  - Remove-Tag: Remove duplicate tag from scene.
  - Flag-Scene: Flag (mark) scene in report as reviewed (or as requiring further review). Optional flags (yellow, green, orange, cyan, pink, red, strike-through, & disable-scene)
  - Merge: Copy Metadata (tags, performers,& studios) from left to right.
Can merge potential source in the duplicate file names for tag names, performers, and studios.
- Normally when Stash searches the file name for tag names, performers, and studios, it only does so using the primary file.
Advance menu
- Advance menu can be access from the Stash->Settings->Tools->[Duplicate File Report] menu or from the DupFileManager Tools & Util.
- Here are some of the options available in the Advance Menu.
  - Delete specially tagged duplicates in blacklist path.
  - Delete duplicates with specified file path.
  - Delete duplicates with specific string in File name.
  - Delete duplicates with specified file size range.
  - Delete with specified duration range.
  - Delete with resolution range.
  - Delete duplicates having specified tags.
  - Delete duplicates with specified rating.
  - Delete duplicates with any of the above combinations.
Bottom extended portion of the Advanced Menu is for customizing the report.
Delete duplicate file task with the following options:
- Tasks (Stash->Settings->Task->[Plugin Tasks]->DupFileManager)
  - Tag Duplicates - Set tag DuplicateMarkForDeletion to the duplicates with lower resolution, duration, file name length, and/or black list path.
  - Delete Tagged Duplicates - Delete scenes having DuplicateMarkForDeletion tag.
  - Delete Duplicates - Deletes duplicate files. Performs deletion without first tagging.
- Plugin UI options (Stash->Settings->Plugins->Plugins->[DupFileManager])
  - Has a 3 tier path selection to determine which duplicates to keep, and which should be candidates for deletions.
    - Whitelist - List of paths NOT to be deleted.
      - E.g. C:\Favorite,E:\MustKeep\
    - Gray-List - List of preferential paths to determine which duplicate should be the primary.
      - E.g. C:\2nd_Favorite,H:\ShouldKeep\
    - Blacklist - List of LEAST preferential paths to determine primary candidates for deletion.
      - E.g. C:\Downloads,F:\DeleteMeFirst\
  - Permanent Delete - Enable to permanently delete files, instead of moving files to trash can.
  - Max Dup Process - Use to limit the maximum files to process. Can be used to do a limited test run.
  - Merge Duplicate Tags - Before deletion, merge metadata from duplicate. E.g. Tag names, performers, studios, title, galleries, rating, details, etc...
  - Swap High Resolution - When enabled, swaps higher resolution files between whitelist and blacklist/graylist files.
  - Swap Longer Duration - When enabled, swaps scene with longer duration.
- Options available via DupFileManager_config.py
  - dup_path - Alternate path to move deleted files to. Example: "C:\TempDeleteFolder"
  - toRecycleBeforeSwap - When enabled, moves destination file to recycle bin before swapping files.
  - addPrimaryDupPathToDetails - If enabled, adds the primary duplicate path to the scene detail.
Tools UI Menu
- Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.

Requirements

pip install --upgrade stashapp-tools
pip install requests
pip install Send2Trash

Installation

Follow Requirements instructions.
In the stash plugin directory (C:\Users\MyUserName.stash\plugins), create a folder named DupFileManager.
Copy all the plugin files to this folder.(C:\Users\MyUserName\.stash\plugins\DupFileManager).
Click the [Reload Plugins] button in Stash->Settings->Plugins->Plugins.

That's it!!!

Options

Options are accessible in the GUI via Stash->Settings->Plugins->Plugins->[DupFileManager].
Also see:
- Stash->Settings->Tools->[Duplicate File Report]
- Stash->Settings->Tools->[DupFileManager Tools and Utilities]
More options available on the following link:
- advance_options.html
- When using a Stash installation that requires a password or that is not using port 9999...
  - Access above link from Stash->Settings->Tools->[Duplicate File Report]->[Advance Duplicate File Menu]
  - Or add the GQL and apiKey as parameters to the URL.
    - Example: https://stash.axter.com/1.1/advance_options.html?GQL=http://localhost:9999/graphql&apiKey=1234567890abcdefghijklmnop
  - See following for more details: Stash Password

Advanced Options

Users can setup a private or alternate remote site by changing variables remoteReportDirURL and js_DirURL in file DupFileManager_report_config.py.

The following files are needed at the remote site that is pointed to by remoteReportDirURL.
- DupFileManager_report.js
- DupFileManager_report.css
- file.html
- advance_options.html
The js_DirURL path requires the following:
- jquery-3.7.1.min.js
- EasyUI associated files
- jquery.prompt.js and jquery.prompt.css

Stash Password

Stash installation configured with a password, need to generate an API-Key.
- To generate an API-Key:
  - Go to Stash->Settings->Security->Authentication->[API Key]
  - Click on [Generate API-Key]
Once the API key is generated, DupFileManager will automatically fetch the key.

Screenshots

Example DupFileManager duplicate report. (file names have been edited to PG).
- The report displays preview videos that are playable. Will play a few seconds sample of the video. This requires scan setting [Generate animated image previews] to be enabled when scanning all files.
- If there's a scene on the left side that has a higher resolution or duration, it gets a yellow highlight on the report.
- There's an optional setting that allows both preview videos and preview images to be displayed on the report. See settings htmlIncludeImagePreview in the DupFileManager_report_config.py file.
- There are many more options available for how the report is created. These options are targeted for more advanced users. The options are all available in the DupFileManager_report_config.py file, and the settings have commented descriptions preceding them. See the DupFileManager_report_config.py file in the DupFileManager plugin folder for more details.
Tools UI Menu
- Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.
DupFileManager Report Menu
DupFileManager Tools and Utilities
Full bottom extended portion of the Advanced Menu screen.

Future Planned Features, Changes, or Fixes

Scheduled Changes
- Remove [Max Dup Process] from the Stash->Plugins GUI. This option already exist in advance menu. Planned for 1.2.0 Version.
- Add chat icon to report which on hover, displays a popup window showing scene details content. Planned for 1.2.0 Version.
- Add studio icon to report; on hover show studio name. Planned for 1.2.0 Version.
- Add option on report to view rating and change it. Use an icon with a number on it to show rating. Planned for 1.2.0 Version.
- On report make flag toggle logic. In other words, when flag button is selected, and scene is already that color, remove the colored flag. Planned for 1.2.0 Version.
- Add report option to set flag to all scenes missing metadata on Scene-ToKeep or having higher resolution or longer duration on Duplicate-to-Delete scene. Planned for 1.2.0 Version.
- Add option to [Advance Duplicate File Menu] to display cover image for preview image. Planned for 1.2.0 Version.
- On report, when move operation is performed, have it automatically delete the Duplicate-to-Delete scene. Planned for 1.2.0 Version.
- On [Advance Duplicate File Menu], add a flag option to the menu on the left side, and shrink the tag option. Planned for 1.2.0 Version.
  - Change the TagOrFlag option on the right side to a check mark.
- Add option to delete by file type. Planned for 1.2.0 Version.
- On report, have [Flag or Tag] option still enabled after manually disabling scene. Planned for 1.2.0 Version.
- On [Advance Duplicate File Menu], have the details section hide fully, or partially when the page is first displayed. Planned for 1.4.0 Version.
Unscheduled Changes
- Change splitbutton so it hides tooltip when hovering over dropdown side of the button.
- Fix errors on HTML page listed in https://validator.w3.org.
- Add logic to merge performers and galleries separately from tag merging on report.
- Add logic to merge group metadata when selecting merge option on report.
- Add advanced menu directly to the Stash->Settings->Tools menu. (This change does not look doable!!!)
- Add report directly to the Stash->Settings->Tools menu. (This change does not look doable!!!)
- Create cookies for the options in the [Advance Duplicate File Menu].
- Add double strike-through option to flagging.
- Add option to report to automatically strip width & height from name on rename. Example: foofoo - 1920x1080P.mp4
- Move [Merge Duplicate Tags], [Whitelist Delete In Same Folder], and [Swap Better **] field options from the Stash->Plugins GUI to the advance menu.

11 KiB Raw Blame History