Axter-Stash/plugins/DupFileManager

DupFileManager: Ver 1.1.4 (By David Maisonave)

DupFileManager is a Stash plugin which manages duplicate files in the Stash system. It has both task and tools-UI components.

Features

• Creates a duplicate file report which can be accessed from the Stash->Settings->Tools menu options.The report is created as an HTML file and stored in local path under plugins\DupFileManager\report\DuplicateTagScenes.html.
  • See screenshot at the bottom of this page for example report.
  • Items on the left side of the report are the primary duplicates designated for deletion. By default, these duplicates are given a special _duplicate tag.
  • Items on the right side of the report are designated as primary duplicates to keep. They usually have higher resolution, duration and/or preferred paths.
  • The report has the following options:
    • Delete: Delete file and remove from Stash library.
    • Remove: Remove from Stash library.
    • Rename: Rename file.
    • Copy: Copy file from left (source) to right (to-keep).
    • Move: Copy file and metadata left to right.
    • Cpy-Name: Copy file name left to right.
    • Add-Exclude: Add exclude tag to scene,so that scene is excluded from deletion.
    • Remove-Tag: Remove duplicate tag from scene.
    • Flag-Scene: Flag (mark) scene in report as reviewed (or as requiring further review). Optional flags (yellow, green, orange, cyan, pink, red, strike-through, & disable-scene)
    • Merge: Copy Metadata (tags, performers,& studios) from left to right.
• Can merge potential source in the duplicate file names for tag names, performers, and studios.
  • Normally when Stash searches the file name for tag names, performers, and studios, it only does so using the primary file.
• Advance menu
  • 
  • Advance menu can be access from the Stash->Settings->Tools->[Duplicate File Report] menu or from the DupFileManager Tools & Util.
  • Here are some of the options available in the Advance Menu.
    • Delete specially tagged duplicates in blacklist path.
    • Delete duplicates with specified file path.
    • Delete duplicates with specific string in File name.
    • Delete duplicates with specified file size range.
    • Delete with specified duration range.
    • Delete with resolution range.
    • Delete duplicates having specified tags.
    • Delete duplicates with specified rating.
    • Delete duplicates with any of the above combinations.
• Bottom extended portion of the Advanced Menu is for customizing the report.
  • 
• Delete duplicate file task with the following options:
  • Tasks (Stash->Settings->Task->[Plugin Tasks]->DupFileManager)
    • Tag Duplicates - Set tag DuplicateMarkForDeletion to the duplicates with lower resolution, duration, file name length, and/or black list path.
    • Delete Tagged Duplicates - Delete scenes having DuplicateMarkForDeletion tag.
    • Delete Duplicates - Deletes duplicate files. Performs deletion without first tagging.
  • Plugin UI options (Stash->Settings->Plugins->Plugins->[DupFileManager])
    • Has a 3 tier path selection to determine which duplicates to keep, and which should be candidates for deletions.
      • Whitelist - List of paths NOT to be deleted.
        • E.g. C:\Favorite,E:\MustKeep\
      • Gray-List - List of preferential paths to determine which duplicate should be the primary.
        • E.g. C:\2nd_Favorite,H:\ShouldKeep\
      • Blacklist - List of LEAST preferential paths to determine primary candidates for deletion.
        • E.g. C:\Downloads,F:\DeleteMeFirst\
    • Permanent Delete - Enable to permanently delete files, instead of moving files to trash can.
    • Max Dup Process - Use to limit the maximum files to process. Can be used to do a limited test run.
    • Merge Duplicate Tags - Before deletion, merge metadata from duplicate. E.g. Tag names, performers, studios, title, galleries, rating, details, etc...
    • Swap High Resolution - When enabled, swaps higher resolution files between whitelist and blacklist/graylist files.
    • Swap Longer Duration - When enabled, swaps scene with longer duration.
  • Options available via DupFileManager_config.py
    • dup_path - Alternate path to move deleted files to. Example: "C:\TempDeleteFolder"
    • toRecycleBeforeSwap - When enabled, moves destination file to recycle bin before swapping files.
    • addPrimaryDupPathToDetails - If enabled, adds the primary duplicate path to the scene detail.
• Tools UI Menu
  • Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.

Requirements

• pip install --upgrade stashapp-tools
• pip install requests
• pip install Send2Trash

Installation

• Follow Requirements instructions.
• In the stash plugin directory (C:\Users\MyUserName.stash\plugins), create a folder named DupFileManager.
• Copy all the plugin files to this folder.(C:\Users\MyUserName\.stash\plugins\DupFileManager).
• Click the [Reload Plugins] button in Stash->Settings->Plugins->Plugins.

That's it!!!

Options

• Options are accessible in the GUI via Stash->Settings->Plugins->Plugins->[DupFileManager].
• Also see:
  • Stash->Settings->Tools->[Duplicate File Report]
  • Stash->Settings->Tools->[DupFileManager Tools and Utilities]
• More options available on the following link:
  • advance_options.html
  • When using a Stash installation that requires a password or that is not using port 9999...
    • Access above link from Stash->Settings->Tools->[Duplicate File Report]->[Advance Duplicate File Menu]
    • Or add the GQL and apiKey as parameters to the URL.
      • Example: https://stash.axter.com/1.1/advance_options.html?GQL=http://localhost:9999/graphql&apiKey=1234567890abcdefghijklmnop
    • See following for more details: Stash Password

Advanced Options

Users can setup a private or alternate remote site by changing variables remoteReportDirURL and js_DirURL in file DupFileManager_report_config.py.

• The following files are needed at the remote site that is pointed to by remoteReportDirURL.
  • DupFileManager_report.js
  • DupFileManager_report.css
  • file.html
  • advance_options.html
• The js_DirURL path requires the following:
  • jquery-3.7.1.min.js
  • EasyUI associated files
  • jquery.prompt.js and jquery.prompt.css

Stash Password

• Stash installation configured with a password, need to generate an API-Key.
  • To generate an API-Key:
    • Go to Stash->Settings->Security->Authentication->[API Key]
    • Click on [Generate API-Key]
• Once the API key is generated, DupFileManager will automatically fetch the key.

Screenshots

• Example DupFileManager duplicate report. (file names have been edited to PG).
  • The report displays preview videos that are playable. Will play a few seconds sample of the video. This requires scan setting [Generate animated image previews] to be enabled when scanning all files.
  • 
  • If there's a scene on the left side that has a higher resolution or duration, it gets a yellow highlight on the report.
  • There's an optional setting that allows both preview videos and preview images to be displayed on the report. See settings htmlIncludeImagePreview in the DupFileManager_report_config.py file.
  • There are many more options available for how the report is created. These options are targeted for more advanced users. The options are all available in the DupFileManager_report_config.py file, and the settings have commented descriptions preceeding them. See the DupFileManager_report_config.py file in the DupFileManager plugin folder for more details.
• Tools UI Menu
  • Can access either Duplicate File Report (DupFileManager) or DupFileManager Tools and Utilities menu options.
• DupFileManager Report Menu
  • 
• DupFileManager Tools and Utilities
  • 
• Full bottom extended portion of the Advanced Menu screen.
  • 

Future Planned Features, Changes, or Fixes

• Scheduled Changes
  • Add option to report to avoid reloading page after updating report. Planned for 1.2.0 Version.
  • After deleting scene from report, disable preview for the deleted scene on the report. Planned for 1.2.0 Version.
  • Remove [Max Dup Process] from the Stash->Plugins GUI. This option already exist in advance menu. Planned for 1.2.0 Version.
  • Add chat icon to report which on hover, displays a popup window showing scene details content. Planned for 1.2.0 Version.
  • Add image icon to report; on hover show scene cover image. Planned for 1.2.0 Version.
  • Add studio icon to report; on hover show studio name. Planned for 1.2.0 Version.
  • Add option on report to view rating and change it. Use an icon with a number on it to show rating. Planned for 1.2.0 Version.
  • On report make flag toggle logic. In other words, when flag button is selected, and scene is already that color, remove the colored flag. Planned for 1.2.0 Version.
  • Add report option to set flag to all scenes missing metadata on Scene-ToKeep or having higher resolution or longer duration on Duplicate-to-Delete scene. Planned for 1.2.0 Version.
  • Add option to [Advance Duplicate File Menu] to display cover image for preview image. Planned for 1.2.0 Version.
  • On report, when move operation is performed, have it automatically delete the Duplicate-to-Delete scene. Planned for 1.2.0 Version.
  • On [Advance Duplicate File Menu], add a flag option to the menu on the left side, and shrink the tag option. Planned for 1.2.0 Version.
    • Change the TagOrFlag option on the right side to a check mark.
  • Add option to delete by file type. Planned for 1.2.0 Version.
  • On report, have [Flag or Tag] option still enabled after manually disabling scene. Planned for 1.2.0 Version.
  • On [Advance Duplicate File Menu], have the details section hide fully, or partially when the page is first displayed. Planned for 1.4.0 Version.
• Unscheduled Changes
  • Change splitbutton so it hides tooltip when hovering over dropdown side of the button.
  • Fix errors on HTML page listed in https://validator.w3.org.
  • Add logic to merge performers and galaries seperatly from tag merging on report.
  • Add logic to merge group metadata when selecting merge option on report.
  • Add advanced menu directly to the Stash->Settings->Tools menu. (This change does not look doable!!!)
  • Add report directly to the Stash->Settings->Tools menu. (This change does not look doable!!!)
  • Create cookies for the options in the [Advance Duplicate File Menu].
  • Add doulbe strike-through option to flagging.
  • Add option to report to automatically strip width & height from name on rename. Example: foofoo - 1920x1080P.mp4
  • Add link to version history to [Advance Duplicate File Menu] and to [DupFileManager Tools and Utilities]
  • Move [Merge Duplicate Tags], [Whitelist Delete In Same Folder], and [Swap Better **] field options from the Stash->Plugins GUI to the advance menu.