[BUG] Duplicate VMRs with different hash cause "file already in flight" error

[sdelliot]

Bug Report

Description

When FIREWHEEL/minimega is running in a cluster and there are duplicate (same name) VM resources but have a different hash, minimega will throw a "file already in flight". This is because FIREWHEEL detects the differing hashes and tries to upload the most recently modified file. This works great when continually updating VM resources within a model component, but it starts to fail if there are unintentionally VM resources with the same name (and different content) across an experiment.

Steps to Reproduce

0. Run a cluster of FIREWHEEL nodes.
1. Add a new VM resources called test.txt with a random string in the tests.vm_gen model component.
2. Modify the tests.vm_gen MANIFEST file to recognize this new VMR.
3. Add a new VM resources called test.txt with a different random string in the tests.connect_all model component.
4. Modify the tests.connect_all MANIFEST file to recognize this new VMR.
5. Run the experiment firewheel experiment tests.vm_gen:6 tests.connect_all:1 minimega.launch
6. The error will display to the user and also show up in firewheel.log.

Expected Behavior

I think the expecte4d behavior is debatable. At a minimum, a more clear error message should be displayed to the user. Alternatively, this could some how be handled and a warning could be shown to the user with the assumption/decision that FIREWHEEL made in handling the error.

Actual Behavior

A very odd error is displayed.

Screenshots

[2025-12-16 21:27:59 UTC DEBUG ModelComponentManager] Processing model component tests.connect_all
[2025-12-16 21:27:59 UTC DEBUG ModelComponentManager] Checking model component objects for model component tests.connect_all
[2025-12-16 21:27:59 UTC DEBUG ModelComponentManager] Checking plugin objects for model component tests.connect_all at path "plugin.py"
[2025-12-16 21:27:59 UTC DEBUG ModelComponentManager] Loading plugin for entity ('tests.connect_all',)
[2025-12-16 21:27:59 UTC DEBUG ModelComponentManager] Checking for plugin in ('/opt/firewheel/model_components/firewheel_repo_base/src/firewheel_repo_base/tests/connect_all/plugin.py',)
[2025-12-16 21:27:59 UTC DEBUG ModelComponentManager] Found plugin class Plugin
[2025-12-16 21:27:59 UTC DEBUG ModelComponent] Uploading resource test.txt from model component tests.connect_all
[2025-12-16 21:27:59 UTC DEBUG ModelComponent] Resource test.txt in tests.connect_all has modified time of 2025-12-16 21:26:06.465897
[2025-12-16 21:27:59 UTC DEBUG FileStore] in get_file_upload date with filename=test.txt and host_file_path=/scratch/minimega/files/vm_resources/test.txt
[2025-12-16 21:27:59 UTC DEBUG FileStore] basename test.txt  has upload time of 2025-12-16 21:25:49.273971
[2025-12-16 21:27:59 UTC DEBUG ModelComponent] VM Resource store file has upload date of 2025-12-16 21:25:49.273971
[2025-12-16 21:27:59 UTC DEBUG ModelComponent] Resource on disk is different from store. Checksuming.
[2025-12-16 21:27:59 UTC DEBUG ModelComponent] Resource tests.connect_all//test.txt on disk has hash 8334b2a59b2e467517a2bbcc9396a0e243df2882 and in store has 9e736902dbbbee7cb3390c9626d2a83520988486
[2025-12-16 21:27:59 UTC DEBUG ModelComponent] Newer resource checksum differs. Uploading. 
[2025-12-16 21:27:59 UTC INFO FileStore] in add_file with source_dir=/opt/firewheel/model_components/firewheel_repo_base/src/firewheel_repo_base/tests/connect_all and filename=test.txt, basename=test.txt, from path=/opt/firewheel/model_components/firewheel_repo_base/src/firewheel_repo_base/tests/connect_all/test.txt
[2025-12-16 21:27:59 UTC ERROR FileStore] Adding test.txt to vm_resources at vm_resources/test.txt
[2025-12-16 21:27:59 UTC ERROR FileStore] file already in flight 
Traceback (most recent call last): 
  File "/opt/firewheel/src/firewheel/lib/minimega/file_store.py", line 891, in add_file
    self.mm_api.mm.mesh_send("all", f"file get {mm_file_path}")
  File "/opt/firewheel/fwpy/lib/python3.10/site-packages/minimega.py", line 7070, in mesh_send
    return self._run("mesh", "send", hostname, command)
  File "/opt/firewheel/fwpy/lib/python3.10/site-packages/minimega.py", line 176, in _run
    response = self._get_response()
  File "/opt/firewheel/fwpy/lib/python3.10/site-packages/minimega.py", line 149, in _get_response
    raise Error(resp['Error'])
minimega.Error: file already in flight
[2025-12-16 21:27:59 UTC INFO FirewheelCLI] Command returned: 1

Environment

• Operating System: Ubuntu
• Version: main
• Any Model Components Impacted: N/A

Checklist

• I have included no proprietary/sensitive information in my issue.