Batched device memcpy for CUDA/HIP

[devreal]

Since CUDA 12.8 we can use batched memory copy to perform multiple operations in a single CUDA call. For tasks with multiple stage in/out flows this can be beneficial. Prominent example
are eviction tasks. There is no reason to order pushouts of a single task.

In practice, I am observing higher bandwidth for small batched data (9x2GB/s vs 1x12GB/s for 9 8KB chunks).

[Copilot]

Pull request overview

This PR introduces a new multi-item GPU memcpy interface and updates the default GPU stage-in/stage-out paths to batch multiple transfers into a single backend call when possible (notably leveraging CUDA 12.8’s cudaMemcpyBatchAsync) to improve bandwidth for workloads with multiple in/out flows (e.g., evictions).

Changes:

• Adds a new memcpy_multi_async function pointer to the GPU device module interface, with a generic fallback that issues multiple memcpy_async calls.
• Updates default GPU stage-in/stage-out logic to aggregate per-flow copies and issue one combined multi-copy operation.
• Implements a CUDA 12.8+ native backend using cudaMemcpyBatchAsync, and wires Level Zero to the generic fallback.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
parsec/mca/device/level_zero/device_level_zero_module.c	Wires Level Zero GPU device to use the generic multi-memcpy fallback.
parsec/mca/device/device_gpu.h	Adds the memcpy_multi_async API + documents semantics and declares the generic fallback.
parsec/mca/device/device_gpu.c	Implements the generic fallback and updates default stage-in/stage-out and push/pop paths to batch transfers.
parsec/mca/device/cuda/device_cuda_module.c	Adds CUDA 12.8+ native batched memcpy implementation and selects it at init time.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bosilca]

This is a good idea, but the scope of the current implementation is narrow and only benefit to tasks with small data and only if the copy latency is important, which means for tasks that cannot keep the GPU busy (aka. not compute intensive). It also work per task, so at best it batches MAX_PARAM items.

If we can merge this with the task batching we could have a much stronger support for small compute-trivial tasks.

[bosilca]

This loop is too carefree. I know that if any error is returned everything will fail, but if we try to expose the faults at least we should do it properly. Here if anything fails we return the error but we continue to submit all the others copies. However, once we return from this function with an error we call parsec_device_kernel_push_release_readers_on_failure directly, releasing the readers for the entire mask (and that's one problem), and then return without enquing an event (which allow the data use right away on another stream).

The interface needs either atomic submission semantics, a returned submitted mask, or the old per-flow path for backends without native batching.

[devreal]

Bailing out on error now. It's hard to reliably report which operation failed though (the new CUDA API doesn't report it anymore). I'm not sure it's important to know that. Repeating all transfers would do no harm in terms of correctness.

[bosilca]

releases a GPU reader without locking its original or coordinating with the source GPU manager. When this removes the last reader, parsec_gpu_data_copy_release_reader() can relink the copy into a source-device LRU.
The normal D2D completion path serializes with the source manager, locks the original, and updates data_avail_epoch. The failure path should reuse that protocol.

[devreal]

Factored out the d2d release and used it here. Please recheck

[devreal]

If we can merge this with the task batching we could have a much stronger support for small compute-trivial tasks.

I'm torn on this. I agree that batching memcpy for batched tasks could be helpful. However, for tasks with large inputs we don't gain much (since a single transfer can saturate the bandwidth) already and in fact may harm performance because the tasks are delayed collectively waiting for larger transfers.

For small transfers and kernels this would be helpful, although we probably want to cap the maximum number of batched transfers (for the same reason as above).

I would suggest we take what we have here as a first step and explore more batching later. Small kernel are a problem that is hitting us in MADNESS so it is something I'm looking at. If we can benefit from task-level batched inputs I will explore that further.

[devreal]

One way this would be helpful for us is if we could transfer the ring of batched tasks from execute to pushout and batch their outputs. Not sure if #792 covers that.

Alternatively, we could just collect a bounded number of PARSEC_DEV_CHORE_ALLOW_BATCH tasks in the pushout and combine them that way. That should be easy to do but runs the risk of delaying successor releases because of large transfers.