Find new cuopt testing environment or consider deprecation?

[jsiirola]

Summary

Maintaining a working cuopt installation in our testing infrastructure has become increasing difficult, and is beginning to exceed our limited development/support/infrastructure time. We should consider an alternative approach / environment for testing cuopt, or consider deprecating the solver interface.

Steps to reproduce the issue

Core error: run the tests on the Jenkins build host (pytest pyomo/solvers)

Install error: pip install cuopt-cu12

Error Message

A recent server upgrade led to cuopt failing with:

Error Details

RuntimeError: CUDA error at: /__w/rmm/rmm/cpp/src/cuda_stream.cpp:20: cudaErrorSystemDriverMismatch system has unsupported display driver / cuda driver combination

Stack Trace

self = <pyomo.solvers.tests.checks.test_no_solution_behavior.Test_LP_infeasible1_kernel testMethod=test_cuopt_python>

    def return_test(self):
>       return failed_solve_test(self)
               ^^^^^^^^^^^^^^^^^^^^^^^

pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:90: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:59: in failed_solve_test
    opt, results = model_class.solve(
pyomo/pyomo/solvers/tests/models/base.py:121: in solve
    results = opt.solve(
pyomo/pyomo/solvers/plugins/solvers/direct_solver.py:140: in solve
    _status = self._apply_solver()
pyomo/pyomo/solvers/plugins/solvers/cuopt_direct.py:73: in _apply_solver
    self.solution = cuopt.linear_programming.solver.Solve(
[...]/lib/python3.11/site-packages/cuopt/utilities/exception_handler.py:48: in func
    raise e
[...]/lib/python3.11/site-packages/cuopt/utilities/exception_handler.py:24: in func
    return f(*args, **kwargs)
[...]/lib/python3.11/site-packages/cuopt/linear_programming/solver/solver.py:98: in Solve
    s = solver_wrapper.Solve(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   RuntimeError: CUDA error at: /__w/rmm/rmm/cpp/src/cuda_stream.cpp:20: cudaErrorSystemDriverMismatch system has unsupported display driver / cuda driver combination

cuopt/linear_programming/solver/solver_wrapper.pyx:524: RuntimeError

Further, attempts to pip install cuopt-cu12 also fail:

Collecting cuopt-cu12
  Downloading cuopt_cu12-26.8.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (1.4 kB)
Collecting cudf-cu12==26.8.* (from cuopt-cu12)
  Downloading cudf_cu12-26.8.0-cp311-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (8.0 kB)
Collecting cupy-cuda12x!=14.1.0,>=14.0.1 (from cupy-cuda12x[ctk]!=14.1.0,>=14.0.1->cuopt-cu12)
  Downloading cupy_cuda12x-14.1.1-cp311-cp311-manylinux2014_x86_64.whl.metadata (2.8 kB)
Collecting libcuopt-cu12==26.8.* (from cuopt-cu12)
Collecting libcuopt-cu12==26.8.* (from cuopt-cu12)
  Downloading libcuopt_cu12-26.8.0.tar.gz (4.3 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [97 lines of output]
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 249, in download_wheel
          return download_manual(wheel_directory, distribution, version, config)
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 176, in download_manual
          raise RuntimeError(f"Failed to open project URL {project_url}") from e

      Traceback (most recent call last):
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 174, in download_manual
          index_response = urlopen_with_retry(project_url)
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 92, in urlopen_with_retry
          return urlopen(url, **kwargs)
        File "[...]/lib/python3.11/urllib/request.py", line 216, in urlopen
          return opener.open(url, data, timeout)
        File "[...]/lib/python3.11/urllib/request.py", line 525, in open
          response = meth(req, response)
        File "[...]/lib/python3.11/urllib/request.py", line 634, in http_response
          response = self.parent.error(
        File "[...]/lib/python3.11/urllib/request.py", line 557, in error
          result = self._call_chain(*args)
        File "[...]/lib/python3.11/urllib/request.py", line 496, in _call_chain
          result = func(*args)
        File "[...]/lib/python3.11/urllib/request.py", line 749, in http_error_302
          return self.parent.open(new, timeout=req.timeout)
        File "[...]/lib/python3.11/urllib/request.py", line 525, in open
          response = meth(req, response)
        File "[...]/lib/python3.11/urllib/request.py", line 634, in http_response
          response = self.parent.error(
        File "[...]/lib/python3.11/urllib/request.py", line 563, in error
          return self._call_chain(*args)
        File "[...]/lib/python3.11/urllib/request.py", line 496, in _call_chain
          result = func(*args)
        File "[...]/lib/python3.11/urllib/request.py", line 643, in http_error_default
          raise HTTPError(req.full_url, code, msg, hdrs, fp)
      urllib.error.HTTPError: HTTP Error 403: Forbidden

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 249, in download_wheel
          return download_manual(wheel_directory, distribution, version, config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 176, in download_manual
          raise RuntimeError(f"Failed to open project URL {project_url}") from e
      RuntimeError: Failed to open project URL https://pypi.nvidia.com/libcuopt-cu12/

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "[...]/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
          main()
        File "[...]/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
          json_out["return_val"] = hook(**hook_input["kwargs"])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "[...]/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 178, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/buildapi.py", line 29, in build_wheel
          return download_wheel(pathlib.Path(wheel_directory), config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/wheel.py", line 251, in download_wheel
          report_install_failure(distribution, version, config, exception_context)
        File "/tmp/pip-build-env-c5m4vhbi/overlay/lib/python3.11/site-packages/wheel_stub/error.py", line 67, in report_install_failure
          raise InstallFailedError(
      wheel_stub.error.InstallFailedError:
      *******************************************************************************

      The installation of libcuopt-cu12 for version 26.8.0 failed.

      This is a special placeholder package which downloads a real wheel package
      from https://pypi.nvidia.com/. If https://pypi.nvidia.com/ is not reachable, we
      cannot download the real wheel file to install.

      You might try installing this package via
      ```
      $ pip install --extra-index-url https://pypi.nvidia.com/ libcuopt-cu12
      ```

      Here is some debug information about your platform to include in any bug
      report:

      Python Version: CPython 3.11.14
      Operating System: Linux 5.14.0-687.22.1.el9_8.x86_64
      CPU Architecture: x86_64
      Error in running nvidia-smi:
      *******************************************************************************

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 26.1 -> 26.2.1
[notice] To update, run: python -m pip install --upgrade pip
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> libcuopt-cu12

note: This is an issue with the package mentioned above, not pip.

Information on your system

Pyomo version: 6.10.2.dev0
Python version: 3.11.14
Operating system: Linux
How Pyomo was installed (PyPI, conda, source): source
Solver (if applicable): cuopt

Additional information

[jsiirola]

We have currently disabled local cuopt testing in an effort to clear the current PR backlog (see #4002).