engine: do not stall the host command queue on a failed send in Agent…

[nagaboinaramgopal]

Description

When sending the next queued request threw AgentUnavailableException, the request
was cancelled but _currentSequence was still set to that dead sequence. No answer
ever arrives for a cancelled command, so sendNext was never driven again and every
later in-sequence command to the host queued behind it and timed out, until the
attache was rebuilt.

Advance _currentSequence only on a successful send, and on failure move on to the
next queued request.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• Minor

How Has This Been Tested?

Added a unit test where the first queued command fails to send and asserts the
next queued command is dispatched and becomes the current sequence, instead of
stalling on the failed one. Also built the standard packages and deployed on a KVM
advanced zone.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 19.78%. Comparing base (2cd8c5e) to head (dab7711).

Files with missing lines	Patch %	Lines
...ain/java/com/cloud/agent/manager/AgentAttache.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #14040      +/-   ##
============================================
+ Coverage     16.34%   19.78%   +3.43%     
- Complexity    13574    19993    +6419     
============================================
  Files          5669     6371     +702     
  Lines        501368   575900   +74532     
  Branches      60903    70495    +9592     
============================================
+ Hits          81964   113953   +31989     
- Misses       410219   449513   +39294     
- Partials       9185    12434    +3249     

Flag	Coverage Δ
uitests	3.53% <ø> (-0.62%)	⬇️
unittests	21.06% <0.00%> (+3.85%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 19095

[DaanHoogland]

@nagaboinaramgopal , an compile error in your test code:

09:17:39 [ERROR] /jenkins/workspace/acs-centos8-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.4.0-SNAPSHOT/engine/orchestration/src/test/java/com/cloud/agent/manager/AgentAttacheSendNextTest.java:[38,12] error: constructor AgentAttache in class AgentAttache cannot be applied to given types;
09:17:39   required: AgentManagerImpl,long,String,String,boolean
09:17:39   found: <null>,long,String,String,<null>,boolean
09:17:39   reason: actual and formal argument lists differ in length
09:17:39 [INFO] 1 error

can you have a look?

[nagaboinaramgopal]

@nagaboinaramgopal , an compile error in your test code:

09:17:39 [ERROR] /jenkins/workspace/acs-centos8-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.4.0-SNAPSHOT/engine/orchestration/src/test/java/com/cloud/agent/manager/AgentAttacheSendNextTest.java:[38,12] error: constructor AgentAttache in class AgentAttache cannot be applied to given types;
09:17:39   required: AgentManagerImpl,long,String,String,boolean
09:17:39   found: <null>,long,String,String,<null>,boolean
09:17:39   reason: actual and formal argument lists differ in length
09:17:39 [INFO] 1 error

can you have a look?

fixed and pushed, Thanks Daan