Skip to content

poller(ios/iosxe): anchor prompt on hostname at connect to avoid truncated output - #1011

Open
igorseixashr wants to merge 4 commits into
netenglabs:developfrom
igorseixashr:fix/ios-prompt-anchor-hostname
Open

igorseixashr wants to merge 4 commits into
netenglabs:developfrom
igorseixashr:fix/ios-prompt-anchor-hostname

Conversation

@igorseixashr

Copy link
Copy Markdown

Related Issue

No existing issue found for this (searched issues/PRs for Device#, IOS prompt truncation).

Description

For IOS/IOS-XE the persistent SSH session waits for a bare # or > until show version has been parsed and self.prompt becomes <hostname>#/<hostname>>.

On IOS 15.x (seen on CISCO1905 with 15.2(4)M3 and 15.7(3)M9) the license table in show version contains a line starting with Device#:

Device#   PID                   SN
-------------------------------------------------
*0        CISCO1905BR-SEC/K9    XXXXXXXXXXX

readuntil(('>', '#')) stops at that line, so the captured output starts at \t PID\t\t\tSN. Hostname and uptime are never parsed (Cannot parse uptime), the prompt stays bare, and every following command returns the tail of the previous command's output.

Symptom: the device, interfaces and bgp tables stay empty for these nodes while sqPoller reports OK for every service.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

New Behavior

Right after the interactive session is opened in IosXENode._ssh_connect(), the hostname is taken from the login prompt (last line, ending in # or >, without spaces) and self.prompt is anchored on it immediately, before privilege escalation and terminal length 0.

Contrast to Current Behavior

Today the prompt is only anchored after show version is parsed, which is exactly the command that gets truncated on these platforms. With the change the prompt is anchored before any command runs.

Discussion: Benefits and Drawbacks

  • Fixes silent data loss (empty tables with OK status) on IOS 15.x routers whose show version contains Device#.
  • Backwards-compatible: if the login prompt can't be read, self.prompt keeps the default ('>', '#') and the current flow is unchanged. The later anchoring from show version still runs.
  • Only IosXENode / IOSNode are touched; other NOS use different connect paths.

Testing

Tested with sq-poller --run-once=process against real devices, on 0.25.0 (same hunk as develop at 75198b5):

Device OS Before After
CISCO1905BR-SEC/K9 IOS 15.2(4)M3 device 0 rows, interfaces 0, bgp 0 device 1, interfaces 13, bgp 3
CISCO1905BR-SEC/K9 IOS 15.7(3)M9 device 0 rows device OK
WS-C2960X IOS 15.2(2)E6 / 15.2(7)E13 OK OK (no regression)
C9300-24S IOS-XE 17.15.3 OK OK (no regression)

No unit test added: this path needs a live interactive SSH session and I couldn't find existing tests covering _ssh_connect(). Happy to add one if you point me to a preferred pattern.

Changes to the Documentation

None.

Proposed Release Note Entry

IOS/IOS-XE: anchor the CLI prompt on the device hostname at connect time, fixing truncated show version and empty device/interfaces/bgp data on IOS 15.x routers whose output contains Device#.

On IOS 15.x the license table in `show version` contains a line
starting with 'Device#'. Until show version is parsed the persistent
SSH session waits for a bare '#'/'>' prompt, so the output is cut at
that line, hostname/uptime are never parsed and every following
command returns the tail of the previous one (device/interfaces/bgp
end up empty while sqPoller reports OK).

Take the hostname from the login prompt right after the interactive
session opens and anchor self.prompt on it. Behaviour is unchanged
when the prompt cannot be read.

Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
Login banners with '#'/'>' (ASCII art) made the first readuntil stop
inside the banner, so the hostname-anchored prompt was never found.
Read until the device goes quiet (2s idle), then take the hostname
from the last line.

Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
@igorseixashr

Copy link
Copy Markdown
Author

Pushed a follow-up commit: some devices have ASCII-art login banners containing many # characters, so the first readuntil(('>', '#')) returned inside the banner and the real hostname# prompt was never seen. The connect path now drains the banner until the device goes quiet (2s idle, bounded) and then anchors the prompt on the last line.

Re-tested with --run-once=process: two CISCO1905 (15.2(4)M3) with such a banner went from empty device/interfaces to 1 device row and 15 interfaces each; no regression on 1905 without banner art, 2960X (15.2(2)E6, 15.2(7)E13) or C9300 (17.15.3).

…opped

Some NOS (seen on NX-OS 7.3(13)N1(1)) silently drop an idle SSH
connection ~40s after the last channel closes. SuzieQ keeps the
connection object and the first command of the next poll fails with
'SSH connection closed', so the service (often 'device') is lost for
that cycle and the node shows as dead.

On ChannelOpenError/ConnectionLost/DisconnectError (and broken pipe /
reset), close the connection, reconnect without re-running the init
device data, and retry the command once.

Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
@igorseixashr

Copy link
Copy Markdown
Author

Added a third commit, related but broader than the IOS prompt fix: reconnect-and-retry in the base _ssh_gather when the persistent SSH connection was dropped by the device.

Seen on NX-OS 7.3(13)N1(1): the device closes an idle SSH connection ~40s after the last channel closes, and the first command of the next poll fails with SSH connection closed (the node shows as dead for that cycle). With the change, 4 such switches polled every 5 min logged 4 reconnects and device succeeded in 19/20 cycles (previously failing from the 2nd cycle on).

Happy to split this into a separate PR if you prefer to review it independently.

Several services share the node connection. When one of them hit the
dropped connection and closed it, the others saw self._conn set to
None and failed with "'NoneType' object has no attribute 'run'".
Reconnect when self._conn is None and close only the connection
object that actually failed.

Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant