Repository navigation
poller(ios/iosxe): anchor prompt on hostname at connect to avoid truncated output - #1011
igorseixashr wants to merge 4 commits into
Conversation
On IOS 15.x the license table in `show version` contains a line starting with 'Device#'. Until show version is parsed the persistent SSH session waits for a bare '#'/'>' prompt, so the output is cut at that line, hostname/uptime are never parsed and every following command returns the tail of the previous one (device/interfaces/bgp end up empty while sqPoller reports OK). Take the hostname from the login prompt right after the interactive session opens and anchor self.prompt on it. Behaviour is unchanged when the prompt cannot be read. Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
Login banners with '#'/'>' (ASCII art) made the first readuntil stop inside the banner, so the hostname-anchored prompt was never found. Read until the device goes quiet (2s idle), then take the hostname from the last line. Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
|
Pushed a follow-up commit: some devices have ASCII-art login banners containing many Re-tested with |
…opped Some NOS (seen on NX-OS 7.3(13)N1(1)) silently drop an idle SSH connection ~40s after the last channel closes. SuzieQ keeps the connection object and the first command of the next poll fails with 'SSH connection closed', so the service (often 'device') is lost for that cycle and the node shows as dead. On ChannelOpenError/ConnectionLost/DisconnectError (and broken pipe / reset), close the connection, reconnect without re-running the init device data, and retry the command once. Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
|
Added a third commit, related but broader than the IOS prompt fix: reconnect-and-retry in the base Seen on NX-OS 7.3(13)N1(1): the device closes an idle SSH connection ~40s after the last channel closes, and the first command of the next poll fails with Happy to split this into a separate PR if you prefer to review it independently. |
Several services share the node connection. When one of them hit the dropped connection and closed it, the others saw self._conn set to None and failed with "'NoneType' object has no attribute 'run'". Reconnect when self._conn is None and close only the connection object that actually failed. Signed-off-by: Igor Seixas <159182301+igorseixashr@users.noreply.github.com>
7b1566d to
5ebb1d3
Compare
Related Issue
No existing issue found for this (searched issues/PRs for
Device#, IOS prompt truncation).Description
For IOS/IOS-XE the persistent SSH session waits for a bare
#or>untilshow versionhas been parsed andself.promptbecomes<hostname>#/<hostname>>.On IOS 15.x (seen on CISCO1905 with 15.2(4)M3 and 15.7(3)M9) the license table in
show versioncontains a line starting withDevice#:readuntil(('>', '#'))stops at that line, so the captured output starts at\t PID\t\t\tSN. Hostname and uptime are never parsed (Cannot parse uptime), the prompt stays bare, and every following command returns the tail of the previous command's output.Symptom: the
device,interfacesandbgptables stay empty for these nodes whilesqPollerreportsOKfor every service.Type of change
New Behavior
Right after the interactive session is opened in
IosXENode._ssh_connect(), the hostname is taken from the login prompt (last line, ending in#or>, without spaces) andself.promptis anchored on it immediately, before privilege escalation andterminal length 0.Contrast to Current Behavior
Today the prompt is only anchored after
show versionis parsed, which is exactly the command that gets truncated on these platforms. With the change the prompt is anchored before any command runs.Discussion: Benefits and Drawbacks
show versioncontainsDevice#.self.promptkeeps the default('>', '#')and the current flow is unchanged. The later anchoring fromshow versionstill runs.IosXENode/IOSNodeare touched; other NOS use different connect paths.Testing
Tested with
sq-poller --run-once=processagainst real devices, on 0.25.0 (same hunk asdevelopat 75198b5):No unit test added: this path needs a live interactive SSH session and I couldn't find existing tests covering
_ssh_connect(). Happy to add one if you point me to a preferred pattern.Changes to the Documentation
None.
Proposed Release Note Entry
IOS/IOS-XE: anchor the CLI prompt on the device hostname at connect time, fixing truncated
show versionand empty device/interfaces/bgp data on IOS 15.x routers whose output containsDevice#.