Pin the tunnel's SSH host key so a reused connection name reconnects - #6557
Conversation
`databricks ssh connect --name X` recorded the tunnel's host key in
`~/.ssh/known_hosts` under the connection name, with
`StrictHostKeyChecking accept-new`. A name identifies compute within one
workspace, but `~/.ssh/known_hosts` is global and keyed by name alone, so an
entry recorded for a name kept rejecting later connections that legitimately
had a different host key - reusing a name in a second workspace, or against
compute whose key was regenerated. ssh aborted with
@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@
Host key verification failed.
and recovery meant running `ssh-keygen -R <name>` by hand, which the CLI could
only suggest (#5645).
The server already publishes its host key to the connection's secret scope and
reuses it for every sshd it launches, so the workspace is the authority on the
key. The client now reads it from there and pins it in
`~/.databricks/ssh-tunnel-known-hosts/<name>` before every connection, pointing
ssh at that file with `StrictHostKeyChecking yes`. A key recorded for the name
earlier is replaced instead of failing the connection, and verification gets
stronger: the key is checked against what the workspace published rather than
accepted on first sight. `ssh setup` writes the same two options into its host
block, where the ProxyCommand refreshes the pin before ssh verifies it, so
`ssh <name>` and IDE remote-development sessions take the same path.
Tests: the acceptance test starts from the state a user reconnecting with a
reused name is in - a stale entry for the name - and asserts the connection
succeeds and the entry is replaced. The test server's tunnel now serves the
host key from the session's secret scope, as the real server does, instead of
generating an ephemeral one per connection.
Co-authored-by: Isaac <no-reply@databricks.com>
Two follow-ups on the host key pinning: The acceptance test asserted on the tunnel's stderr, where the last lines depend on which side closes the websocket first - the proxy's receiving loop reports the closed connection only when it loses the race. It now asserts the remote command's output (out.connect-*.txt) and routes stderr to a LOG file, matching the sibling ssh tests. Verified stable over repeated runs. PinHostKey leaves an already-correct file alone. Every ssh invocation refreshes the pin through the ProxyCommand and an IDE opens several at once, so the rewrite is nearly always a no-op - and on Windows renaming over a file another ssh has open fails. Co-authored-by: Isaac <no-reply@databricks.com>
Integration test reportCommit: 2bc3149
Top 12 slowest tests (at least 2 minutes):
|
|
Finding 1 - BLOCKER - #6557 breaks After ssh setup --name --cluster , connecting with ssh fails permanently: $ ssh bv-alias-merged Root cause. The host key is pinned under the session id, but ssh verifies against the host alias, and for a dedicated cluster those differ: setup.go:51 - GetKnownHostsPath(ctx, opts.ClusterID) names the known_hosts file after the cluster id. setup.go:56 - GenerateHostConfig(opts.HostName, ...) makes the ssh Host alias the user's --name, and adds StrictHostKeyChecking yes (main used accept-new with no UserKnownHostsFile). client.go:533 - the ProxyCommand's pinServerHostKey calls PinHostKey(path, sessionID, key), so the single entry is named 0907-190337-c3qelmbu while ssh looks up bv-alias-merged. Evidence (A/B on the same dogfood cluster, same pinned version, same flow): origin/main : ssh setup --name bv-alias-main -> ssh bv-alias-main -> REMOTE_OK root ... exit 0 Deterministic: a second attempt fails identically (not a first-run race). Impact. Affects every ssh setup user: --name is a required, user-chosen label (MarkFlagRequired("name"), no default), so it essentially never equals the cluster id. This is the documented entry point for dedicated clusters and the path IDE Remote-SSH uses via the host alias. Users who merely upgrade without re-running setup keep working (their persisted config still says accept-new); the break lands the moment they re-run ssh setup. Not affected: serverless (alias == connection name == session id) and ssh connect --cluster / --ide (both use SessionIdentifier() as the alias, so the names match). Suggested fix. Pin under the name ssh actually verifies: plumb the host alias into the ProxyCommand (it currently carries only --cluster) and pass it to PinHostKey; or record both names on the known_hosts line; or add a HostKeyAlias directive to the generated host config so ssh looks the key up under the session id. An acceptance test that drives a real ssh client through a setup-generated config would have caught this - the current tests never run ssh, and acceptance/ssh/reconnect-reused-name only covers the serverless shape where the two names coincide. |
`ssh setup --name <alias> --cluster <id>` writes a `Host <alias>` block, but the ProxyCommand pins the server's host key in the known_hosts file under the session ID (the cluster ID for a dedicated cluster). With StrictHostKeyChecking yes and no HostKeyAlias, OpenSSH looks the key up under the connect target - the alias - so when the user-facing name differs from the cluster ID the file has no matching entry and every connection fails host-key verification. Emit `HostKeyAlias <session-id>` in the generated block so the key lookup uses the exact name the entry is pinned under. GenerateHostConfig now takes the alias and renders the directive when it is non-empty; setup passes the cluster ID (which is both the known_hosts file name and the pinned entry's host field), and the IDE path passes the session identifier it already pins under. Co-authored-by: Isaac <no-reply@databricks.com>
| // Optional path to the known hosts file for this session. Defaults to | ||
| // ~/.databricks/ssh-tunnel-known-hosts/<session>. The CLI owns this file and rewrites | ||
| // it with the server's host key on every connection. |
There was a problem hiding this comment.
We should guard this flag somehow rather than just changing the semantics in a comment. This now points to known hosts for a session, not a user. If a user passes ~/.ssh/known_hosts (or some other shared files) we will delete the contents without warning
There was a problem hiding this comment.
Good catch - fixed in dfa498b by making the override name a directory instead of a file.
--user-known-hosts-file is now --known-hosts-dir, mirroring the SSHKeysDir option next to it, and the file inside it is always named after the session (GetKnownHostsPath(ctx, sessionID, dir), same shape as keys.GetLocalSSHKeyPath). Relocating the directory is all a caller can do, so pointing it at ~/.ssh writes ~/.ssh/<session-id> rather than truncating known_hosts - the footgun is gone by construction rather than by validation.
Two things fell out of it:
- The resolved path no longer gets written back into
ClientOptions(opts.UserKnownHostsFile, err = pinServerHostKey(...)), where the input flag doubled as the pin location - that overloading was what made the semantics ambiguous in the first place. It is threaded alongsidekeyPath, whichRunalready resolves the same way. acceptance/ssh/connection/known_hostsis deleted: it described the entryaccept-newused to append, and nothing appends any more. The four ssh acceptance tests pass--known-hosts-dir=known-hostsso their pins stay inside the test directory (verified locally against a real sshd - nothing lands in the real~/.databricks/ssh-tunnel-known-hosts/).
Co-authored-by: Russell Clarey <russell.clarey@databricks.com>
Co-authored-by: Russell Clarey <russell.clarey@databricks.com>
`--user-known-hosts-file` used to hand its path straight to ssh, which appended to it. Now that the CLI pins the workspace's host key it rewrites the file it is given, so the same flag pointed at `~/.ssh/known_hosts` - or any other shared file - would replace its contents with a single tunnel entry. A comment saying the CLI owns the file is not a guard. Take a directory instead: `--known-hosts-dir`, mirroring the SSHKeysDir option next to it, with the file inside it always named after the session. Relocating the directory is then all a caller can do, so no file the CLI does not own can be overwritten - pointing it at `~/.ssh` writes `~/.ssh/<session-id>` rather than truncating `known_hosts`. The resolved path also stops being written back into ClientOptions, where an input flag doubled as the pin location; it is threaded alongside keyPath, which Run already resolves the same way. The acceptance tests keep their pins inside the test directory, and the `connection` fixture goes away: it described the entry `accept-new` used to append, and nothing appends any more. Co-authored-by: Isaac <no-reply@databricks.com>
4 participants
Why
Reconnecting to an SSH tunnel with a connection name that was used before could
fail with
Host key verification failed, even when the connection was perfectlylegitimate. A
--nameis a stable handle for serverless compute, not a livesession id, so the compute behind a name changes over time (a new server on the
same name once the previous one shut down). It could also collide with an entry
left in the global
~/.ssh/known_hostsby the same name in another workspace,since a name is unique only within one workspace. Users had to run
ssh-keygen -Rby hand to recover. (DECO-27882)
What
The tunnel server generates its host key once, stores it in the connection's
secret scope in the workspace, and reuses it for every
sshdit launches — sothe workspace is the authority on the key. This change has the client read that
key and pin it before every connection:
sshconfig.GetKnownHostsPath/PinHostKey: tunnel host keys are recordedin
~/.databricks/ssh-tunnel-known-hosts/<name>instead of~/.ssh/known_hosts,so a name can never collide with an entry left by other compute. The file is
owned by the CLI and rewritten (not edited) with the workspace's key before each
connection, via an atomic write-and-rename; an already-correct pin is left alone
so concurrent IDE connections don't race.
client.Runpins the key (pinServerHostKey) before the tunnel carries a byte,and
ssh setupwrites the same path into its host block.StrictHostKeyCheckingmoves fromaccept-newtoyes, pointed at the pinnedfile — there is no trust-on-first-use window left, since the key is always
refreshed from the workspace immediately beforehand.
ssh-keygen -R(that advice nolonger applies); it now explains that the server presented a key the workspace
does not know about.
Tests
PinHostKey(comment stripping, replacing a hashed entry,leaving a correct pin untouched, rejecting a malformed key) and
GetKnownHostsPath.buildSSHArgs/GenerateHostConfigtests assert strict checking againstthe pinned file.
acceptance/ssh/reconnect-reused-namedrives a realsshd(backed by the test server, whose host key now comes from the session secret
scope) through a full handshake over a stale local entry, and asserts the stale
key is replaced and the reconnect succeeds. Linux-only, skipped when
sshdisabsent.
Changelog fragment added under
.nextchanges/cli/.