Hardening a Linux fleet over SSH without locking yourself out

Hardening a fleet always ends up touching sshd_config eventually — tightening PermitRootLogin, trimming the cipher and key-exchange lists, adding an AllowUsers line. It's also the one config change with a uniquely bad failure mode: if you get it wrong, the mistake takes down the exact channel you'd use to fix it.

How people actually lock themselves out

The usual sequence: edit the file directly over an existing SSH session, run systemctl restart sshd, and find out a moment later. Sometimes it's a syntax error the daemon refuses to load. More often it's syntactically valid and wrong anyway — an AllowUsers line that doesn't include the account you're using, a Match block that unintentionally captures your connection, a cipher list that excludes what your client offers. The restart succeeds. Your session is still open. And the next connection attempt, from anywhere, fails. On a remote host with no console access, that's not an inconvenience — that's a machine you now need physical hands or a hypervisor console to recover.

Three guardrails that make it structurally hard to get wrong

Validate before you ever restart. sshd -t -f <candidate config> checks syntax against the live binary before it touches the running daemon. This catches the syntax-error case entirely, for free, before anything is at risk.

Never close your only channel. Apply the change, then open a second, independent connection — a fresh authentication, not a second tab riding the session you already have — and confirm it succeeds against the new config before you let the first one go. If the new session can't authenticate, you still have the old one to revert from.

Put a deadline on it. Schedule an automatic revert — a one-shot systemd timer or an at job — that restores the previous config and restarts sshd in a fixed number of minutes unless it's explicitly cancelled. A dead-man's switch means a mistake resolves itself even if nobody's watching when it happens.

The same discipline, at fleet scale

The sshd-specific guardrails are really a special case of a broader rule for fleet hardening: dry-run and diff every change before it's applied, back up the exact file you're about to touch rather than a whole directory snapshot, roll out to a small canary batch before the rest of the fleet, and require an explicit, typed confirmation for anything flagged as disruptive rather than a plain y/n you can reflexively hit.

That's the shape Linux Fleet Harden enforces mechanically: it re-verifies every finding against the live host before touching it, backs up each file first, applies its sshd guard — validate, watchdog, fresh-session check — around any daemon-affecting change, and supports full rollback. Dry-run by default; anything disruptive needs a typed confirmation. The point isn't that the discipline above is exotic. It's that at 11pm, on the fortieth host in a run, discipline that lives only in your head is the first thing to slip.