|
|
Subscribe / Log in / New account

Restricting SSH agent keys

Please consider subscribing to LWN

Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

By Jake Edge
January 5, 2022

The OpenSSH suite of tools for secure remote logins is used widely within our communities; it also underlies things like remote Git repository access. A recent experimental feature for the upcoming OpenSSH 8.9 release will help close a security hole that can be exploited by attacker-controlled SSH servers (e.g. sshd) when the user is forwarding authentication to a local ssh-agent. Instead of allowing the keys held in the agent to be used for authenticating to any host where they might work, SSH agent restriction will allow users to specify where and how those keys can be used.

The ssh-agent is used to to simplify making repeated connections to the same host; it stores and manages SSH keys so that the passphrases protecting them do not need to be entered each time a connection is made. Normally, the passphrase is used once to unlock a key, which then gets stored by the agent when ssh-add is used; alternatively, the ssh_config option AddKeysToAgent can be used for the same purpose. ssh-agent is a "deliberately simple program" since it holds private keys. Damien Miller's description of the new feature (linked above) described it this way:

It speaks a simple, client-initiated protocol with a small number of operations including adding or deleting keys, retrieving a list of public halves for loaded keys and, critically, making a signature using a private key. Most interactions with the agent are through the ssh-add tool for adding, deleting and listing keys and ssh, which can use keys held in an agent for user authentication, but other tools can also be used if they speak the protocol.

Since the agent contains high-value keys, it is "a desirable and frequently-exploited target for attackers". The agent is only accessible from the local system, which greatly limits the attack surface, unless access to the agent has been forwarded to a remote system. Using the -A option to ssh (or the ForwardAgent configuration directive) will arrange for the remote host to be able to communicate with the local agent. That remote host can then perform all of the agent operations in the same way that local programs can.

This kind of agent forwarding is generally used so that multi-hop SSH connections can be made without needing to re-enter the passphrase to unlock the key on the remote host—possibly many times. It also means that the private keys do not need to stored on the remote hosts. A user who remotely connects to HostA, and from there to one or more other hosts using the same SSH key, will likely find it convenient to make the initial SSH connection to HostA with agent-forwarding enabled; SSH connections from HostA may extend the agent-forwarding path, as well.

The problem occurs when agent access is forwarded to an attacker-controlled system that then can use the keys stored in the agent to authenticate to any other host the user's keys give them access to. So the user may be using forwarding for HostA, HostB, and HostC, but their key will grant them access to HostV or HostZ that an attacker wants to target for some reason. Currently, SSH has no way to restrict how the keys held by the agent can be used; that is the problem that the new feature is meant to address.

Part of the solution is separating local access from remote access to the agent, so that some keys can only be used from the local system even if agent access is forwarded. Arguably, conflating those two types of agent access was a mistake made long ago, so being able to add keys with restrictions on how they can be used will help to rectify that. A new -h option has been created for ssh-add to describe the legal uses of a key, as an example from the feature description shows:

These extensions allow the user to add destination constraints to keys they add to a ssh-agent and have ssh enforce them. For example, this command:
    $ ssh-add -h "perseus@cetus.example.org" \
              -h "scylla.example.org" \
    	      -h "scylla.example.org>medea@charybdis.example.org" \
              ~/.ssh/id_ed25519
Adds a key that can only be used for authentication in the following circumstances:
  1. From the origin host to scylla.example.org as any user.
  2. From the origin host to cetus.example.org as user perseus.
  3. Through scylla.example.org to host charybdis.example.org as user medea.
Attempts to use this key to authenticate to other hosts will be refused by the agent because they weren't explicitly listed, as would an attempt to authenticate through scylla.example.org to cetus.example.org because the path was not permitted. Likewise, trying to authenticate as any other user then perseus to cetus.example.org or medea to charybdis.example.org would fail because the destination users are not permitted.

More complicated paths can be specified, but each hop needs to be listed in its own -h option. So a multi-hop path might look something like:

    $ ssh-add -h "HostA" \
              -h "HostA>HostB" \
    	      -h "HostB>HostC" \
              key-file
It should be noted that an agent configuration like the above would not allow the agent's key to be used to go directly from the local system to HostB or HostC, they could only be reached via the appropriate hops. The user may still be able to bypass the agent and fall back to typing in the passphrase for the key when prompted by ssh in order to go directly to HostB or HostC, however.

The new feature requires updates to the client-side tools, but also needs updated SSH servers on the remote systems. The agent protocol needed to change to incorporate the server host key in the authentication requests, so older SSH servers will not be able to participate in the new scheme. The feature will "fail safe" if ssh-add or the SSH server do not support the agent restrictions; ssh-add will fail if it does not understand the destination constraints and the agent will decline authentication requests that are not sent with the host key.

There are some caveats. The biggest is that attackers can still hijack the agent connection so they can request authentication to hosts (and users) that have been authorized, but from different hosts than expected:

Less obviously, they will also be able to forward use of the agent to other hosts, e.g. by using an SSH implementation that doesn’t cooperate with ssh-agent, or another tool entirely, such as socat. Note that the attacker isn’t gaining any new access to keys here, they are still forced to act via the compromised host and their access is still restricted to the keys that were permitted for use though the intended host only.

[...] Because of these subtleties, it’s better to think of key constraints as permitting use of a key through a given host rather than as from a particular host, and, more generally, that any forwarding path is only as strong as its weakest link. Another helpful way to think about key constraints is that each one represents a delegation of a key to a host, that is only slightly more trustworthy than the delegate is.

Overall, this seems like a welcome addition to the SSH toolbox; the restrictions provided will be useful. It is nice to know that agent forwarding will no longer provide carte blanche access to any host where the key will work. The document describing the feature is admirably detailed, with a look at the implementation details, plans for the future, and more. Interested readers are encouraged to take a look.


Index entries for this article
SecurityOpenSSH


(Log in to post comments)

Restricting SSH agent keys

Posted Jan 5, 2022 23:14 UTC (Wed) by lucaswerkmeister (subscriber, #126654) [Link]

> A user who remotely connects to HostA, and from there to one or more other hosts using the same SSH key, will likely find it convenient to make the initial SSH connection to HostA with agent-forwarding enabled

Isn’t it usually better to use ProxyJump for such situations, leaving the authentication with the other hosts to the local SSH instance? Or am I missing something?

Restricting SSH agent keys

Posted Jan 6, 2022 1:52 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

It may be the case that you want to have a long-lived session on the bastion, and a bunch of short-lived sessions on various individual servers, perhaps with -L and/or -R forwarding between the bastion and the servers. I'm fairly certain you can still do that with an appropriate ProxyJump configuration, but it seems like it would be more complicated to set up (as opposed to just typing a bunch of individual ssh commands with appropriate flags after you have ssh'd into the bastion). Meanwhile, if you *mostly* trust the bastion but want defense in depth against extraordinary insider threats, applying ssh-agent key restrictions might be a good compromise for that use case.

Restricting SSH agent keys

Posted Jan 6, 2022 9:31 UTC (Thu) by taladar (subscriber, #68407) [Link]

Personally I would find having to type any commands on the bastion host a lot less convenient than just putting whatever ProxyCommand or (in newer versions) ProxyJump configurations into my ~/.ssh/config and literally forgetting about which hosts even need one or more levels of connections via bastion hosts.

Restricting SSH agent keys

Posted Jan 6, 2022 18:21 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

If you need to run a long-lived thing on the bastion, you may have no choice but to SSH into it in order to configure that long-lived thing. At that point, it's largely a matter of convenience.

Another possibility is that there are internal services which are inaccessible from your local host, and you need to SSH to a bastion just to do anything interesting at all. It just so happens that one of the various interesting things you need to do is SSH into other servers. But you also need to be on the bastion to do things like send RPCs, check out (proprietary) source code, etc., because your local host is not trusted enough to do those things itself.

Restricting SSH agent keys

Posted Jan 8, 2022 17:46 UTC (Sat) by atnot (subscriber, #124910) [Link]

One example might be to run programs like ansible that uses ssh to manage a large number of machines at once. Scripts that rsync data between two servers are also not that uncommon in oldschool sysadmining. There's definitely better solutions for those things these days but it'll take a while for everyone to get there.

Restricting SSH agent keys

Posted Jan 9, 2022 5:07 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

This is a good example, but I also think that there's an element of pets-vs-cattle here. In the cattle case, you're absolutely right: Ideally, you never SSH into cattle at all. If you need to SSH into something, it's broken and you should probably either redeploy the offending container or just reimage the silly thing, but in practice that's not always the case (You use containers, right? You can safely and easily reimage everything whenever you want, right?).

But the other practical reality is that not everything is going to be cattle in the first place. Some machines (workstations, mostly) are pets, and will always be pets, because each individual machine has slightly different requirements and there's no reasonable way to fully and completely standardize them. For those machines, a certain amount of manual remote administration is inevitable, especially in the brave new world of everyone working from home. Once you realize that this is a real use case, then ProxyJump starts to look a lot less reasonable as (the only) solution. Sometimes, manual multi-hop SSH is just *easier* in the context of everything else you're doing at the time.

Restricting SSH agent keys

Posted Jan 8, 2022 3:18 UTC (Sat) by droundy (subscriber, #4559) [Link]

One scenario where that wouldn't help is if you want to scp between two remote hosts. I think.

Restricting SSH agent keys

Posted Feb 25, 2022 19:48 UTC (Fri) by misc (guest, #73730) [Link]

Or doing git commit or git clone with a repo served over ssh, on that remote server.

Restricting SSH agent keys

Posted Jan 21, 2022 23:18 UTC (Fri) by djm (subscriber, #11651) [Link]

Yes, you should definitely use ProxyJump over agent forwarding if you can. There are some circumstances where that isn't possible or convenient though.

SSH and ProxyJump

Posted Jan 5, 2022 23:30 UTC (Wed) by nickodell (subscriber, #125165) [Link]

Glad to see this work being done. I want to mention a more secure alternative to agent forwarding which is easy to set up.

Many people who use agent forwarding are using it to log into a bastion host of some kind, and then logging into a second host, where the second host is not directly accessible from the internet.

For this use case, you can use a jump host instead of agent forwarding. Rather than having the bastion host decrypt and re-encrypt the SSH tunnel, the jump host works by asking the bastion to forward encrypted traffic to the destination server. This way, you don't need to trust the bastion host.

For more information, take a look at the -J option in the ssh manpage, or the ProxyJump option in the ssh_config manpage. It's available since OpenSSH 7.3.

SSH and ProxyJump

Posted Jan 5, 2022 23:47 UTC (Wed) by rra (subscriber, #99804) [Link]

ProxyJump can be easily simulated with older versions of ssh that don't support it by using ProxyCommand in .ssh/config instead:

ProxyCommand ssh <bastion> nc -w 1 %h 22

ProxyJump is basically shorthand for that, without requiring netcat be installed on the bastion host. (There are variations using other ssh options that were added later, but I think the above ProxyCommand syntax will work with quite old versions of ssh.)

SSH and ProxyJump

Posted Jan 6, 2022 10:52 UTC (Thu) by grawity (subscriber, #80596) [Link]

Realistically, if someone holds valuable SSH keys on the local system, then they're likely using at least OpenSSH 5.2 (2010) which is when the netcat-free -W %h:%p option became available. So I'd rather describe ProxyJump as shorthand for ProxyCommand "ssh <bastion> -W %h:%p". Although netcat-based fallbacks are definitely still useful in situations where the jumphost's admin deliberately disabled TCP tunnelling...

Requiring confirmation before using the key

Posted Jan 6, 2022 5:59 UTC (Thu) by fmarier (subscriber, #19894) [Link]

There is a related ssh-add option which makes keys subject to a user confirmation (i.e. pressing Enter or clicking a button) via ssh-askpass before using the key for authentication. I have the following in my ~/.bash_aliases:
alias ssh-add='ssh-add -c'

Restricting SSH agent keys

Posted Jan 6, 2022 9:37 UTC (Thu) by taladar (subscriber, #68407) [Link]

Unless the agent itself initiates the whole connection how does it know that the SSH process on the compromised server does not present the host key for host A and says it will use user a while actually using the signing result to connect to host B and use user b? Does the content it has to sign to connect contain that information and it inspects it?

Restricting SSH agent keys

Posted Jan 6, 2022 10:58 UTC (Thu) by grawity (subscriber, #80596) [Link]

The agent protocol has been extended (see session-bind in the PROTOCOL.agent file) to associate that specific socket with a hostkey, so if you connect to host A, your local client informs ssh-agent that future commands will originate from host A.

The publickey SSH auth method has also been extended (well, forked) to make the to-be-signed challenge include the server's hostkey, so the agent knows what server you're authenticating to. I think that's how the destination constraint is enforced; if the final server doesn't support the extended auth method, but only the standard one, then I assume the agent will not allow the key to be used at all.

Restricting SSH agent keys

Posted Jan 6, 2022 15:10 UTC (Thu) by pj (subscriber, #4506) [Link]

I wonder if the guardian agent project (https://github.com/StanfordSNR/guardian-agent) will take advantage of the agent protocol changes? It's a local agent that pops up a local "did you auth this? Yes/No/AllLikeThis" kind of prompt whenever a remote agent request is forwarded to the local agent. Due to the mentioned agent protocol limitations it currently uses its own encrypted agent channel so it can add that information, but that may not be necessary now.

Restricting SSH agent keys

Posted Jan 8, 2022 23:58 UTC (Sat) by tiwe (guest, #100154) [Link]

Some years from now this might make many uses of ssh-agent-filter obsolete.

physical ssh keydevices - not an issue?

Posted Jan 20, 2022 13:12 UTC (Thu) by Klavs (guest, #10563) [Link]

This only works at solving the problem which only (as I understood it) is relevant for on-disk private keys.
So for those of us, using ssh-keys via gnupg daemon (backed by a hardware key like yubikey) - this feature is not a security benefit at all?

physical ssh keydevices - not an issue?

Posted Jan 21, 2022 23:17 UTC (Fri) by djm (subscriber, #11651) [Link]

It's of benefit to FIDO security keys too. ATM for forwarded agents, it's possible for an attacker to "phish" FIDO key touches. E.g. they wait for you to do something that will require a touch and jump in first with their own request. These protocol changes make this more difficult.


Copyright © 2022, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds