Installation
Please refer to Amazon's documentation for instructions on how to install AWS CLI for Linux, macOS, or Windows.
Note that for Linux, there are packages for AWS CLI (aws in Fedora, or awscli for Ubuntu), sometimes pre-installed, but these are often version 1. If you want the latest features and fixes that come with version 2, you need to download and manually install the package directly from AWS according to the guide in the link above.
You can check which version you have installed (Unix) as follows:
$ aws --version aws-cli/2.13.23
Using the tool
The AWS CLI tool offers a wide range of options, but for communication with S3, you will only need aws s3 or aws s3api. Manuals can be checked as follows:
aws s3 help aws s3api help
In short, aws s3 is used to perform simple high-level tasks and treat the bucket as if it were a folder, while aws s3api allows the use of any AWS REST API endpoint for specific, low-level requests. You can read more about the differences here.
This guide will cover some common use cases for both options.
Configuration
F?rst m? du sette opp dine n?kler i ~/.aws, som beskrevet i oppstartsguiden v?r:
This should be sufficient to test your connection by listing available buckets with:
aws s3 ls --endpoint https://s3-oslo.educloud.no
Alternatively, use the --profile option if you have multiple key pairs. Without specifying --endpoint, it will default to using an AWS endpoint, which will be incorrect. But the default can be changed so you don't have to specify this every time.
If you have AWS CLI version 2, you can define our endpoint as the default by adding this to ~/.aws/config:
[profile default] region = oslo endpoint_url = https://s3-oslo.educloud.no
You can also add different configurations for different profiles in ~/.aws/credentials if you use buckets from multiple services/endpoints.
If you have AWS CLI version 1 and cannot upgrade to version 2, there is unfortunately a known bug that prevents the endpoint defined in the config file from being picked up. In this case, you can work around this by creating an alias for the aws command in your shell configuration:
alias aws='aws --endpoint https://s3-oslo.educloud.no'
This will work with both aws s3 and aws s3api.
High-level file interaction
By using the "aws s3-" option, files can be managed with regular Unix shell commands: ls, rm, cp, and mv.
The path to the object(s) must start with s3://
Listing all your buckets
aws s3 ls [--profile <profile name>] Ex. $ aws s3 ls --profile markusor 2023-03-15 11:40:04 1003-green-markusor-test 2023-02-20 10:35:55 1003-markusor-benchmark
Note that IAM keys do not have permission to attempt to list all the owner's buckets, even if they are all specified in the policy document. Instead, you must explicitly attempt to list the contents from a specific bucket, see below.
List the contents of a bucket
aws s3 ls s3://<bucket name> Ex. $ aws s3 ls s3://1003-green-markusor-test PRE testmappe/ 2023-04-20 15:04:07 578885 foo.jpeg 2023-10-03 20:34:28 4897 scanner.sh (PRE corresponds to directories)
Further, you can list the contents of a folder as follows:
aws s3 ls s3://<bucket name>/<directory name>/ Ex. $ aws s3 ls s3://1003-green-markusor-test/testmappe/ 2023-10-03 20:34:28 301111 test.log 2023-10-03 20:34:28 1675 foo.md
Note: The path must end with "/", or only the folder itself will be listed.
Moving or copying files to or from the bucket
# From local disk to bucket aws s3 mv ./local_file.txt s3://<bucket_name>/ # From bucket to local disk aws s3 mv s3://<bucket_name>/remote_file.txt ./ Similarly for copying with aws s3 cp. In practice, this uploads or downloads objects from the bucket.
Deleting objects
aws s3 rm s3://<bucket name/remote_file.txt
You can also add the option --recursive to remove a folder with all its contents.
Low-level interactions
Access to all AWS S3 API calls can be made via aws s3api. For example, you can use it to retrieve metadata about a specific file:
aws s3api head-object --bucket <bucket_name> --key <filsti>
Ex.
$ aws s3api head-object --bucket 1003-green-markusor-test --key scanner.sh
{
"AcceptRanges": "bytes",
"LastModified": "Thu, 26 Oct 2023 11:19:33 GMT",
"ContentLength": 4899,
"ETag": "\"ce794b44508bce5be18b172cc8ee5203\"",
"CacheControl": "private, max-age=0, no-cache, no-store",
"ContentType": "text/x-shellscript",
"Metadata": {}
}
Advanced queries
s3api has a query option that offers advanced searches and can format the output in a user-friendly format.
The example below retrieves the name and size of all files in a bucket:
aws s3api list-objects --bucket <bucket namen> --output json \
--query "[Contents][].{Key:Key,Size:Size}"
Ex. $ aws s3api list-objects --bucket 1003-green-markusor-test --output json \
--query "[Contents][].{Key:Key,Size:Size}"
[
{
"Key": "scanner.sh",
"Size": 4899
},
{
"Key": "test.md",
"Size": 6
}
]
The argument to query follows JMESpath syntax, with an excellent guide on their own site: