AWS S3 with Terraform is one of the most effective ways to provision and manage cloud storage using Infrastructure as Code (IaC). With Terraform, you can automate the entire lifecycle of S3 buckets, from creation to policy attachment, making your deployments more reliable and reproducible.
In this guide, you’ll learn how to create S3 buckets, upload files, and attach access policies using Terraform. Whether you’re just getting started or already have experience with cloud infrastructure, this tutorial is crafted for all levels.
Table of Contents
Why Automate S3 with Terraform?
Using Terraform to manage AWS S3 brings consistency, security, and repeatability to your infrastructure.
- Scalability: Easily replicate S3 setups across environments.
- Version Control: Keep track of changes with Git.
- Security: Avoid manual misconfigurations by defining access policies in code.
With Terraform’s declarative language (HCL), you define your desired infrastructure state, and Terraform makes it happen.
Creating an S3 Bucket with Terraform
To get started, use the aws_s3_bucket resource. You need to provide a unique bucket name and can also include metadata like tags for better clarity.
Example Configuration
resource "aws_s3_bucket" "finance" {
bucket = "finance-21092020"
tags = {
Description = "Used by finance and payroll team"
}
}
If you omit the bucket name, Terraform generates a unique name automatically. However, it’s best practice to specify it for predictability.
Once applied with terraform apply, your S3 bucket will be provisioned in AWS.
Uploading Files to an S3 Bucket
To upload objects into your S3 bucket, use the aws_s3_bucket_object resource.
Required Fields
- bucket – Reference to the S3 bucket.
- key – Name of the object.
- content – File content to upload.
resource "aws_s3_bucket_object" "upload" {
bucket = aws_s3_bucket.finance.id
key = "invoice.csv"
content = "Invoice data content here"
}
By referencing the bucket’s ID, the file uploads directly into the specified S3 bucket.
Managing Access with S3 Bucket Policies
Once your bucket and object are in place, the next step is to define who can access them. This is handled through bucket policies, written in JSON and attached to the bucket using the aws_s3_bucket_policy resource.
Step 1: Reference External IAM Groups
Assume there’s an IAM group already created in AWS (e.g., finance-analysts) that needs access to your bucket. Since Terraform didn’t create the group, it needs to read its details using a data source.
data "aws_iam_group" "finance_data" {
group_name = "finance-analysts"
}
This enables Terraform to reference the group’s ARN when defining the policy.
Step 2: Attach the Bucket Policy
You can now attach a policy to your S3 bucket granting access to this IAM group.
resource "aws_s3_bucket_policy" "bucket_policy" {
bucket = aws_s3_bucket.finance.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "FullAccessToFinanceTeam",
"Effect": "Allow",
"Principal": {
"AWS": "${data.aws_iam_group.finance_data.arn}"
},
"Action": "s3:*",
"Resource": [
"${aws_s3_bucket.finance.arn}",
"${aws_s3_bucket.finance.arn}/*"
]
}
]
}
EOF
}
This setup ensures your bucket remains accessible only to authorized users and is compliant with security best practices.
Keeping State Consistent
Terraform keeps a record of resources in a file named terraform.tfstate. This file tracks the actual state of infrastructure and compares it with your defined code during each execution.
However, if resources are created outside of Terraform, like the finance-analysts IAM group in our case, they won’t appear in this state file unless imported or declared via data sources. Hence, always use data sources for existing AWS resources to keep Terraform aware of them.
Conclusion
Using AWS S3 with Terraform provides a structured, scalable, and secure way to manage your cloud storage environment. From creating buckets and uploading files to attaching access policies, you can fully automate S3 operations with code.
By leveraging Terraform, teams avoid the pitfalls of manual cloud configuration and gain confidence in repeatable, trackable infrastructure deployments.
Frequently Asked Questions (FAQs)
1. Can I use Terraform to upload large files to S3?
Yes, but for large files, it’s more efficient to use tools like the AWS CLI or SDK. Terraform is ideal for smaller files and initial setups.
2. How do I manage versioning on S3 buckets using Terraform?
Add versioning configuration inside the aws_s3_bucket resource:versioning { enabled = true }
3. Is it secure to hardcode content in aws_s3_bucket_object?
No. For better security and flexibility, consider using the source argument to reference local files instead of embedding content directly.
4. Can I manage bucket lifecycle rules with Terraform?
Absolutely. You can define lifecycle policies inside the aws_s3_bucket block using the lifecycle_rule configuration.
5. What’s the best way to attach IAM roles or groups to S3 buckets?
Use bucket policies with data sources to reference existing IAM roles or groups, as shown in this guide.