Amazon S3 Folders Demystified
Amazon S3 is a highly-scalable object storage system. Amazon S3 can contain any number of objects (files), and those objects can be organized into “folders”. However, to S3, folders don’t really exist.
huh?
That’s right. “Folders” are a human concept, applied to S3 keys for organizational purposes. But they’re nothing special to S3 itself.
Before we begin, forget everything you know about the S3 Management Console. The S3 Management Console is a graphical user interface (GUI), and GUIs are built for humans.
We’re going to start with the AWS CLI today, so you can see the truth.
Uploading an Object to S3
When you add an object to S3 using the AWS CLI, you must specify it’s name (or “key”).
$ aws s3api put-object \
--bucket 'bucket1' \
--key 'object1.txt' \
--body 'data.txt'
The above command will upload a local file called “data.txt”, but in S3 it will be stored as “object1.txt”.
Listing the bucket shows us our object.
$ aws s3api list-objects-v2 \
--bucket 'bucket1'
{
"Contents": [
{
"Key": "object1.txt",
"Size": 8
}
]
}
Uploading an Object into a Folder
Now, imagine that we have many files to store and we want to organize the files into folders. The only difference is to add the folder into the key during upload:
$ aws s3api put-object \
--bucket 'bucket1' \
--key 'folder1/object1.txt' \
--body 'data.txt'
Notice that we’ve inserted “folder1/” into the key before the desired file name.
Now, when we list the contents of our bucket:
$ aws s3api list-objects-v2 \
--bucket 'bucket1'
{
"Contents": [
{
"Key": "folder1/object1.txt",
"Size": 8
},
{
"Key": "object1.txt",
"Size": 8
}
]
}
Some important things to make note of:
- There are two objects, one for each object we uploaded.
- The folder name is simply part of the object’s key. There’s nothing else special about that object.
- There nothing in S3 called “folder1”, a folder or otherwise.
What the S3 Management Console Shows Us
Until now, we’ve worked only with the AWS CLI. But what will the S3 Management Console show?
If we look at the bucket using the S3 Management Console now, we’ll see two entries:
- A folder called “folder1”, and
- An object called “object1.txt”.
The S3 Management Console helps us humans organize our buckets. It interprets the “/” characters in the object keys and filters the objects as if we were navigating a file system.
When we navigate into the “folder1” folder, we’ll see “folder1/object1.txt”, but with the name cleaned up.
Deleting Objects Makes Folders Disappear
At this point in time, we’re going to delete the object in the folder.
$ aws s3api delete-object \
--bucket 'bucket1' \
--key 'folder1/object1.txt'
Return to the S3 Management Console and refresh your view. The object will disappear.
Navigate to the parent folder, and “folder1” will have disappeared too.
This is because there are no more objects whose key starts with “folder1/”.
Folders as Objects
So, now I’m going to really bake your noodle: your folder can be an object.
What??
Try this one.
$ aws s3api put-object \
--bucket 'bucket1' \
--key 'folder2/' \
--body 'data.txt'
Did you see what I did? I uploaded the file as “folder2/”. Note the “/” character at the end of the key.
$ aws s3api list-objects-v2 \
--bucket 'bucket1'
{
"Contents": [
{
"Key": "folder2/",
"Size": 8
},
{
"Key": "object1.txt",
"Size": 8
}
]
}
Now we have two objects:
- An object named “folder2/” that has 8 bytes, and
- An object not in any folders.
Return to the Management Console and refresh. The folder will appear.
Even though “folder2/” is an object, it’s interpretted by the Management Console as a folder because it ends with the “/” character.
As much as you may try, you will not be able to access the “folder2/” object data using the Management Console. The console will only show it as a folder. To access the object (either to download or delete), you must use the AWS CLI or SDKs.
Navigate into that folder and you’ll see it’s empty.
Why is this different than before?
Previously, we deleted the object in the folder and the folder disappeared. This was because there wasn’t any more objects that began with “folder1/”. The S3 Management Console didn’t need to fake a folder for us anymore.
However, now, we have an object called “folder2/”, so the folder is not being faked, it’s actually there as an object.
Creating a Folder Using the S3 Management Console
Return to the root folder and create a folder called “folder3”.
You’ll see it in the list.
But when we use the CLI, we’ll see the new folder as an object.
$ aws s3api list-objects-v2 \
--bucket 'bucket1'
{
"Contents": [
{
"Key": "folder2/",
"Size": 8
},
{
"Key": "folder3/",
"Size": 0
},
{
"Key": "object1.txt",
"Size": 8
}
]
}
In this listing, there’s an 0-byte object called “folder3/”. That’s our empty folder.
Some Other Tricks
You can create objects that start with “/”.
aws s3api put-object \
--bucket 'bucket1' \
--key '/abc' \
--body 'data.txt'
When you do this, you’ll see a “blank” folder in the Management Console. If you hover your mouse correctly, you’ll be able to click on the “blankness” and navigate into that folder.
Believe it or not, you can even create an object simply called “/”.
aws s3api put-object \
--bucket 'bucket1' \
--key '/' \
--body 'data.txt'
But Surely Folders Must Exist
No they don’t, not within S3 itself.
The S3 Management Console “magics” folders for your viewing pleasure.
The AWS CLI and many AWS SDKs also include API helpers to help you “folderize” your S3 buckets. With the AWS CLI, you can use the --delimiter parameter for this:
$ aws s3api list-objects-v2 \
--bucket 'bucket1' \
--delimiter '/'
{
"CommonPrefixes": [
{
"Prefix": "/"
},
{
"Prefix": "folder2/"
},
{
"Prefix": "folder3/"
}
],
"Contents": [
{
"Key": "object1.txt",
"Size": 8
}
]
}
Here, it includes “object1.txt” in the bucket contents. But it’s also showing you 3 so-called folders: “/”, “folder2/”, and “folder3/”.
Now, using the --prefix parameter, we can “navigate” into those folders.
$ aws s3api list-objects-v2 \
--bucket 'bucket1' \
--delimiter '/' \
--prefix 'folder2/'
{
"Contents": [
{
"Key": "folder2/",
"Size": 8
},
{
"Key": "folder2/object2.txt",
"Size": 8
}
]
}
Here’s two objects:
- The actual folder object “folder2/”, because that folder is an object, and
- “folder2/object2.txt”, because I put it there.
You’ll need to remove the prefix from the key yourself before displaying it to the user.
Folder Delimiters
Usually, you’ll use the “/” character to separate folders in the key. Many of S3’s tools use this character by default. However, there’s nothing to say you cannot use another character.
$ aws s3api list-objects-v2 \
--bucket 'bucket1' \
--delimiter 'd'
{
"CommonPrefixes": [
{
"Prefix": "fold"
}
],
"Contents": [
{
"Key": "/",
"Size": 8
},
{
"Key": "object1.txt",
"Size": 8
}
]
}
As you can see, I’m using the “d” character for my folder delimiter instead. All objects that don’t have a “d” in the key are considered to be in the “root folder”, and there’s a single subfolder called “fold” that I can navigate into.
$ aws s3api list-objects-v2 \
--bucket 'bucket1' \
--delimiter 'd'
--prefix 'fold'
{
"Contents": [
{
"Key": "folder2/",
"Size": 8
},
{
"Key": "folder2/object2.txt",
"Size": 8
},
{
"Key": "folder3/",
"Size": 0
}
]
}
Final Thoughts
AWS is “helping” us humans in the AWS Management Console by allowing us to navigate a bucket like a file system, and they have added helpers to the tools to aid with this as well.
But folders are a human concept bolted on top of S3’s efficient management of the objects using the full key alone.
Try Skeddly Today
Skeddly is the premier automation and scheduling service for your AWS account. Automatically start & stop your EC2 and RDS instances, create daily backups, and automate daily tasks.
Sign-up for our 30 day free trial or sign-in to your Skeddly account to get started.