S3
2 min read
The S3 Protocol is a widely-adopted RESTful interface for interacting with object storage systems. Originating from AWS, it has become a standard approach for managing and accessing unstructured data across many platforms, both cloud-based and on-premises.
The S3 Protocol defines a set of HTTP(S) operations for storing, retrieving, and managing objects in a scalable object storage environment. It uses standard web protocols and conventions to offer a simple, extensible interface for developers.
• Interface type: RESTful HTTP API
• Operations: CRUD (Create, Read, Update, Delete) on buckets and objects
• Data format: Objects are stored as opaque blobs; metadata is exchanged in XML or JSON
• Transport: HTTP/HTTPS
• Authentication: HMAC-based signatures or token-based mechanisms
Core Components and Architecture
The protocol revolves around two main abstractions:
Buckets
• Containers or namespaces for objects.
• Must have globally unique names (depending on implementation).
• Represented as URLs or hostnames.
Objects
• The fundamental unit of storage, consisting of:
• Data: Raw bytes (file, image, video, etc.)
• Metadata: Key-value pairs describing the object
• Identified uniquely within a bucket by an object key (or name).
Additional Concepts
• Regions/Endpoints: Logical or physical locations where data is stored.
• Versioning: Optional feature to maintain multiple versions of an object.
• Access Control: Policies or ACLs to manage read/write permissions.
Protocol Fundamentals
The S3 protocol uses standard HTTP verbs mapped to operations on resources:
| HTTP Verb | Resource | Action |
|---|---|---|
GET | Bucket/Object | Retrieve metadata or data |
PUT | Bucket/Object | Create or replace resource |
DELETE | Bucket/Object | Remove resource |
HEAD | Bucket/Object | Retrieve metadata only |
POST | Multipart Upload | Initiate or complete uploads |
URL Structure
Requests often use one of two URL styles:
• Virtual-hosted style:
https://{bucket-name}.storage-service.com/{object-key}
• Path style:
https://storage-service.com/{bucket-name}/{object-key}
S3-Compatible Implementations
Numerous storage solutions implement the S3 protocol (or subsets of it) to provide interoperability:
| Implementation | Description |
|---|---|
| MinIO | Open-source, high-performance object store supporting S3 API |
| Ceph Object Gateway | Scalable distributed storage with S3-compatible interface |
| DigitalOcean Spaces | Cloud object storage service with S3 compatibility |
| Wasabi | Cloud object storage with S3 API support |
| Backblaze B2 | Cloud storage service with optional S3 compatibility |