S3

The S3 Protocol is a widely-adopted RESTful interface for interacting with object storage systems. Originating from AWS, it has become a standard approach for managing and accessing unstructured data across many platforms, both cloud-based and on-premises.

The S3 Protocol defines a set of HTTP(S) operations for storing, retrieving, and managing objects in a scalable object storage environment. It uses standard web protocols and conventions to offer a simple, extensible interface for developers.

• Interface type: RESTful HTTP API
• Operations: CRUD (Create, Read, Update, Delete) on buckets and objects
• Data format: Objects are stored as opaque blobs; metadata is exchanged in XML or JSON
• Transport: HTTP/HTTPS
• Authentication: HMAC-based signatures or token-based mechanisms

Core Components and Architecture

The protocol revolves around two main abstractions:

Buckets

• Containers or namespaces for objects.
• Must have globally unique names (depending on implementation).
• Represented as URLs or hostnames.

Objects

• The fundamental unit of storage, consisting of:
  • Data: Raw bytes (file, image, video, etc.)
  • Metadata: Key-value pairs describing the object
• Identified uniquely within a bucket by an object key (or name).

Additional Concepts

• Regions/Endpoints: Logical or physical locations where data is stored.
• Versioning: Optional feature to maintain multiple versions of an object.
• Access Control: Policies or ACLs to manage read/write permissions.

Protocol Fundamentals

The S3 protocol uses standard HTTP verbs mapped to operations on resources:

HTTP Verb	Resource	Action
GET	Bucket/Object	Retrieve metadata or data
PUT	Bucket/Object	Create or replace resource
DELETE	Bucket/Object	Remove resource
HEAD	Bucket/Object	Retrieve metadata only
POST	Multipart Upload	Initiate or complete uploads

URL Structure

Requests often use one of two URL styles:

• Virtual-hosted style:

https://{bucket-name}.storage-service.com/{object-key}

• Path style:

https://storage-service.com/{bucket-name}/{object-key}

S3-Compatible Implementations

Numerous storage solutions implement the S3 protocol (or subsets of it) to provide interoperability:

Implementation	Description
MinIO	Open-source, high-performance object store supporting S3 API
Ceph Object Gateway	Scalable distributed storage with S3-compatible interface
DigitalOcean Spaces	Cloud object storage service with S3 compatibility
Wasabi	Cloud object storage with S3 API support
Backblaze B2	Cloud storage service with optional S3 compatibility