S3¶

omni_aws provides functionality for accessing S3 and compatible storages.

Limited API coverage

Please note that only a bare minimum of the API is covered at this time.

API calls¶

S3 calls are prepared in a functional manner before they are executed. This allow s us to execute multiple calls at the same time using the same event loop.

Functions listed below form requests to be executed with omni_aws.aws_execute.

`s3_create_bucket` (CreateBucket)¶

Create a bucket.

Parameter	Type	Description
bucket	text	Bucket name. Required.
region	text	Region override. Optional.

`s3_list_objects_v2` (ListObjectsV2)¶

List objects in a bucket.

Parameter	Type	Description
bucket	text	Bucket nane. Required.
path	text	Path (can have a leading slash). Optional.
continuation_token	text	Continuation token. Optional.
delimiter	text	Delimiter. Optional.
encoding_type	text	Encoding type. Optional.
fetch_owner	bool	Fetch owner? Defaults to false.
max_keys	int8	Maximum number of keys. Optional.
start_after	text	Start after key. Optional.
region	text	Region override. Optional

`s3_put_object` (PutObject)¶

Put object into a bucket.

Parameter	Type	Description
bucket	text	Bucket name. Required.
path	text	Path (can have a leading slash). Required.
payload	bytea	Content of the object. Required.
content_type	text	Content type. Defaults to application/octet-stream
region	text	Region override. Optional.

API calls execution¶

The above API calls can be executed using omni_aws.aws_execute that follows the same patterns.

Singular request¶

This one is used for executing one request at a time (where TYPE is s3_create_bucket, s3_list_objects_v2 and so on):

Parameter	Type	Description
access_key_id	text	Access Key ID. Required.
secret_access_key	text	Secret Acces Key. Required.
request	`TYPE`	Request of `TYPE`. Required.
content_type	text	Content type. Defaults to application/octest-stream
endpoint	text	Custom endpoint. Optional.
region	text	Default region. Defaults to us-east-1.

These functions raise exceptions on error.

Multiple request¶

This one is used for executing ma ny requests at a time (where TYPE is s3_create_bucket, s3_list_objects_v2 and so on) for efficiency reasons:

Parameter	Type	Description
access_key_id	text	Access Key ID. Required.
secret_access_key	text	Secret Acces Key. Required.
requests	`TYPE[]`	Requests of `TYPE`. Required.
content_type	text	Content type. Defaults to application/octest-stream
endpoint	text	Custom endpoint. Optional.
region	text	Default region. Defaults to us-east-1.

These functions include errors in their output.

Return value of `aws_execute` for `s3_create_bucket`¶

Returns void for singular form and a table with error column (of type text) for multiple form.

Return value of `aws_execute` for `s3_list_objects_v2`¶

Column	Type	Description
key	text	Object key
etag	text	ETag (quoted)
last_modified	timestamp	Last modified time.
storage_class	s3_storage_class	Storage class
checksum_algorithm	s3_checksum_algorithm	Checksum algorithm
owner	s3_owner	Object owner (only if `fetch_owner` was set to `true`)
restore_status	s3_restore_status	Restore status
meta	s3_list_objects_v2_meta	Response meta-information (continuation token, etc.)
error	text	Error if any (only in the multiple request form)

Return value of `aws_execute` for `s3_put_object`¶

Returns void for singular form and a table with error column (of type text) for multiple form.

Utility functions¶

Pre-signed URL¶

s3_presigned_url can be used to create a pre-signed URL. It returns a URL as a text.

Parameter	Type	Description
bucket	text	Bucket name. Required.
path	text	Path to object. Required
access_key_id	text	Access Key ID. Required.
secret_access_key	text	Secret Acces Key. Required.
expires	int	Expiration in seconds. Defaults to 604800 (7 days)
region	text	Default region. Defaults to us-east-1.
endpoint	text	Custom endpoint. Optional.

Endpoints¶

By default, this extension works with AWS S3. However, it also works with other providers like DigitalOcean or software like MinIO.

DigitalOcean Spaces¶

In order to use DigitalOcean Spaces, simply pass omni_aws.digitalocean_s3_endpoint() to endpoint parameter (where accepted).

 select *
 from
     omni_aws.aws_execute(access_key_id => 'ACCESS_KEY_ID',
                          secret_access_key => 'SECRET_ACCESS_KEY',
                          request => omni_aws.s3_list_objects_v2(bucket => 'BUCKET', path => '/',
                                                                 region => 'nyc3'),
                          endpoint => omni_aws.digitalocean_s3_endpoint());

Ensure you specify region

Region names are different in DigitalOcean spaces and if you omit it, it'll set default to us-east-1.

Custom¶

Custom endpoints can be specified with omni_aws.s3_endpoint():

Parameter	Type	Description
url	text	URL pattern with `${region}` and `${bucket}` variables. Required.
path_style	text	Append `url` pattern with `/${bucket}` for convenience. Optional, defaults to `true`

This way one can use it with, for example, MinIO:

 select *
 from
     omni_aws.aws_execute(access_key_id => 'ACCESS_KEY_ID',
                          secret_access_key => 'SECRET_ACCESS_KEY',
                          request => omni_aws.s3_list_objects_v2(bucket => 'BUCKET', path => '/'),
                          endpoint => omni_aws.s3_endpoint('http://localhost:9000'));

S3¶