5 S3 (Storage)

(require (planet gh/aws:1:=4/s3))

S3 provides a fairly simple and REST-ful interface. Uploading an object to S3 is an HTTP PUT request. Download an object is a GET request. And so on. As a result, you may feel you don’t need a lot of “wrapper” around this.

Where you definitely will want help is constructing the Authorization header S3 uses to authenticate requests. Doing so requires making a string out of specific elements of your request and “signing” it with your AWS private key. Even a small discrepancy will cause the request to fail authentication. As a result, aws/s3 makes it easy for you to create the authentication header correctly and successfully.

Plus, aws/s3 does provide wrappers and tries to help with some wrinkles. For example, S3 may give you a 302 redirect when you do a PUT or POST. You don’t want to transmit the entire entity, only to have S3 ignore it and you have to transmit it all over again. Instead, you want to supply the request header Expect: 100-continue, which lets S3 respond before you transmit the entity.

5.1 Endpoint

5.2 Authentication signatures

5.3 Conveniences

Keep in mind that bucket names on S3 are global—shared among all users of S3. You may want to make your bucket names include a domain name that you own.

If you try to create a bucket with a name that is already used by another AWS account, you will get a 409 Conflict response.

If you create a bucket that already exists under your own account, this operation is idempotent (it’s not an error, it’s simply a no-op).

This operation is idempotent (it is a no-op to delete a bucket that has already been deleted).

• name (as with ls)

• response headers from a HEAD request (as with head)

• an xexpr representing the ACL (as with get-acl)

bucket+path is the form "bucket/path/to/resource".

Copy an existing S3 object bucket+path/from to bucket+path/to, including its metadata. Both names are of the form "bucket/path/to/resource".

It is not an error to copy to an existing object (it will be replaced). It is even OK to copy an existing object to itself.

S3 responds with an XML representation of the ACL, which is returned as an xexpr?.

Make a GET request for bucket+path (which is the form "bucket/path/to/resource").

The reader procedure is called with an input-port? and a string? respresenting the response headers. The reader should read the response entity from the port, being careful to read the exact number of bytes as specified in the response header’s Content-Length field. The return value of reader is the return value of get.

You may pass request headers in the optional heads argument.

The optional arguments range-begin and range-end are used to supply an HTTP Range request header. This header, which Amazon S3 supports, enables a getting only a subset of the bytes. Note that range-end is exclusive to be consistent with the Racket convention, e.g. subbytes. (The HTTP Range header specifies the end as inclusive, so your range-end argument is decremented to make the value for the header.)

You may pass request headers in the optional heads argument.

The optional arguments range-begin and range-end are used to supply an optional Range request header. This header, which Amazon S3 supports, enables a getting only a subset of the bytes. Note that range-end is exclusive to be consistent with the Racket convention, e.g. subbytes. (The HTTP Range header specifies the end as inclusive, so your range-end argument is decremented to make the value for the header.)

The response entity is held in memory; if it is very large and you want to "stream" it instead, consider using get.

You may pass request headers in the optional heads argument.

Makes a PUT request for bucket+path (which is the form "bucket/path/to/resource"), using the writer procedure to write the request entity and the reader procedure to read the response entity. Returns the response header (unless it raises exn:fail:aws).

The writer procedure is given an output-port? and a string? representing the response headers. It should write the request entity to the port. The amount written should be exactly the same as data-length, which is used to create a Content-Length request header. You must also supply mime-type (for example "text/plain") which is used to create a Content-Type request header.

The reader procedure is the same as for get. The response entity for a PUT request usually isn’t interesting, but you should read it anyway.

Note: If you want a Content-MD5 request header, you must calculate and supply it yourself in heads. Supplying this allows S3 to verify the upload integrity.

To use reduced redundancy storage, supply (hash 'x-amz-storage-class "REDUCED_REDUNDANCY") for heads.

Makes a PUT request for bucket+path (which is the form "bucket/path/to/resource"), sending data as the request entity and creating a Content-Type header from mime-type. Returns the response header (unless it raises exn:fail:aws).

A Content-MD5 request header is automatically created from data. To ensure data integrity, S3 will reject the request if the bytes it receives do not match the MD5 checksum.

To use reduced redundancy storage, supply (hash 'x-amz-storage-class "REDUCED_REDUNDANCY") for heads.

Upload the file pathname to bucket+path.

Makes a PUT request for bucket+path (which is the form "bucket/path/to/resource") and copy the the request entity directly from the file specified by pathname. The #:mode-flag argument is identical to that for call-with-input-file*, which is used. Returns the response header (unless it raises exn:fail:aws).

If #:mime-type is #f, then the Content-Type header is guessed from the file extension, using a (very short!) list of common extensions. If no match is found, then "application/x-unknown-content-type" is used. You can customize the MIME type guessing by setting the path->mime-proc parameter to your own procedure.

A Content-MD5 request header is automatically created from the contents of the file represented by path. To ensure data integrity, S3 will reject the request if the bytes it receives do not match the MD5 checksum.

A Content-Disposition request header is automatically created from pathname. For example if pathname is "/foo/bar/test.txt" or "c:\\foo\\bar\\test.txt" then the header "Content-Disposition:attachment; filename=\"test.txt\"" is created. This is helpful because a web browser that is given the URI for the object will prompt the user to download it as a file.

To use reduced redundancy storage, supply (hash 'x-amz-storage-class "REDUCED_REDUNDANCY") for heads.

5.4 Multipart uploads

In addition to uploading an entire object in a single PUT request, S3 lets you upload it in multiple 5 MB or larger chunks, using the multipart upload API. Amazon recommends using this when the total data to upload is bigger than about 100 MB.

5.4.1 Convenience

Each part must be at least 5 MB, except the last part.

The parts are uploaded using a small number of worker threads, to get some parallelism and probably better performance.

The parts are uploaded using a small number of worker threads, to get some parallelism and probably better performance.

5.4.2 Building blocks

Use these if the data you’re uploading is computed on the fly and you don’t know the total size in advance. Otherwise you may simply use multipart-put or multipart-put/file.

Note that S3 part numbers start with 1 (not 0).

bstr must be at least 5 MB, unless it’s the last part.

Returns a cons of part-number and the ETag for the part. You will need to supply a list of these, one for each part, to complete-multipart-upload.

Returns S3’s XML response in the form of an xexpr?.

5.5 S3 examples

(require (planet gh/aws/keys)

(planet gh/aws/s3))

(define (member? x xs)

(not (not (member x xs))))

;; Make a random name for the bucket. Remember bucket names are a

;; global space shared by all AWS accounts. In a real-world app, if

;; you have a domain name, you probably want to include that as part

;; of your name.

(define test-bucket

(for/fold ([s "test.bucket."])

([x (in-range 32)])

(string-append s

(number->string (truncate (random 15)) 16))))

(ensure-have-keys)

(create-bucket test-bucket)

(member? test-bucket (list-buckets))

(define test-pathname "path/to/file")

(define b+p (string-append test-bucket "/" test-pathname))

(define data #"Hello, world.")

(put/bytes b+p data "text/plain")

(get/bytes b+p)

(get/bytes b+p '() 0 5)

(head b+p)

(ls (string-append test-bucket "/"))

(ls (string-append test-bucket "/" test-pathname))

(ls (string-append test-bucket "/" (substring test-pathname 0 2)))

(define p (build-path 'same

"tests"

"s3-test-file-to-get-and-put.txt"))

(put/file b+p p #:mime-type "text/plain")

(get/file b+p p #:exists 'replace)

(head b+p)

(member? test-pathname (ls b+p))

(define b+p/copy (string-append b+p "-copy"))

(copy b+p b+p/copy)

(ls (string-append test-bucket "/"))

(head b+p/copy)

(delete b+p/copy)

(delete b+p)

(delete-bucket test-bucket)