The TwentyThree™ API

Resumable uploads

The TwentyThree platform supports upload of large files through the Resumable.js pattern. This allows for parallelizable, fault-tolerant and resumable file delivery to the platform.

We generally recommend using the resumable pattern whenever possible, and strongly recommend it for all uploads of files larger than 500 MB. The maximum file size for non-resumable uploads is capped at 4 GB. The max size with resumable chunks depends on the purpose and endpoint, and it can vary by workspace and product. For video files, the maximum file size ranges from 4 GB to 16 GB.

Approach

All resumable uploads work through a double request pattern:

  • An OAuth-authenticated preflight request is made to get an upload token. (For example, a call is made to /api/photo/get-upload-token, which returns an upload_token property for later use.)
  • The token is used to authenticate subsequent upload requests to carry chunks of the file. (For example, chunks are uploaded to /api/photo/redeem-upload-token with upload_token=<upload_token>)

When a file is uploaded using the resumable pattern, chunks of that file in predefined sizes are uploaded in concurrent request. Any failed chunk upload request may be retried. When all the relevant chunks of the file are uploaded, the server will stitch together the full file and process it for upload.

All large-file uploads on the TwentyThree platform uses this same pattern, and to see the process in practice you may observe your browser’s network monitor while uploading a video file within the product.

The upload process sets a predefined chunk size, for example 1 MB and then splits up the upload based on that file’s total size and the chunk size. Every uploaded chunk must match this defined chunk size, except the last one – which must be at least the chunk size, but less than double of the chunk size.

The size of the last chunk is important: If your chunk size is 1 MB and the uploaded file totals 100.5 MB, you must have 100 chunks (not 101). This approach of an oversized last chunk increases the chance that we can read the video’s metadata from the first and last chunk alone.

Any resumable upload request to the TwentyThree platform must be a multi-part POST request carrying the chunk in in the file=... part of the form data. Additionally, all POST requests must include:

  • resumableChunkNumber: The index of the chunk in the current upload. First chunk is 1 (no base–0 counting here).
  • resumableTotalChunks: The total number of chunks.
  • resumableChunkSize: The general chunk size. Using this value and resumableTotalSize you can calculate the total number of chunks.
  • resumableTotalSize: The total file size.
  • resumableIdentifier: A unique identifier for the file being uploaded.
  • resumableFilename: The original file name.

The platform also allows previously stalled uploads to be resumed, by having the upload client test whether a given chunk has already been delivered prior to upload. This is helpful for low-bandwidth clients, and is covered in the Resumable.js documentation.

Step 1: Define chunk size and calculate chunk count

For any context supporting resumable uploads, there are two endpoints: One issues a token, and another receives the upload using that token. In this step-by-step guide, we’re using the video upload flow and assume a chunk size of 1 MB and a total file size of 100.5 MB.

resumableFilename    = "resumable-test-upload.mp4"
resumableTotalSize   = 105381888    ; // 100.5 MB
resumableChunkSize   = 1048576      ; // 1 MB
resumableTotalChunks = 
   floor(resumableTotalSize/resumableChunkSize)
   // =100
resumableIdentifier = 
  md5(resumableTotalSize + "-" + resumableFilename) 
  // ="403c8c362f30a2c973e5a514459f4baa"

Step 2: Get an upload token to start an upload

Call the /api/photo/get-upload-token endpoint to get a token:

GET https://video.mycompany.com/api/photo/get-upload-token
    ?format=json
    &raw
    &valid_minutes=60
    &title=A+test+resumable+upload
    &...

This responds with upload_token=f0kTcuIZykCdz13kzCwBE2L5HD1nKE8oNdxCaU7sjzYZfOgTSnTLbmPl9tDikkR0:

{
   "status":"ok",
   "message":"The upload token is ready to use",
   "permission_level":"write",
   "uploadtoken":{
      "description":"",
      "valid_minutes":"60",
      "max_uploads":"1",
      "title":"A test resumable upload",
      "user_id":"7520028",
      "valid_until":"1608587100",
      "publish_date":"",
      "upload_token":"f0kTcuIZykCdz13kzCwBE2L5HD1nKE8oNdxCaU7sjzYZfOgTSnTLbmPl9tDikkR0",
      "background_return_p":0,
      ...
   }
}

Step 3: Issue the upload for each chunk

Now, you will be ready to issue the chunks uploads to /api/photo/redeem-upload-token:

POST https://video.mycompany.com/api/photo/get-upload-token
    ?resumableChunkNumber: 1
    &resumableChunkSize: 1048576
    &resumableCurrentChunkSize: 1048576
    &resumableTotalSize: 105381888
    &resumableIdentifier: 403c8c362f30a2c973e5a514459f4baa
    &resumableFilename: resumable-test-upload.mp4
    &upload_token: f0kTcuIZykCdz13kzCwBE2L5HD1nKE8oNdxCaU7sjzYZfOgTSnTLbmPl9tDikkR0
    &file=<binary data>

Each request will return HTTP 200 upon completion. The last chunk upload will be this (note resumableCurrentChunkSize):

POST https://video.mycompany.com/api/photo/get-upload-token
    ?resumableChunkNumber: 100
    &resumableChunkSize: 1048576
    &resumableCurrentChunkSize: 1572864
    &resumableTotalSize: 105381888
    &resumableIdentifier: 403c8c362f30a2c973e5a514459f4baa
    &resumableFilename: resumable-test-upload.mp4
    &upload_token: f0kTcuIZykCdz13kzCwBE2L5HD1nKE8oNdxCaU7sjzYZfOgTSnTLbmPl9tDikkR0
    &file=<binary data>

Should any of these requests fail, the request should be retried after a back-off interval.

A plain, unstyled error message is returned if the upload_token is invalid, consumed or expired. Otherwise the client is redirected to return_url using the method specified in the flow description.

If background_return_p was set to 1 when retrieving the upload token, the callback to return_url is made from the server-side and one of the following information sets is returned:

ok <domain> <tree_id> <photo_id> <token> <callback_url>

Or:

error <error_message>

For more control over return and background notification upon upload completion, please refer to /api/photo/get-upload-token.