AWS S3 component

The aws2-s3 component uploads, downloads, lists, copies and deletes objects in Amazon S3 buckets, and polls a bucket for new objects.

Scheme: aws2-s3: · Crate: cameleer-cloud · Cargo feature: cloud · Consumer: yes · Producer: yes · Apache Camel: AWS S3 Storage Service

When to use it

• Archive files, reports or message payloads to S3.
• Read files that partners or other systems drop into a bucket, then move or delete them.
• Hand large documents to other services through presigned download or upload links.
• Collect sensor data on an edge device and upload it in multipart batches (streamingUploadMode).

URI format

aws2-s3://bucketNameOrArn[?options]

Example

Upload a report. The object key comes from the CamelAwsS3Key header (or the keyName option):

- route:
    id: upload-report
    from:
      uri: "timer:daily-report?repeatCount=1"
      steps:
        - setBody:
            constant: "region,total;eu,42;us,17"
        - setHeader:
            name: CamelAwsS3Key
            constant: reports/daily.csv
        - setHeader:
            name: CamelAwsS3ContentType
            constant: text/csv
        - to: "aws2-s3://{{env:S3_BUCKET:my-reports}}?region={{env:AWS_REGION:us-east-1}}&autoCreateBucket=true"
        - log: "uploaded s3://{{env:S3_BUCKET:my-reports}}/${header.CamelAwsS3Key}"

Poll a bucket for invoices. Each object becomes one exchange; after successful processing it is copied to processed-invoices and deleted from the source bucket:

- route:
    id: invoices-from-s3
    from:
      uri: "aws2-s3://incoming-invoices?region={{env:AWS_REGION:us-east-1}}&prefix=invoices/&moveAfterRead=true&destinationBucket=processed-invoices"
      steps:
        - log: "received ${header.CamelAwsS3Key} (${header.CamelAwsS3ContentLength} bytes)"
        - to: "log:invoices"

Credentials come from the default AWS chain; see Cloud credentials.

Rust DSL

ctx.add_routes(|b| {
    b.from("aws2-s3://incoming-invoices?region={{env:AWS_REGION:us-east-1}}&prefix=invoices/&moveAfterRead=true&destinationBucket=processed-invoices")
        .log("received ${header.CamelAwsS3Key} (${header.CamelAwsS3ContentLength} bytes)")
        .to("log:invoices");
})?;

Options

Common AWS options (region, credentials, endpoint override, proxy) are listed in Cloud credentials.

The consumer also accepts the scheduled poll options (delay default 500 ms, initialDelay, greedy, scheduler.cron, ...).

Message headers

Run it with Docker

docker run --rm -e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e S3_BUCKET \
  -v "$PWD:/routes:ro" pmsudhi/cameleer run /routes/s3.camel.yaml

Common questions

How do I upload a large file without loading it in memory?

Set the CamelAwsS3SourceFile header to the file path. With multiPartUpload=true the file is uploaded one part at a time; otherwise it is streamed in a single PUT.

How do I create a presigned download link?

Use operation=createDownloadLink with CamelAwsS3Key; the body becomes the URL, valid for CamelAwsS3DownloadLinkExpirationTime milliseconds.

How do I keep the objects in the bucket after reading them?

Set deleteAfterRead=false. The consumer then sees the same objects again on every poll, so usually you combine it with an idempotent consumer or move the objects with moveAfterRead.

Does it work with MinIO or LocalStack?

Yes: uriEndpointOverride=http://localhost:9000&overrideEndpoint=true (path-style addressing is used automatically with an override).

Related

• AWS SQS, AWS Textract, AWS Rekognition
• Azure Storage Blob
• Cloud credentials
• Apache Camel AWS S3

Found a problem?

If something on this page doesn't match what Cameleer does, or you hit a bug while building with it, please help fix it, whether you are a person or a coding agent: open an issue that describes what you saw, then fork the repository, fix it, add or update a test, run cargo test --workspace, and open a pull request. CONTRIBUTING.md explains the ground rules.