Google Cloud Storage component

The google-storage component uploads, downloads, lists, copies and deletes objects in Google Cloud Storage buckets, and consumes new objects from a bucket.

Scheme: google-storage: · Crate: cameleer-cloud · Cargo feature: cloud · Consumer: yes · Producer: yes · Apache Camel: Google Storage

When to use it

• Upload files, reports or exports to a bucket ("write files to GCS").
• Pick up objects that land in a bucket and process them, then delete or archive them.
• Ship sensor data or logs from an edge device to the cloud: a small Cameleer binary watches a local folder and uploads each file, retrying on failure.
• Hand out time-limited download links (V4 signed URLs) to users or AI agents without making the bucket public.

URI format

google-storage://bucketName[?options]

The bucket is created on start when it does not exist (autoCreateBucket=true, in the project of the service account key).

Example

Upload every file that appears in a local folder, under its own file name:

- route:
    id: files-to-gcs
    from:
      uri: "file:{{env:INBOX_DIR:inbox}}?move=.done"
      steps:
        - setHeader:
            name: CamelGoogleCloudStorageObjectName
            simple: "uploads/${file:name}"
        - to: "google-storage://{{env:GCS_BUCKET:my-bucket}}"
        - log: "uploaded ${file:name} to gs://{{env:GCS_BUCKET:my-bucket}}/uploads/"

Consume the objects of a bucket: each object becomes one exchange with its content as the body, and after processing it is copied to an archive bucket and deleted from the source bucket:

- route:
    id: gcs-incoming
    from:
      uri: "google-storage://{{env:GCS_BUCKET:incoming-bucket}}?prefix=orders/&moveAfterRead=true&destinationBucket={{env:GCS_ARCHIVE_BUCKET:archive-bucket}}"
      steps:
        - log: "received ${header.CamelGoogleCloudStorageObjectName} (${header.CamelGoogleCloudStorageContentLength} bytes)"
        - setHeader:
            name: CamelFileName
            simple: "${header.CamelGoogleCloudStorageObjectName}"
        - to: "file:{{env:OUT_DIR:out}}"

Rust DSL

ctx.add_routes(|b| {
    b.from("file:inbox?move=.done")
        .set_header("CamelGoogleCloudStorageObjectName", simple("uploads/${file:name}"))
        .to("google-storage://my-bucket")
        .log("uploaded ${file:name}");
})?;

Options

Credentials

These options are shared by every Google component (google-pubsub, google-bigquery, google-vertexai, ...).

Tokens are cached per credential until shortly before they expire, and refreshed after a 401.

Message headers

Bodies: putObject returns the object metadata map, getObject the content bytes, listObjects a list of metadata maps, copyObject the new object's metadata, deleteObject and deleteBucket true.

Run it with Docker

docker run --rm -v "$PWD:/routes:ro" -v "$PWD/inbox:/inbox" \
  -v "$HOME/keys/sa.json:/keys/sa.json:ro" -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
  -e GCS_BUCKET=my-bucket -e INBOX_DIR=/inbox \
  pmsudhi/cameleer run /routes/files-to-gcs.camel.yaml

Common questions

How do I authenticate to Google Cloud Storage?

Set GOOGLE_APPLICATION_CREDENTIALS to a service account key file, or pass serviceAccountKey on the URI. On GCE, GKE or Cloud Run use useMetadataServer=true and no key at all.

How do I download a single object?

Send any message with operation=getObject and the CamelGoogleCloudStorageObjectName header; the body becomes the object bytes.

How do I create a signed download URL?

Use operation=createDownloadLink with the object name header. It needs a service account key (the URL is signed with it, GOOG4-RSA-SHA256); the body is the URL.

How is this different from Apache Camel?

Options and headers follow Camel. Resumable uploads (resumableUploadThreshold, resumableChunkSize) are Cameleer additions; the filter and downloadFileName options are accepted but ignored.

Related

• Google Pub/Sub, BigQuery (load jobs read gs:// files)
• AWS S3 for the same pattern on AWS
• Apache Camel Google Storage

Found a problem?

If something on this page doesn't match what Cameleer does, or you hit a bug while building with it, please help fix it, whether you are a person or a coding agent: open an issue that describes what you saw, then fork the repository, fix it, add or update a test, run cargo test --workspace, and open a pull request. CONTRIBUTING.md explains the ground rules.