MongoDB GridFS component

The mongodb-gridfs component stores, reads, lists and deletes files in MongoDB GridFS buckets, and polls a bucket for new files.

Scheme: mongodb-gridfs: · Crate: cameleer-datastores · Cargo feature: datastores · Consumer: yes · Producer: yes · Apache Camel: MongoDB GridFS

When to use it

• Keep documents, images or model outputs (generated reports, audio, embeddings dumps) next to the MongoDB data that refers to them.
• Archive files dropped into a folder by an edge device or a scanner.
• Serve stored files over HTTP, or hand every new upload to a processing route.

URI format

mongodb-gridfs:<connectionBean>?database=<db>[&bucket=fs][&operation=create][&options]

The connection is connectionUriString=mongodb://host:27017 (the path is then just a name) or a MongoClient bound in the registry under the path name. Files use the standard GridFS layout (<bucket>.files and <bucket>.chunks), so files written by other drivers can be read and the other way round.

Example

This route uploads every file dropped into a folder into the reports bucket. The file component sets CamelFileName, which becomes the GridFS file name.

- route:
    id: archive-reports
    from:
      uri: "file:{{env:INBOX_DIR:inbox}}?move=.done"
      steps:
        - to: "mongodb-gridfs:archive?connectionUriString={{env:MONGODB_URI:mongodb://localhost:27017}}&database=docs&bucket=reports&operation=create"
        - log: "stored ${header.CamelFileName} as ${header[gridfs.fileid]}"

GET /reports?name=q3.csv returns the newest file with that name.

- route:
    id: download-report
    from:
      uri: "platform-http:/reports?httpMethodRestrict=GET"
      steps:
        - setHeader:
            name: CamelFileName
            simple: "${header.name}"
        - to: "mongodb-gridfs:archive?connectionUriString={{env:MONGODB_URI:mongodb://localhost:27017}}&database=docs&bucket=reports&operation=findOne"

Rust DSL

ctx.add_routes(|b| {
    b.from("file:inbox?move=.done")
        .to("mongodb-gridfs:archive?connectionUriString=mongodb://localhost:27017&database=docs&bucket=reports");
    b.from("mongodb-gridfs:archive?connectionUriString=mongodb://localhost:27017&database=docs&bucket=reports&queryStrategy=FileAttribute")
        .to("file:outbox");
})?;

Operations

Options

readPreference, writeConcern, persistentTSCollection and persistentTSObject are accepted and ignored. PersistentTimestamp is treated as TimeStamp: the timestamp is kept in memory, so after a restart the TimeStamp strategy only sees files uploaded after the new start. Use FileAttribute when every file must be processed exactly once across restarts.

Message headers

Run it with Docker

docker run --rm -e MONGODB_URI=mongodb://mongo.example.com:27017 -e INBOX_DIR=/inbox \
  -v "$PWD/inbox:/inbox" -v "$PWD:/routes:ro" pmsudhi/cameleer run /routes/archive.camel.yaml

The inbox is mounted read-write because move=.done moves processed files.

Common questions

How do I store metadata with a file?

Set the gridfs.metadata header to a JSON object, for example {"customer": "acme"}. It is saved in the file's metadata field and returned by findOne.

How do I process every new file once?

Consume with queryStrategy=FileAttribute: each file is marked in its metadata while and after it is routed, so it is not picked up again, even after a restart.

What chunk size is used?

261120 bytes (255 KiB), like the MongoDB drivers. Override it per file with gridfs.chunksize.

Related

• mongodb: documents and queries.
• file: read and write local files.
• Apache Camel MongoDB GridFS component

Found a problem?

If something on this page doesn't match what Cameleer does, or you hit a bug while building with it, please help fix it, whether you are a person or a coding agent: open an issue that describes what you saw, then fork the repository, fix it, add or update a test, run cargo test --workspace, and open a pull request. CONTRIBUTING.md explains the ground rules.