File and SFTP pipelines that run in megabytes

Q: Which file protocols does Cameleer support?

Local folders (file), FTP and FTPS (explicit and implicit TLS, tested against vsftpd), and SFTP over Cameleer's own SSH-2 implementation, tested against OpenSSH, with password or OpenSSH and PEM key authentication and known_hosts checking. S3, Azure Blob and Data Lake, Google Cloud Storage and Google Drive cover object storage.

Q: Which file formats can it parse?

CSV (csv and univocity), fixed-length and key-value records (bindyFixed, bindyKvp, bindyCsv), JSON, XML, YAML, Avro, Protobuf, zip, tar and gzip archives, Base64, PGP and symmetric crypto, HL7 v2 and more; 29 data formats in all.

Q: How are processed files handled?

As in Apache Camel: move to a done folder, moveFailed to a failed folder, or delete, with include and exclude patterns, read locks and idempotent repositories so a file is processed once.

Partner files still arrive by SFTP at 02:00. Cameleer polls the folder or the server, parses CSV and fixed-width records, splits, validates, deduplicates, loads into PostgreSQL or S3, and moves each file to done or failed. The routes are Apache Camel's file patterns; the runtime is one binary.

2.1xApache Camel's CSV batch throughput: 998 vs 468 files/s

8 MBpeak memory on that batch, against 337 MB on the JVM

SSH-2and SFTP written from scratch, tested against OpenSSH; FTP and FTPS against vsftpd

29data formats: CSV, fixed-width, JSON, XML, Avro, zip, PGP, HL7 and more

The tested route: files to S3

From examples/files-to-s3.camel.yaml, which the test suite runs against a mock S3. Swap file: for sftp: and the route polls a partner's server instead.

- route:
    id: files-to-s3
    from:
      uri: "file:{{env:INBOX_DIR:inbox}}?include=.*\\.csv&move=.done&moveFailed=.failed"
      steps:
        - setHeader:
            name: CamelAwsS3Key
            simple: "${file:name}"
        - to: "aws2-s3://{{env:S3_BUCKET:my-bucket}}?region={{env:AWS_REGION:us-east-1}}&useDefaultCredentialsProvider=true"
        - log: "uploaded ${file:name} to s3://{{env:S3_BUCKET:my-bucket}}"

The same shape, other ends

Pick up from SFTP: sftp://user@host/outbox?privateKeyFile=/routes/id_ed25519&knownHostsFile=/routes/known_hosts&strictHostKeyChecking=yes&delete=true&delay=30000
Parse and load: unmarshal: csv (or bindyFixed for fixed-width), split by row, to: sql:insert into ... with named parameters.
Batch the writes: aggregate with completionSize: 100 and one insert per batch.
Reject bad rows: validate or json-validator, and a dead letter channel to a rejects folder the partner can fetch.

Components and formats

file, ftp, ftps, sftp; aws2-s3, azure-storage-blob, google-storage, google-drive
csv, univocity, bindy, zipfile, tarfile, gzip, pgp
sql for PostgreSQL, kafka for downstream consumers
EIPs: split and aggregate, idempotent consumer, dead letter channel

The CSV batch, measured

A 1,000-row file is split into rows; EU rows are kept, reshaped and written out in batches of 100. The same route on three runtimes, on identical machines.

runtime	files/s	p99 per file	peak memory
Cameleer	998	1.7 ms	8 MB
Apache Camel (JVM)	468	3.2 ms	337 MB
Apache Camel (Quarkus native)	163	12.5 ms	126 MB

From the cloud benchmark; method, machines and raw data in how we tested and BENCHMARKS.

What operations teams ask

Exactly-once-ish file handling

move, moveFailed and delete after processing; include and exclude patterns; read locks for files still being written.
An idempotent repository keyed by file name (in memory, JDBC or Redis) so a re-uploaded file is skipped, or processed again on purpose.
sftp checks the host key against known_hosts with strictHostKeyChecking=yes.

Scheduling and scale

Poll every delay milliseconds, back off when idle, or start on a cron expression.
Run as a cron-triggered container, a Kubernetes CronJob, or an AWS Lambda on an EventBridge schedule; the binary starts in milliseconds, so there is no warm pool to keep.
One binary per partner feed costs tens of MB, not hundreds.

Common questions

Which file protocols does Cameleer support?

Local folders (file), FTP and FTPS (explicit and implicit TLS, tested against vsftpd), and SFTP over Cameleer's own SSH-2 implementation, tested against OpenSSH, with password or OpenSSH and PEM key authentication and known_hosts checking. S3, Azure Blob and Data Lake, Google Cloud Storage and Google Drive cover object storage.

Which file formats can it parse?

CSV (csv and univocity), fixed-length and key-value records (bindyFixed, bindyKvp, bindyCsv), JSON, XML, YAML, Avro, Protobuf, zip, tar and gzip archives, Base64, PGP and symmetric crypto, HL7 v2 and more; 29 data formats in all.

How are processed files handled?

As in Apache Camel: move to a done folder, moveFailed to a failed folder, or delete, with include and exclude patterns, read locks and idempotent repositories so a file is processed once.

How fast is a CSV batch?

In the cloud benchmark a route that splits a 1,000-row file, keeps the EU rows, reshapes them and aggregates batches of 100 processed 998 files per second on Cameleer against 468 on Apache Camel 4.23 (JVM) and 163 on Camel Quarkus native, with 8 MB of peak memory against 337 MB.

Does it parse EDI (EDIFACT, X12)?

There is no dedicated EDIFACT or X12 data format. Delimited and fixed-width partner files are handled with csv, univocity and bindy; segment-based EDI needs a tokenize split and your own mapping.

Run it

Drop a CSV file into ./inbox and watch it land in the bucket and move to .done.

export S3_BUCKET=my-bucket AWS_REGION=eu-west-1 AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
docker run --rm -e S3_BUCKET -e AWS_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  -e INBOX_DIR=/routes/inbox --user "$(id -u):$(id -g)" -v "$PWD:/routes" \
  pmsudhi/cameleer run /routes/files-to-s3.camel.yaml

See the route The sftp component The csv data format