Tokenize language

The tokenize language cuts text into a list of pieces (by a token, a regular expression, a start and end token, or an XML element name), mostly to drive the split EIP.

Name: tokenize · Crate: cameleer-core · Cargo feature: none, always included (build features) · Apache Camel: Tokenize

When to use it

• Process a file line by line: split with tokenize: "\n".
• Send records in batches, for example 100 lines per request to an embedding API, with group.
• Skip a CSV header line with skipFirst.
• Cut repeated XML elements out of a large document (xml: true) without parsing it as a whole.

Syntax

The expression text, if any, is the name of a header to tokenize; otherwise the body is used. Everything else is set with options. In YAML, tokenize: "," is short for tokenize: {token: ","}.

group: N joins every N pieces with groupDelimiter (default: the token in token mode, nothing in pair and xml mode). skipFirst: true drops the first piece. \n, \r and \t in token, endToken and groupDelimiter are unescaped. A body that is already a list is passed through (only skipFirst applies).

Examples

Lines in groups, skipping a header

- route:
    id: tokenize-lines
    from:
      uri: direct:tokenize-lines
      steps:
        - split:
            tokenize:
              token: "\\n"
              skipFirst: true
              group: 2
            aggregationStrategy: GroupedBodyAggregationStrategy
            steps:
              - setBody:
                  simple: "batch[${body.replace('\n', '|')}]"

id,name
1,Ada
2,Grace
3,Linus

["batch[1,Ada|2,Grace]","batch[3,Linus]"]

Regular expression

- route:
    id: tokenize-regex
    from:
      uri: direct:tokenize-regex
      steps:
        - setBody:
            tokenize:
              token: "\\s*[;,]\\s*"
              regex: true

red ; green,blue  ,  black

["red","green","blue","black"]

Text between start and end tokens

Pull the code blocks out of a model's Markdown answer:

- route:
    id: tokenize-pair
    from:
      uri: direct:tokenize-pair
      steps:
        - setBody:
            tokenize:
              token: "<sql>"
              endToken: "</sql>"

First run <sql>SELECT 1</sql>, then <sql>SELECT 2</sql>.

["SELECT 1","SELECT 2"]

XML elements

- route:
    id: tokenize-xml
    from:
      uri: direct:tokenize-xml
      steps:
        - split:
            tokenize:
              token: "<order>"
              xml: true
            aggregationStrategy: GroupedBodyAggregationStrategy
            steps:
              - setBody:
                  xpath:
                    expression: /order/@id
                    resultType: String

<orders><order id="1"><line/></order><order id="2"/><order id="3">x</order></orders>

["1","2","3"]

Tokenize a header

- route:
    id: tokenize-header
    from:
      uri: direct:tokenize-header
      steps:
        - setBody:
            tokenize:
              expression: recipients
              token: ";"

ignored

["ada@example.com","grace@example.com"]

Options

Rust DSL

b.from("direct:tokenize-rust")
    .split(tokenize("\n"))
    .to("mock:tokenize-line")
    .end()
    .set_body(tokenize_pair("[", "]", false));

Also tokenize_regex(regex) and tokenize_xml(tag); add group/skipFirst with LanguageOptions::with_option.

Common questions

How do I split a file into lines?

split: {tokenize: "\n"}. Add group: 100 to process 100 lines per exchange, and skipFirst: true to skip a header line.

How do I split a big XML file into records?

tokenize: {token: "<record>", xml: true} returns each <record> element as text, which you can then query with XPath.

Does Cameleer stream the tokens?

No. The tokens of one body are computed as a list, so the whole body is held in memory. This differs from Apache Camel, where tokenize can scan a stream lazily.

Related

• Split EIP, XPath language, Simple language (${split(...)}, ${collate(n)})
• CSV data format
• Apache Camel Tokenize language

Found a problem?

If something on this page doesn't match what Cameleer does, or you hit a bug while building with it, please help fix it, whether you are a person or a coding agent: open an issue that describes what you saw, then fork the repository, fix it, add or update a test, run cargo test --workspace, and open a pull request. CONTRIBUTING.md explains the ground rules.