Docs
Quickstart and file format
What exists today: the local CLI and the file format. The hosted service, API, MCP server and web console are planned; this page does not describe them as if they were live.
Quickstart
The CLI is not published yet; in the prototype it is built from source. Create a pipeline, check it, run it:
terminal
waddle init orders-sync # writes pipeline.toml and pipeline.sql
waddle validate orders-sync/pipeline.toml
waddle run orders-sync/pipeline.toml --state orders-sync/stateThe state file stores the watermark between runs. It is written only after the sink succeeded and the checks passed.
Secrets and environment
Configs never contain secrets. A connector names an environment variable (dsn_env) and the runner reads it at start. A config with an inline password field is rejected.
environment
export SRC_DSN='host=db.example.com user=reader dbname=app'
export WADDLE_S3_ENDPOINT=<account>.r2.cloudflarestorage.com # S3 or R2 sinks and sources
export WADDLE_S3_KEY_ID=...
export WADDLE_S3_SECRET=...Pipeline file reference
| Field | Meaning |
|---|---|
name | Required. 1 to 64 characters of letters, digits, _ and -. |
sql | Required. Path to the SQL file, relative to the config. No absolute paths, no "..". |
schedule | Optional. Five-field cron expression, used by the hosted service. |
[source] kind = "file" | url (https:// or s3://) and format (csv, json or parquet). Plain http and local paths are refused. |
[source] kind = "postgres" | dsn_env (the name of an environment variable holding the connection string) and table (name or schema.name). |
[sink] kind = "file" | url (https:// or s3://) and format. |
[sink] kind = "postgres" | dsn_env, table, mode (append, replace or upsert) and key (required for upsert). |
[incremental] | column (output column whose maximum becomes the next watermark) and initial (the value for the first run). |
[[checks]] | rule = not_null or unique (column), row_count (min, max) or reconcile (tolerance, default 0). |
Unknown fields are errors. Table and column names must be plain SQL identifiers.
Writing the SQL
- The source is loaded as a table named
src. Write one SELECT over it; the result is what the sink receives. - Incremental pipelines filter with the bound parameter
$watermark. It is passed as a value, never pasted into the SQL, so it cannot inject code. The SQL must use it if[incremental]is set, and must not if it is not. - Comments are allowed, including a trailing one. Only one statement is accepted.
- Your SQL runs after the machine is locked down: it cannot read files, reach the network, attach databases or load extensions.
Checks
not_nullanduniqueinspect the query result before anything is written. A failure stops the run and leaves the sink and the watermark untouched.row_countbounds the number of result rows. On an incremental pipeline a run with no new rows has zero, so omitminthere.reconcileruns after the write. For an upsert sink it checks that every key landed; for a replace sink or a file it compares row counts after reading the sink back.