How Ingestion Works

This page explains how Matia ingests data in ETL integrations: the path from source to destination, how sync modes and schema changes work, and why emitted and committed record counts can differ.

Source → Matia → Destination

An ETL integration reads from a source (e.g. Postgres, Salesforce) and writes to a destination (e.g. Snowflake, BigQuery). You choose which tables to sync in the Schema tab and how each table is synced (full refresh, incremental, or append-only). Matia uses connector-specific logic to extract only the data that needs to be moved (for incremental, based on a cursor column). Data is normalized and transformed as required (e.g. flattening nested structures, type coercion). The result is loaded into the destination in the configured schema or database.

Emitted vs Committed Records

• Emitted records: Rows that Matia reads from the source and processes for the sync.
• Committed records: Rows that are successfully written and confirmed in the destination.

The two counts can differ because Matia applies transformations, flattening, and normalization. One source row can become multiple destination rows, or multiple source rows can be merged. Deduplication or filtering can also change counts. The Status tab shows both numbers per sync and per table so you can verify that data movement matches expectations.

Sync Modes and Cursors

• Full refresh: Every sync reads the full table from the source and replaces or overwrites the destination. No cursor is used.
• Incremental: After the initial load, only rows that are new or updated since the last sync are read. This requires a cursor field (e.g. updated_at, id) that increases over time. Matia stores the last cursor value and uses it on the next run. Support depends on the connector.
• Append only: Only new rows are read; existing destination rows are not updated or deleted. Suited to event or log data.

When the source does not expose a suitable cursor, Matia falls back to full refresh for that table.

Schema Changes

When the source adds or removes schemas, tables, or columns, Matia can detect these changes. Your Schema changes setting (in Configure or Settings) controls whether the integration adopts them:

• Enable all: New schemas, tables, and columns are added automatically.
• Enable new columns only: Only new columns on existing tables are added; new schemas and tables are ignored.
• Ignore all: No new structures are added; the integration keeps syncing only what was previously configured.

The Schema changes tab (when available) lists detected changes (tables/columns added, removed, or type-altered) so you can review before or after adjusting the global setting.

For authoritative definitions, see Sync modes and Sync logs and run history.