Every data pipeline works on the day you build it. The interesting question is what it does on the Tuesday nine months later when a vendor adds a field to an API response, renames another, and changes a string column to an object.
That is the actual job. Moving rows from A to B is the easy part and has been a solved problem for two decades. What you are buying, when you buy an ETL tool, is somebody else’s commitment to keep a few hundred connectors working against APIs that change without telling anyone, and a defined behaviour for the moment one of them changes in a way nobody anticipated.
Teams evaluate these tools on connector count, which is the least informative number available. A catalogue of five hundred connectors tells you nothing about whether the six you need are maintained, how deep their coverage of each source goes, or what happens when one breaks at 2am on the night before the board deck is due. Two connectors with the same name from two vendors can differ enormously in which objects they sync, whether they support incremental extraction, whether they capture deletes, and whether they handle a source that paginates inconsistently.
Then there is the bill. Per-row pricing is the dominant commercial model in this category, and it has a structural property that nobody explains during the sales process: your invoice is determined by your sources’ behaviour, not by your own. A source system that touches an updated-at column on every row during a nightly maintenance job makes every row active, and your bill reflects a full re-sync you did not ask for.
Key takeaways
- Connector count is a vanity metric. What matters is the maintenance status, object coverage and incremental-sync support of the specific connectors you need, and that is per-connector information you have to check individually.
- Per-row pricing makes your bill a function of your sources’ update behaviour. A bulk update upstream, a schema change that triggers a full refresh, or a soft-delete pattern can multiply a month’s cost without any change on your side.
- Schema evolution handling is the single biggest reliability difference between these tools, and the options are genuinely different products: propagate silently, halt the sync, quarantine the rows, or ignore the new field entirely.
- Extraction and transformation are separate concerns with separate failure modes. Buying one tool that claims both usually means you are getting a good version of one of them.
Connector breadth is a maintenance bill somebody pays
A connector is a small program that knows one source’s authentication, pagination, rate limits, object model, incremental cursor semantics and error behaviour. Every one of those can change without notice, and every one of them has to be fixed by somebody when it does.
There are three ways that maintenance gets paid for, and they produce three very different experiences.
The vendor pays, and charges you. A commercial connector catalogue where the vendor employs people to monitor API changes and ship fixes. You get a support channel and an SLA-shaped promise. You pay a premium per row moved, and you have no recourse when the connector does not cover the object you need beyond filing a request and waiting.
The community pays, mostly. An open catalogue where connectors are contributed by users and the vendor certifies a subset. The certified ones are maintained. The long tail varies from actively maintained to abandoned, and the catalogue does not always make the difference obvious. This is where the connector-count number does the most damage, because a large fraction of a large catalogue can be community connectors that nobody has touched in a year.
You pay, with your own engineers. You write and maintain the connector. Entirely reasonable for one or two important sources with stable APIs, and completely unsustainable as a general strategy, because the cost is not the initial build. It is the ongoing obligation to notice when something changes, which nobody budgets for and which lands on whoever wrote it originally.
The practical evaluation step nobody does: take your list of sources, and for each one open the actual connector documentation and answer four questions. Which objects does it sync, and is the one you need among them? Does it support incremental extraction, or does it re-read everything each run? Does it capture deletes, or do deleted records live forever in your warehouse? When was it last meaningfully updated?
That exercise takes an afternoon and eliminates most of the disagreement in a tool selection. It also frequently reveals that your hardest source is unsupported everywhere and you were always going to write that one yourself.
Schema evolution is where pipelines actually break
A source changes its schema. Your pipeline has to decide what to do. There are only a few possible behaviours and the choice between them is the single most consequential thing about an ELT tool.
A new column appears. The friendly case. The tool can add the column to the destination and start populating it, or ignore it until you explicitly enable it. Propagating automatically means your destination schema changes without a code review, which matters if downstream models do SELECT *. Ignoring it means data silently not arriving until someone notices, which is worse in a different way. Most tools let you configure this per connection and most teams never look at the setting.
A column’s type changes. The genuinely hard case. A field that was an integer is now a string, or a string is now a nested object. The destination column cannot hold both. The options are: widen the destination column to a permissive type, create a new column with a type suffix and split the data across two columns, fail the sync, or quarantine the offending rows. Every one of those breaks something downstream, and the difference between tools is which thing they break and whether they tell you.
A column disappears. Usually handled by leaving the destination column in place and filling nulls, which is right. The problem is that nothing distinguishes “this field is now absent from the source” from “this field is legitimately null for these records”, so a silently removed field looks like a data quality issue rather than a schema change.
A column is renamed. Indistinguishable from one column being dropped and another added, because the wire format carries no rename information. Your destination now has an old column full of history and a new column starting from today, and reconciling them is manual work.
A primary key changes. The worst case and the one that triggers a full re-sync in most tools, because incremental state is keyed on the old identity. This is also the most common cause of a surprise invoice under per-row billing.
Two design points worth building into whatever you choose.
Land raw, transform separately. If the extraction layer writes source data to the warehouse with minimal reshaping and all transformation happens afterwards in SQL you control, then a schema change breaks a model you own, with a diff and a test, rather than a vendor component you cannot inspect. This is the actual argument for ELT over ETL, and it is a better argument than the performance one people usually give.
Test the contract, not the pipeline. Assertions on the shape of landed data (this column is never null, this key is unique, this enum has these values) catch schema drift at the point it arrives rather than three models downstream when a dashboard shows a number that is wrong but plausible. Wrong-but-plausible is the expensive failure. A broken pipeline is an alert; a silently wrong number is a decision.
Needs first-hand data: For each of your top five sources, deliberately introduce each of the four schema changes above in a sandbox account (add a field, change a field’s type, remove a field, rename a field) and record exactly what your candidate tool does: does the sync fail, does it warn, does it silently propagate, does it trigger a full refresh, and how long until anyone would have noticed. Run the same five changes against each candidate. This is a day of work and it is the only evaluation that predicts your on-call experience.
Per-row pricing turns a source bug into an invoice
The dominant commercial meter in this category counts distinct rows changed and synced in a billing period. Sometimes it is called monthly active rows, sometimes credits derived from row volume, but the mechanism is the same: a row that changed and got synced is billable, and a row that did not is not.
The intent is reasonable. It aligns your cost with the work done. The problem is that “changed” is decided by the source system, not by you, and several common upstream behaviours make everything change.
Bulk updates upstream. A migration, a backfill, a data quality fix, or a vendor’s own maintenance job that rewrites a timestamp on every record. Every row is now active. You did nothing and your bill for that month is a full-table sync.
Incremental cursors that are not selective. If a connector uses an updated-at column as its cursor, and the source updates that column on records for reasons unrelated to the fields you care about, you resync rows whose meaningful content is unchanged. You pay for movement, not for information.
Full refreshes triggered by schema changes. Several of the schema changes above force a full re-sync to rebuild state. A primary key change on a large table is an expensive month, and the trigger was a decision made in somebody else’s product.
Soft deletes. A source that marks records deleted rather than removing them keeps them in the active set forever, and a bulk archival operation marks a large fraction of your history as changed at once.
Frequent sync schedules on slow-changing data. This one cuts the other way and is worth knowing: if the meter counts distinct rows per period rather than per sync, syncing hourly instead of daily does not multiply the cost for a row that changed once. If the meter counts per-sync volume, it does. Ask which, because it changes the right sync frequency by a factor of twenty-four.
The defences are unglamorous and effective. Sync only the tables and columns you actually use, which is usually a much smaller set than the default selection. Set sync frequency per table by how fresh that table genuinely needs to be, rather than setting everything to the fastest option. And put a monitor on row volume per connection per day, with an alert on a step change, so a bulk update upstream is a notification rather than a discovery at the end of the month.
Open-source self-hosted tools remove this meter and replace it with a compute bill plus an engineer. That is a genuinely better trade above a certain data volume and a genuinely worse one below it, and the crossover point depends on what your engineers cost relative to your row counts. The same build-versus-buy shape appears one layer down in the CDC tooling decision.
Singer: a spec for taps and targets, not a product
Several tools in this category either implement or descend from Singer, and it is worth understanding what it is before the product blocks, because it is not something you can buy.
What it gives you
Singer is a specification for how data extraction programs communicate. A tap reads from a source and writes a stream of newline-delimited JSON messages to standard output. A target reads that stream from standard input and writes it to a destination. Three message types carry everything: SCHEMA describes the structure of a stream as a JSON Schema, RECORD carries one row, and STATE carries a checkpoint that the tap can be given on its next run to resume incrementally.
That is the entire contract, and its simplicity is the point. Any tap composes with any target, in any language, connected by a Unix pipe. A tap is a normal program you can run in a terminal and watch. Debugging is reading the messages. The STATE message means incremental sync is a first-class part of the protocol rather than each connector’s private invention.
Because the interface is that small, a very large number of community taps and targets exist, and writing a new one for an unusual internal source is a genuinely small project rather than a framework exercise.
What it does not do
It has no scheduler, no orchestration, no retry policy, no monitoring, no credential storage and no concurrency model. A tap and a target connected by a pipe is one sync run; everything about making that run reliably every hour is outside the spec and is exactly what the products below sell.
It also says nothing about quality. Any tap conforming to the message format is a valid Singer tap, including one that was written for a hackathon four years ago and no longer works. The spec’s permissiveness is why the ecosystem is large and why the ecosystem is uneven. And the JSON message format itself is verbose, which puts a practical ceiling on throughput compared with binary or columnar transport for very large tables.
Fivetran

The commercial reference point in this category: a fully managed extraction service with a maintained connector catalogue, automatic schema handling, and no infrastructure for you to run. It lands normalised source data in your warehouse and leaves transformation to you, which is the ELT position, and it integrates with SQL-based transformation frameworks for the second half.
What you are actually buying is the promise that when a source API changes, somebody who is not you fixes the connector. For a small data team that promise is worth a great deal, and it is priced accordingly.
Pros
- Connectors are vendor-maintained with a support channel, so a broken source is somebody else’s escalation rather than your evening
- Schema change handling is automatic and documented per change type, which removes the most common source of silent pipeline failure
- Zero infrastructure to run, which for a team without a data platform engineer is the difference between a pipeline existing and not
- Deep coverage within its supported sources, including objects and incremental strategies that thinner connectors skip
Cons
- Per-row pricing makes your bill a function of upstream behaviour you do not control, and a bulk update at a vendor is an invoice event
- No practical way to modify a connector, so a missing object or an unsupported field is a feature request with no workaround
- Cost scales with data volume in a way that becomes hard to justify once you have the engineers who could maintain the same connections
Best for: Small data teams without a dedicated platform engineer, syncing a handful of well-supported SaaS sources where reliability matters more than unit cost.
Pricing: Metered by monthly active rows, with the rate stepping down at higher volumes and a separate meter for some source categories.
Airbyte

An open-source extraction platform with a large connector catalogue, available both self-hosted and as a managed cloud service. Connectors run as containers with a defined interface, which means you can fork one, fix it, and run your version, or build a new one with a low-code connector builder for simple REST APIs.
That modifiability is the actual differentiator. When a connector does not cover the object you need, you are not filing a ticket, you are editing code. That is worth more than the connector count to any team that has engineers.
Pros
- Connectors are containers you can fork and modify, which converts a blocking vendor limitation into an afternoon of work
- Self-hosting removes per-row billing entirely and replaces it with compute you already pay for
- Large catalogue covering long-tail sources that commercial vendors have not prioritised
- Low-code connector builder makes simple REST sources genuinely quick to add without learning a framework
Cons
- Connector quality varies enormously between certified and community connectors, and the catalogue does not make the practical difference as visible as it should
- Self-hosting is a real operational commitment: it is a distributed system with a database, a scheduler and worker containers, not a binary you run
- The platform has been through significant architectural changes, so community knowledge and older documentation frequently describe a version you are not running
Best for: Teams with engineering capacity who need long-tail or modified connectors and would rather own the infrastructure than the per-row bill.
Pricing: Open-source and free to self-host with no licence fee; the managed cloud meters by rows or by compute credits depending on connector type.
Meltano

A command-line data integration tool built around the Singer specification, treating your entire pipeline as a versioned project directory: a YAML configuration naming which taps, targets and transformation steps make up each pipeline, with plugins installed into isolated environments. Everything is a file in Git, which means pipeline changes go through code review like anything else.
It is the option that most fully treats data integration as software engineering rather than as a configured service, and that framing is either exactly what you want or exactly what you do not.
Pros
- The whole pipeline is a Git-versioned project, so changes are reviewable, diffable and revertable like application code
- Built on Singer, so the enormous tap and target ecosystem is available and a custom tap is a small program
- Runs locally exactly as it runs in production, which makes debugging a pipeline a normal development activity
- No per-row meter and no vendor coupling of any kind
Cons
- No scheduler, no orchestration and no monitoring: you must bring your own, and the assembled system is yours to operate
- Inherits Singer’s quality variance, so evaluating each tap individually is unavoidable work
- Command-line and configuration-file driven with no operational UI, which makes it a poor fit for teams where analysts need to manage connections
Best for: Engineering-led data teams who want pipelines in version control and are already running an orchestrator to schedule them.
Pricing: No vendor and no bill. Meltano is an open-source project; your costs are the compute it runs on and the engineer who maintains the project.
dbt

Not an extraction tool at all, and included because half the teams evaluating this category are actually looking for this. dbt transforms data that is already in your warehouse: you write SQL SELECT statements as models, it builds the dependency graph from the references between them, and it materialises each one as a table or view in the right order. Tests, documentation and lineage come from the same project.
It occupies the T in ELT, and the split matters: extraction tools land raw data, dbt turns it into the models your business actually queries.
Pros
- Transformation logic lives in version control as plain SQL with a dependency graph inferred from references, which removes an entire class of ordering bugs
- Built-in tests on models catch schema drift and data quality problems at the point they enter your curated layer
- Generated documentation and lineage give you a real answer to “where did this number come from”, which is otherwise tribal knowledge
- Near-universal adoption means every warehouse supports it and every analytics engineer knows it
Cons
- Does no extraction whatsoever, so it is always half of a stack and never a complete answer to “how do I get data in”
- Full-refresh materialisations recompute entire models, and incremental models require you to reason carefully about late-arriving data yourself
- Warehouse compute cost is entirely driven by your model design, and a badly designed DAG with many full refreshes is expensive in a way the tool does not warn you about
Best for: Any team doing warehouse transformation in SQL, regardless of which extraction tool lands the raw data.
Pricing: Open-source core with no licence fee for self-running; the commercial hosted offering prices per developer seat with usage-based tiers above that.
Dagster

An orchestrator built around data assets rather than tasks. Instead of declaring “run this job then that job”, you declare the tables, files and models that should exist and what each one depends on, and the system works out what to run and tracks the state of each asset. That model fits data pipelines much better than a generic task graph, because the thing you actually care about is whether a table is fresh and correct.
It integrates with extraction tools and transformation frameworks as first-class asset sources, so a Fivetran sync and a dbt model appear in the same lineage graph.
Pros
- The asset model matches how you actually think about data, and makes freshness and lineage first-class rather than something you reconstruct
- Strong local development story: pipelines run on your machine the same way they run in production, with typed inputs and outputs
- Native integrations mean extraction connectors and transformation models appear in one dependency graph instead of two disconnected systems
- Asset checks put data quality assertions in the same place as the asset definition, so they run automatically rather than as a separate job
Cons
- The asset abstraction has a real learning curve, and teams coming from task-based schedulers frequently fight it for a few weeks before it clicks
- Self-hosting a production deployment means a web server, a daemon and a database, plus wherever the compute runs, which is a platform to operate
- Heavier than you need if your requirement is genuinely “run this Python script every night”
Best for: Data platform teams building a pipeline with many interdependent tables who need lineage and freshness tracking across extraction and transformation.
Pricing: Open-source core with no licence fee for self-hosting; the managed cloud prices by compute usage with seat-based tiers.
Prefect

A workflow orchestrator that stays deliberately close to plain Python. You decorate functions as tasks and flows, and they run as normal Python locally with no infrastructure, gaining scheduling, retries, observability and state tracking when connected to a server. The design principle is that your code should not have to be restructured to be orchestrated.
Compared with an asset-oriented orchestrator, it is more general and less opinionated about data specifically, which makes it a better fit for pipelines that are mostly arbitrary work rather than mostly table building.
Pros
- Ordinary Python functions with decorators, so existing scripts become orchestrated workflows without restructuring
- Dynamic workflows are natural: the shape of the graph can depend on runtime data, which fixes the main awkwardness of static DAG tools
- Runs locally with no server at all, which makes development and testing genuinely simple
- Hybrid execution model keeps your code and data in your own infrastructure while the control plane is managed
Cons
- No data-asset model, so lineage and freshness across tables are things you build rather than things you get
- Less opinionated means more decisions land on you, and teams without strong conventions end up with inconsistent pipelines
- The product has been through major version changes with meaningful API differences, so community examples frequently target a different version
Best for: Teams whose pipelines are mostly Python doing varied work, who want orchestration without adopting a data-specific abstraction.
Pricing: Open-source core with no licence fee for self-hosting; the managed cloud prices by workflow runs and seats with a free tier.
Estuary

A streaming-first integration platform built around continuous capture rather than scheduled batch runs. Sources are captured as change streams into a durable log in object storage, and materialisations read from that log into destinations, which means one capture can feed many destinations without re-reading the source. Backfill and streaming are unified rather than separate modes.
The architecture matters commercially as well as technically: because the durable log is the source of truth, adding a second destination does not mean paying to extract the data again.
Pros
- One capture feeds multiple destinations from a durable log, so adding a warehouse or a search index later costs nothing at the source
- Streaming and backfill are the same mechanism, which removes the usual awkward seam between historical load and ongoing sync
- Genuinely low-latency delivery without the batch-window thinking that most tools in this category assume
- Schema inference and enforcement are explicit parts of the model rather than implicit connector behaviour
Cons
- Smaller connector catalogue than the established players, so an unusual SaaS source may simply not be available
- The streaming model requires different thinking about correctness and replay than a nightly batch, and teams used to batch need to adjust
- Smaller community means fewer public answers when something unusual happens
Best for: Teams who need continuous low-latency delivery from a small number of important sources into several destinations at once.
Pricing: Metered by data volume moved and by connector count, with the durable log letting additional destinations reuse a single capture.
How to choose
Start by splitting the problem, because most bad decisions here come from treating it as one purchase.
Separate extraction from transformation from orchestration. These are three jobs with three failure modes. Extraction breaks when a source changes. Transformation breaks when your own logic is wrong. Orchestration breaks when something needs to retry. A tool that claims all three is usually good at one. The common healthy stack is a managed or self-hosted extraction tool, dbt for transformation, and an orchestrator tying them together, and that stack is boring for good reasons.
Count your sources and rank them by importance. If three sources carry ninety percent of the value, the question is whether those three specific connectors are excellent everywhere, not whether the catalogue is large. Go read those three connectors’ documentation before you shortlist anything.
Decide who fixes a broken connector at 2am. If the honest answer is “nobody, we would notice in the morning and file a ticket”, buy the managed option and stop optimising for unit cost. If the answer is “we have a platform engineer who would fork it”, self-hosting is genuinely cheaper and you get to fix things instead of waiting.
Check the meter against your sources’ update behaviour. Before committing to per-row pricing, look at how many rows in your largest tables change in a typical month, and specifically whether any upstream system does bulk updates. One bulk-update-prone source can make per-row pricing the wrong model on its own.
| Tool | Layer | Connector maintenance | Meter shape | Runs where |
|---|---|---|---|---|
| Fivetran | Extraction | Vendor | Monthly active rows | Fully managed |
| Airbyte | Extraction | Vendor plus community, forkable | Rows or credits on cloud, free self-hosted | Both |
| Meltano | Extraction | Community, via Singer | No vendor meter | Self-hosted |
| dbt | Transformation | Not applicable | Seats on cloud, free self-run | Both |
| Dagster | Orchestration | Not applicable | Compute and seats on cloud | Both |
| Prefect | Orchestration | Not applicable | Runs and seats on cloud | Both |
| Estuary | Streaming extraction | Vendor | Data volume and connectors | Managed, private deployment available |
| Singer | (standard, not a product) | Community | None | Wherever you run it |
Two opinions. I would not self-host an extraction platform to save money at a small data volume; the engineer-hours cost more than the invoice, and you have added a stateful system to your on-call rotation to save a line item. And I would not let an extraction tool do meaningful transformation, even where it offers to. Land raw, transform in SQL you own and test, because a schema change should break a model with a diff, not a vendor component you cannot inspect.
Needs first-hand data: For your three highest-volume sources, count the rows that actually changed per day over one full month, using the source system’s own audit or updated-at data rather than the sync tool’s reporting. Compare that to the row counts each tool would bill you for over the same period, which you can get from a trial sync. The gap between “rows that carried new information” and “rows that were billable” is the number that decides whether per-row pricing works for you, and it is different for every source.
Frequently asked questions
What is the actual difference between ETL and ELT?
Where the transformation runs. ETL transforms data in a separate processing layer before loading it into the destination, which was necessary when warehouse compute was expensive and scarce. ELT loads raw data first and transforms it inside the warehouse with SQL. ELT won for most workloads because warehouse compute became cheap and elastic, but the better argument is operational: raw landed data means a schema change breaks a model you own and can test, rather than a pipeline component you cannot inspect.
Should I self-host or buy managed?
Decide by who fixes a broken connector on a Saturday. Managed means the vendor does, and you pay a premium that scales with your data volume. Self-hosted means you do, and you pay compute plus engineer time that scales with how many sources you have rather than how many rows. Self-hosting gets more attractive as volume grows and less attractive as source count grows, and it is a bad idea for any team without someone who already operates stateful services.
How do I stop schema changes from breaking my dashboards?
Land raw data with minimal reshaping, then assert the contract at the boundary: tests on the landed tables checking that keys are unique, required columns are not null, and enums contain expected values. Those tests fail loudly when a source changes, at the point of entry, before the change has propagated into models that feed dashboards. Without them the failure mode is a number that is wrong but plausible, which is far more expensive than a broken pipeline.
Is Singer still relevant?
As a specification, yes: several tools implement it, its message format is the reason a large tap ecosystem exists, and the STATE message is a genuinely good design for incremental extraction. As a thing you adopt directly, only if you are comfortable assembling the scheduling, monitoring and credential handling yourself, because the spec deliberately covers none of that. Most teams meet Singer through a tool built on it rather than using it raw.
Do I need an orchestrator if my ELT tool has scheduling?
Not at first. A managed extraction tool with a built-in schedule plus a transformation framework triggered after it is adequate for a long time. You need an orchestrator when dependencies stop being a straight line: when a model depends on three syncs finishing, when a failure should skip downstream work rather than run it on stale data, and when you need to answer “is this table fresh” without checking three separate systems.
Related reading
- Best managed Postgres providers — the source database most of these pipelines read from, and how its configuration constrains extraction.
- Best change data capture tools — reading changes from a database directly instead of polling it on a schedule.
- Best data warehouses for startups — the destination, and why streaming ingest is metered differently from batch loads.
- Best analytics and OLAP databases — the self-hosted destination alternatives and what they demand from your ingest batching.
- Best CI/CD platforms — where pipeline code should be tested and deployed, because data transformations are software.