Covering 125 SaaS API actions with a contract built for databases

The situation

The catalog was public and MIT-licensed, and the project was actively asking for outside contributions. The README said the core team reviewed and merged community submissions daily. The number of connectors was the go-to-market, so every hour it took to write a connector cost the company something, and so did every connector that shipped without tests.

The published destination interface had three methods: connect, disconnect, and test the connection. The actions the early destinations declared show what it had been designed around: insertData, updateData, deleteData and executeRawQuery for the warehouse, insert / update / delete for the SQL databases, and insertOne and insertMany for the document store. All of it is CRUD over a table.

Accounting and commerce APIs don’t look like that. Xero’s accounting API alone exposes 80 distinct operations, and Shopify’s REST surface needed 45. Between them they have nested resources, per-resource verbs, positional SDK arguments, per-tenant scoping and aggressive rate limits. Done the way the database connectors were done, that would have been 125 hand-written methods across two connectors, each with its own documentation page and its own chance to get an argument order wrong.

What was wrong

The contract was well designed for the assumptions it was written under, and those assumptions no longer held for the connectors the catalog needed.

• The action vocabulary assumed tables

  Real API surfaces are nested, with verbs that belong to a resource: orders.refunds.calculate.create, fulfillment_orders.release_hold.create. Neither maps onto inserting, updating or deleting a row.

• No place for positional SDK arguments

  Events carry a flat payload of named fields, and vendor SDK methods take ordered arguments. Something has to know the mapping between the two for all 125 operations, and the contract didn’t provide for one.

• Multi-tenancy was pushed onto the caller

  A single Xero token can be authorised for several organisations. Nothing in the contract said which organisation an incoming event was meant for, so the ambiguity would have shown up as data written to the wrong tenant.

• The warehouse trusted the payload

  Arbitrary event JSON went into typed warehouse columns. With nothing reconciling the two, a type mismatch ends up as a wrong row in the table and no error anywhere.

• Partial failure wasn’t specified

  The published return type for a source named the registered webhook data and its events. It didn’t say what should happen when you ask for twelve subscriptions and three of them fail. Left unspecified, that’s a way for a pipeline to lose events without anyone noticing.

• Almost nothing in the catalog was tested

  Of the twelve destinations in the catalog, five have a test suite. The others shipped with none, including 208 lines of Postgres, 211 of MSSQL, 137 of Firestore and 124 of MongoDB.

How I got to the design

There were three structural changes, and one question about the contract that I worked out in public over five days.

• Reflective dispatch. The platform already had a Proxy-driver pattern, which came in with its database connectors. What it didn’t have was a way to fill positional arguments from a named payload. My driver reads the SDK function’s parameter names out of its source and uses them as the mapping:

  methodParams = (Function.prototype.toString.call(targetMethod)
    .match(/\((.*?)\)/)?.[1] || "")
    .split(",").map((param) => param.trim());

  invoiceID in an incoming payload ends up in the right slot, and nobody has to write the signature out by hand. That’s one implementation for 80 actions, with no map to keep in sync.
• Multi-tenancy handled in the driver. Connecting enumerates every Xero organisation the token is authorised for. Performing an action fans out across all of them and returns a response map keyed by tenant, with errors caught per tenant so that one bad tenant doesn’t fail the batch. The connection test rejects a token that is scoped to zero organisations or to more than one. It builds its error message by re-querying the vendor, so a refusal names the organisations attached to the token and says what to do about it.
• Serialisation driven by the table schema. The BigQuery destination reads the target table’s schema and converts each incoming JSON value into a SQL literal of the right type. It normalises the temporal types, wraps geography, casts bytes, emits JSON literals, and recurses into nested records to build struct expressions. Numeric and boolean fields that don’t match throw a named schema-mismatch error, so a mismatch shows up as a failure and can’t end up as a wrong row.

The contract question

The webhook lifecycle is the easiest part of this to check, because it’s three dated commits over five days in a public repository.

1. 18 Aug

   Naive concurrency

   A Promise.all over every subscription request, with no error handling. I added a catch that rethrows with a descriptive message. After that, one failed subscription failed the whole init(), and the caller learned nothing about the other eleven.

2. 21 Aug

   Switched to partial success

   Collect the successes, log the failures, and return the topics that did register. This changed what the returned event list means. It now reports what happened, so a caller can see what’s missing and act on it.

3. 23 Aug

   Serial requests under the rate limit

   I dropped Promise.all for a serial loop, because creating webhooks concurrently was tripping the vendor’s rate limit. The loop times each call and sleeps for whatever is left of a 500ms floor before making the next one. A try/catch per event keeps one failure from affecting the others.

The sequence was naive concurrency, then fail-fast, then partial success, then serial requests that respect the rate limit. The final version is slower than the first and it’s correct, which was the right trade for this pipeline. The step I got wrong is still in the history.

What I rejected, and why

I built several of these first and then reverted them.

• Writing 125 action methods by hand

  Every existing destination had been written this way. I went with reflective dispatch. A single 60-line performAction covers 80 Xero actions and 45 Shopify ones, and when the vendor adds an operation there’s nothing new to write.

• A hand-maintained argument map per action

  The obvious way to handle positional SDK arguments is a lookup table from action to parameter order. That would be 125 entries, and they go stale whenever the vendor SDK changes. Reading the parameter names off the live function means there’s no table to maintain.

• Promise.all for webhook creation

  I rejected this twice, for different reasons. The first time it was the error semantics, since one rejection throws away eleven successes. The second time it was the concurrency, which the vendor rate-limited.

• Fail-fast on partial subscription failure

  I implemented this and reverted it three days later. If a source registered nine of twelve webhooks, it’s more useful for it to report nine than to throw.

• Coercing type mismatches into the warehouse

  Numeric and boolean mismatches throw a named error. In an analytics warehouse a wrong row is worse than a rejected one, because nobody goes looking for it.

• Hand-rolled backoff per connector

  Rate limiting isn’t specific to one connector. I pulled in a retry library and configured exponential backoff on HTTP 429, so Shopify didn’t need its own copy of the loop.

• Credentials in test files

  I moved test secrets into environment configuration early on, with a checked-in sample file that has empty values. That’s why a scan of the repository history still comes back clean.

• Linting only my own files

  This would have been the narrower and more polite option. But the point of a shared ruleset is a consistent catalog, so I configured it and fixed all 25 existing files that violated it.

What couldn’t break

This is a public catalog with outside contributors, and some of its connectors were already in production. A careless change would have broken other people’s integrations.

• The published contract. Every destination declares the platform’s destination interface, and every source its integration interface. I extended what the contract could reach without changing its shape. New capability came in through the action vocabulary and the driver layer, so existing connectors and the platform’s driver loader weren’t touched.
• Other people’s connectors. The lint retrofit touched 25 files that belonged to ten other contributors. Changes like that tend to break things quietly, so it went in as two commits that contained only style fixes and the ruleset.
• Three incompatible vendor auth models. WooCommerce verifies with a base64 SHA-256 HMAC over the raw body. Adyen uses a base64 basic-digest compared against the Authorization header. Xero needs an OAuth2 refresh-token exchange plus a JWT decode. All three are implemented behind the same webhook verification method, which has a single signature.
• A live OAuth2 migration under a shipped connector. Xero changed its token lifecycle while the connector was in production. I moved the rework and its test suite in one commit, so there was never a build where the tests described the old flow.
• Ordering and dedupe downstream. The Kafka destination exposes partition, key, timestamp and headers on push. A connector can’t guarantee per-key ordering on its own, but the platform can only do it if the connector passes that information through.
• Documentation. I wrote 80 Xero and 47 Shopify per-action documentation pages against the catalog’s template, and they went in as their own commits, since an undocumented action isn’t going to get used.

Result

I built four of the catalog’s twelve destinations and four source integrations end to end. One reflective dispatcher covers roughly 125 vendor actions, and there are about 1.7 lines of test for every line of implementation.

• Four of the five tested destinations in the catalog. Test suites weren’t part of the house style before this work, and they were afterwards.
• 130 of the 245 catalog documentation files, including 80 per-action pages for Xero and 47 for Shopify.
• 25 files brought under one lint ruleset, from ten other contributors’ work, plus the shared ruleset itself.
• Third by commit count in a repository of 25 contributors, and the heaviest external-facing implementer of the contract.