How I use AI in data engineering (and what I keep for myself)

Migrating 39 QlikView apps to Qlik Sense Cloud used to mean weeks of copying expressions by hand. Today a script reads every QlikView project folder, extracts the expressions and proposes master measures. I review them.

AI has changed my job more in the past two years than any tool in the twenty before. Pretending otherwise would be dishonest.

But “I use AI” says nothing. Everyone says it. Here is what it looks like in practice.

Requirements first, code second

AI writes code fast. It also writes the wrong code fast.

So the work starts where it always did: what does the business need to see, from which source, at which level of detail, how often. I write that down before any code exists. The sharper the requirements, the better the code AI produces from them.

That part hasn’t gotten faster. It has gotten more important.

Repeatable scripts, not one-off chats

I don’t ask AI to transform your data. I ask it to help me write the script that transforms your data.

A chat answer is different every time. A Python script does the same thing on Monday as on Friday. You can version it, test it, run it again next quarter and hand it over to your own team.

Deterministic beats clever. Especially when finance signs off on the numbers.

Proven recipes, built faster

I rarely start from zero. Open-source packages and documented APIs already solve most of the hard problems. Someone else has written the connector, the parser, the deployment tool.

What changed is how fast I can put those pieces together. AI drives the tools through their API or command line, so I spend my time on the parts that are specific to your business.

Four examples

ABC Engines: QlikView to Qlik Sense Cloud. 39 apps, 73 source tables from the Infor LN data lake. Python scripts drive the Qlik command line: extracting QlikView expressions, generating master measures and variables, deploying apps, comparing source data against what landed. Even the sheet thumbnails are a script now. Thirty sheets used to be half an hour of screenshots.

In The Pocket: a data platform on BigQuery. 12 source systems, deployed with Terraform and GitLab CI. I built my own AI tooling (Claude Code skills and MCP connectors) for the repetitive parts of data engineering. That left time for the data model and the automated quality checks that kept reporting consistent through a platform migration.

StaticShift: a product next to the consulting. Fast static websites for law firms and notaries. Hugo, an open-source site generator, does the heavy lifting. AI helped me build what surrounds it: a scanner that scores a prospect’s website and a pricing configurator.

Prima Materials: classifying building materials. A pricing database for architects. Materials get classified with the Claude API and a local open-weight model (Gemma via Ollama). Running a model locally keeps sensitive data in-house where that matters.

What I don’t hand to AI

The vision.

Which question matters for your business. Which KPI actually drives a decision. What to build first, and what not to build at all. AI doesn’t sit in your management meeting. It doesn’t know that sales stopped trusting the numbers after the last reorganisation.

Good data work is a cycle: business case, data, dashboard, and back to the business case. AI speeds up the middle. The beginning and the end are mine.

What this means for you

The same result in fewer days. You pay for judgment and architecture, not for typing.

And your data stays where it belongs. AI works on code, schemas and metadata. Where client data itself needs processing, I prefer models that run locally.

Which part of your data work still runs on manual effort every week?