HUMAN-IN-THE-LOOP DATA SOLUTIONS
Databricks Genie Code and Claude have tremendous capability for data engineering and analytics automation. We built the data engineering and analytics harness, a human-in-the-loop data solution.
AI handles the repetitive coding. Your people govern architecture, documentation and quality. Together, that moves data from an IT cost center to a strategic profit center.
1 week
to roll a new standard across the whole technical footprint, instead of six to eight months
1/3 to 1/10
of the previous implementation cycle, once work moves off isolated coding lanes
15 min
job runtime on small compute, down from hours, once efficiency patterns are applied
Illustrative. Source: our Databricks Genie Code pilot. Results depend on your team, data and stack.
THE PROBLEM WE SOLVE
Data development grew an accidental architecture: each developer in their own lane, writing custom code whose patterns live only in their head. Engineers are exhausted, and the work moves slowly.
HOW WE WORK
Every day, work ships with people in the loop. Every month or quarter, the team steps away from firefighting to sharpen the standards, and those standards flow straight back into AI coding.
DAY TO DAY · EVERY CHANGE
Shape the solution before any code is written.
Human reviewAI coding, dev and prod QA checks, and the workflow, together.
Human review in DevLightweight prod QA watches live work and raises triggers.
Automated QAON A QA TRIGGER
Someone with context makes the call when prod QA flags it.
PERIODIC · MONTHLY OR QUARTERLY
Library, notebooks, QA and more. Written narratives and collaborative boards build shared understanding and align on long-term strategy.
Human-ledEach agreed standard is codified into AI, so the new best practice becomes the default baseline for all code.
Into AI codingWHO DOES WHAT
FROM REQUEST TO CODE
Human logic and queries pass through templates that AI manages, which keeps documentation complete and ownership shared. Databricks Genie Code and your existing repository rules are the foundation for everything AI generates.
A real need, tied to cost or revenue.
A person shapes the logic; AI helps draft it.
AI applies the naming, coding and QA standards.
Consistent, and owned by the whole team.
ON HALLUCINATION
This isn't open-ended chat or document search, where a failed lookup can quietly drop context. It's a strict, simple chain with explicit, deterministic inputs. People supply the logic and the templates; AI is the automation engine.
The seed of every rollout, even if AI helps draft it.
HumanOnce final, the query becomes a standard that dictates implementation.
HumanThe rules AI must follow, written by people.
HumanRepetitive tasks, executed the same way everywhere.
AISTANDARDS VS. DESIGN LOGIC
Standards make tasks run the same way everywhere. Design logic changes with every implementation, because it needs domain knowledge, business context and an eye for data nuance. So design and validation stay human-led, and the standards that come out of them become fully automated.
EXAMPLE: A FIVETRAN DELETE FLAG
A backend change in Fivetran started marking deleted rows with a flag, so old (deleted) and new records both appeared, and pipelines that ignored the flag failed. A person caught it; the fix went into the standard, so every Fivetran table now accounts for the delete flag automatically.
QUALITY, BUILT IN
A mandatory staging environment makes fast, safe rollouts possible. Modern CI/CD keeps it simple to maintain, and cloud compute spins up only when needed, so the extra cost stays low.
DEV → PROD GATE
IN PRODUCTION
Schema evaluations and design specs.
Produced on every cycle.
Feeds forward to every downstream step.
Can't start without the documentation.
REFACTORING LEGACY CODE
Instead of rewriting from scratch, we point Databricks AI at existing logic while the legacy code keeps running, then prove functional parity before anything is promoted.
AI detects redundancies and breaks it into clear, modular parts.
Built in parallel while production keeps running.
Old and new run in temp views for an apples-to-apples check.
Repeat until it matchesOnly when results match production exactly.
PROVING VALUE
We benchmark on deliberately small compute, where inefficiency can't hide, and we hold every project to an economic standard.
Jobs that take hours expose the bottlenecks.
Runtimes drop to 30 or even 15 minutes.
Still on small compute, to prove it repeats.
Runtime and cost savings show up at scale.
EVERY PROJECT
Every initiative and enhancement is measured against the economic value it adds to the bottom line.
THE EXCEPTIONS
Regulatory work and the information leaders need to operate are judged on the shortest turnaround at the lowest possible cost.
CODIFYING EXPERTISE
It takes three to five times more time upfront to make sure the team truly understands a capability. After that, it joins the standard patterns and AI rolls it out across the codebase.
PEOPLE AND CULTURE
Demanding stakeholders push for custom fixes because they want speed. Standardized development gives them quick, dependable code with quality built in, so nobody needs to go around the process.
Feedback is accepted and admired, not merely tolerated, and we visibly act on what people share. That keeps people engaged.
Informed by Theory U, receptive listening is the clearest invitation for people to speak openly.
OUR BELIEF
Much of what we call cognitive work is retelling what we've already done. AI takes that over, the way cars and airplanes replaced walking. That frees people to grow beyond the socialized and self-authoring mind toward a self-transforming mind (Kegan and Lahey, Immunity to Change).
Works from what others expect and what has been done before.
Works from its own internal compass and past experience.
Draws on non-conscious resources to reach new ideas and prototype the future.
"What have I done today that, if AI were involved, could not have been automated?"
BEYOND THE DATA TEAM
Done right, AI should make people's roles more meaningful, not leave them fearing for their jobs. That takes a deliberate choice by leaders to treat employees as assets who can use AI to deliver far more value, not as a cost to optimize away.
To spark that conversation, we're writing a series of novels, beginning with The Beginning, to help readers ask new questions about their work and their potential.
Let AI handle the repetition, so your people can do the work only people can do.