We automated most of our engineering work this year. Sixteen AI agents, each with a job title and a written job description, now work a shared task board across thirteen of our codebases — a dental practice platform with a clinic already live and seeing patients, the marketing sites we and our clients run, and a set of internal products at various stages of becoming real.
That team turns out around a thousand unique tasks a month, and delivers the large majority of them. We are a small company, and six months ago that rate was nowhere close.
This is not a post about how AI can write code. Everyone has read that post. This is about the part nobody writes down: where we decided to stop the agents, and why those particular places.
First, the part that was not automated
The architecture came first, and people built it.
Our medtech products share one architecture — an API gateway in front of several backend services, a dedicated identity provider, a document database, a React front end. The decisions that make it work are the boring, expensive ones. Where the service boundaries fall. That permission checks happen in exactly one place instead of being sprinkled through four services. That a clinician is not a row in a table but a member of a group in the identity provider, so that access and identity never drift apart. That the gateway configuration is generated from a template rather than edited by hand, because a hand-edited gateway config is how a company loses a weekend.
Our engineers made those calls over months, by arguing about them, getting some wrong, and fixing them. No agent proposed that architecture, and no agent would have. Models are good at working inside a structure and noticeably worse at choosing which structure a business should live inside for the next five years.
What changed is not who designs the system. It is who maintains and extends it. The architecture was the hard, human part; keeping it healthy, adding to it, testing it and shipping it day after day is the part we handed over — and that is most of the hours in a software company’s life, not the minority.
So the right way to read everything below is this: humans drew the map, and agents now do the driving, within lanes humans painted.
The team
There is a Chief of Staff, who is the user’s actual point of contact — it turns a request into a specified task, routes it to whoever owns that code, chases them until it moves, and reports back. A CTO sets technical direction as short decision records written on the task, and reviews every finished branch as an actual diff — for correctness, for security, for whether it fits the architecture — before approving it or sending it back. A Project Manager turns a goal into tasks small enough for a coder to execute without guessing: it restates the request, reads the relevant code and docs before writing anything, and asks rather than filling a gap it isn’t sure about.
Five coders split strictly by repository: the Coder owns the dental platform and its clinic fork — gateway, identity, backend services, the React front end; the Landing Coder owns our two Astro sites; the WordPress Coder owns the two WordPress themes; and two single-product specialists each own one more codebase — the Internet.am Coder owns a hosting company’s site and its API, and the Poker Coder owns an internal poker study tool. Two separate testers, because testing a patient-facing platform and testing a public website are not the same job and pretending otherwise produces bad reports of both kinds: Platform QA tests the live system’s behaviour — both sides of a permission, persistence re-checked after a reload, boundary and duplicate input, the service logs, regressions in nearby flows — while Landing QA reports the big, obvious flaws on the public sites — a page that doesn’t load, a broken layout, a dead form, a missing locale — and treats small visual deltas as deliberately not its job.
Then six specialists, each watching one thing across every codebase:
- a Security Auditor, on the security and multi-tenancy posture of the two clinical codebases, including the one unauthenticated internet-facing endpoint, which is never the one it skips
- an Architecture Evangelist, who owns the shape of the system rather than any one bug in it — tenant isolation, where the seams between services fall, what is genuinely tested versus only hoped for — and whose job is to object when a change is about to make that worse
- a Performance Monitor, who measures the live sites on a schedule and turns each regression into a specific fix with the evidence behind it
- a Dependency Watch, who inventories versions across the repos, judges what is worth updating, and prepares the update tasks without ever applying one itself
- an SEO Specialist, on how the sites are marked up, how they describe themselves to crawlers, which languages are actually indexable, and which real questions we don’t answer yet
- a Content Manager, on what the sites say — the words, the order the message is told in, and whether every claim on the page is true
Sixteen agents in all.
Roles that mostly exercise judgment — the Chief of Staff, the CTO, the Coder — run on the strongest model we have access to. The rest run on a faster, cheaper one. That is a per-role decision, revisited when a role turns out to need more or less than we gave it, and it is the single biggest lever on what this costs to operate.
Repository ownership is the rule we enforce hardest. If a task about a marketing page turns out to need a change inside a backend service, the Landing Coder does not reach into the service. It says so on the task and hands it to the agent that owns that code. Shared ownership of a codebase between autonomous workers is how you get two confident, conflicting rewrites of the same file.
Decision making: two gates, and nothing in between
Here is the design choice we are most confident about.
Gate one is what gets worked on. No agent can decide that. Work arrives on the board as a proposal, and a human moving it into “to do” is the approval. That single move covers everything that follows: the plan, the code, the review, the testing, and the fixes that come out of either.
Gate two is what ships. When a branch is finished, reviewed and tested, the agent that wrote it opens a request that only a human can accept. Nobody else can click it — not the CTO, not the Chief of Staff. Until a person accepts, the change does not reach the main branch, and deploying is a separate human action after that.
Between those two gates, nobody asks permission. This was the correction that made the whole thing work. Our first version had agents pausing mid-task to confirm their plan, and the result was a queue of sign-off requests a person had to read in order to discover that nothing had happened yet. Approval that frequent is not oversight; it is a second job. So we collapsed it: one approval at the start, one at the end, and in between the written rules do the governing.
Agents come back mid-task for exactly one reason — when the work turns out to be bigger or different than the task described. Another service. A database migration. A new dependency. A change to authentication. Those are the cases where continuing would mean deciding something nobody asked them to decide, and they are rare because we spend the effort to describe tasks properly up front.
The proof is in the log. Across a month of work at that volume, the one-off permission requests a human had to adjudicate number in the low double digits — and almost all of them are requests to hire a new agent for the team. Everything else moves on rules that were written once.
Development: one path, every time
Every change follows the same path. The work happens off to the side, never on the live line. Commit, and post the evidence on the task — what was run, what it produced. The CTO reviews the actual diff, not a summary of it. The right tester takes the finished work and exercises it against a running system. A human accepts the landing request. Only then does the change join the main line of work, as one clean, reviewed step; if it does not go in cleanly, it goes back to be sorted out where it came from. That order never bends.
Two details inside that are worth stating plainly, because both are the opposite of what you would guess.
Testing happens once, at the end. Not per commit, not per sub-task. A feature that took five tasks gets one testing pass covering all five. And where verifying a change just means reading it carefully, that is the CTO’s review and there is no testing pass at all. Testing everything constantly sounds rigorous and is mostly a way to spend your budget confirming things you already knew.
Every repository carries a register of bugs we told the agents not to fix. This is our favourite artifact in the whole setup. Each codebase’s guide has a section listing behaviour that looks like a mistake, is a mistake, and must be left alone — a form that says “thank you” whether or not the submission actually arrived, a share image at the wrong size, icons that render unfilled because of an old authoring error. Each entry names its ticket and the reason it was kept. Alongside them sit the traps: the CSS rule whose apparent duplication is load-bearing, the dependency that must stay declared even though nothing imports it, the two stylesheet imports whose order silently changes the result.
Without that register, a capable agent reads the code, correctly identifies a flaw, and helpfully repairs a decision somebody already made on purpose. That is the characteristic failure of AI-assisted development at scale, and it is not solved by a smarter model. It is solved by writing down why things are the way they are.
Security: boundaries, not guidelines
The platform handles patient and clinic data, and a real clinic depends on it. So the boundaries are written as absolutes, in the agents’ own instructions, in language that does not leave room for a clever exception. An autonomous worker that can rationalise an exception will eventually find one.
- No agent deploys anything. Not a branch, not the main branch, not a tag, not through a script or an alias. Work lands locally and stops. Shipping to production is a human pressing a button, every time, with no exceptions and no automation path around it.
- No agent touches production credentials or environment files. Not to read them, not to “check” them.
- No agent starts, stops or rebuilds the running environment. A person brings the system up before work begins. Reading logs is allowed; controlling the system is not. If a service is unhealthy, the correct behaviour is to say so and let the work wait.
- Authorization lives in one place. Agents are explicitly forbidden from adding permission checks inside individual services, so there is one configuration to audit rather than an access rule hidden in four codebases.
- Browser automation is fenced. Local addresses of our own projects only, a throwaway browser profile that never touches anyone’s real one, test data only, and never real patient or clinic information.
- Two permanent watchers. A Security Auditor and a Dependency Watch are standing roles, not a checklist someone remembers to run before a release.
Notice that almost none of this is about the agents being untrustworthy. It is about making the expensive mistakes structurally unavailable, which is the same reason we put authorization behind one gateway in the first place.
The honest numbers
We would rather publish these than have you assume them.
The team opens and works around a thousand tasks in a month. Of the ones that reach a conclusion — finished or cancelled — roughly 85 percent were delivered. Measured against every task we ever opened, including the ones still open or cancelled, it is closer to 80 percent. Read whichever number you trust more; we are publishing both rather than picking the one that flatters us.
Those tasks take more than two thousand agent runs a month, and around two percent of them fail outright: an expired credential, a failed workspace check, a lost process. Not zero, and each one needs a person to glance at it.
And there is always a queue of work waiting on a human — a finished branch sitting in front of its landing gate, or a question an agent was right to ask instead of guessing at. That queue never empties, and we do not want it to. It is where the two gates actually live.
What we would tell you if you were about to try this
Design the system yourself, with people, before you let anything automate work inside it. Then write down why it is that way — including the parts that are wrong on purpose. Give every worker exactly one area it owns. Put your approval gates at the start and the end, not in the middle, because approvals in the middle train everyone to stop thinking. And keep the irreversible actions — deploying, credentials, controlling the environment real people depend on — entirely in human hands.
Everything in the middle, which is most of the work, can be delegated now. We were surprised how much.
Sky Lab builds software products and platforms. Want to talk about how your own engineering could work this way? Get in touch