Your Agency Is Writing Your Code With AI. Here Is What to Require Before You Sign

Posted on Oct 21 • Originally published at pykero.com

Yes, your agency is almost certainly using AI to write part of your code, and that is fine. What is not fine is AI-drafted code that no human read carefully and no test exercises, merged on a Friday because the demo was due. The fix is not a tool ban. It is a short list of controls you put in the contract and the repository before the first sprint.

A developer with a coding assistant can produce a working-looking feature in an afternoon that used to take three days. That is real. The problem is that review, testing and understanding did not get faster at the same rate. If an agency keeps the same review habits it had before and triples its output, you get the same bug rate per line applied to three times as many lines.

Technical debt used to accumulate at human speed. Now it accumulates at machine speed, and it looks clean on the surface. The first version of our own outreach engine, described in section 4, was a five-step chain that read beautifully: every step well named, every function confidently commented, nothing a reviewer would stop on. It also had five places to fail silently, and the clean surface was exactly why nobody questioned the four extra steps. AI-written code is well formatted and well named, which is precisely what makes a weak reviewer skim it. We wrote about the general discipline in technical debt management. This post is about the specific version that shows up when a vendor is writing the code.

"We write tests" is a sentence. A gate is a configuration. Ask for both of these before work starts:

Then ask one question that separates agencies: "Can you show me a pull request from last month that was blocked by a failing test?" If the answer is "our tests never fail", the tests are not testing anything.

Assistants produce code. They do not take responsibility for it. Your contract should name who does, and the repo should prove it: every pull request has a reviewer who is not the author, and the review has comments, not just a green checkmark.

AI assistants love adding packages. They will reach for a library to pad a string. Each one is a supply-chain surface and a license question. Require:

The cheapest code to test is the code that was never written. AI makes it painless to generate a six-step pipeline where one step would do, and every extra step is a place for a silent failure.

We hit this in our own outreach engine. It scrapes a prospect's site with a self-hosted crawler and uses a local model to draft one tailored email per company. The first version was a chain: extract facts, summarise, pick an angle, draft, polish. It was slower, pricier and, more importantly, it had five places to go wrong and five things to test. Collapsing it into a single call that extracts the facts and drafts in one pass beat the chain on both cost and output quality, and it halved the surface we had to cover with tests. We wrote the general case up in single call vs agent chains.

Ask your agency how they decide when generated code is too much code. If they do not have an answer, you will pay for the extra lines twice: once to write and once to maintain.

The real test of AI-written code is whether a new engineer, with or without an assistant, can read it in a week. Require:

Expect fair quotes to drop on boilerplate: CRUD screens, forms, admin tables, standard integrations. Expect them to stay roughly flat on the parts that were never typing-limited: the data model, the edge cases in your business rules, performance, and the review itself.

An agency whose quote dropped 60 percent across the board since adopting AI tools has either cut review or is about to. One whose quote did not move at all is probably not passing the savings through. The honest middle is a lower number on the predictable work and a line item for testing and review that did not shrink. If you are weighing engagement models, fixed price vs time and materials covers how to structure that.

Clear, specific answers mean the agency has thought about this. Vague reassurance means you are the first client to ask. Pair these with the broader software project red flags and you will screen out most of the risk in a single conversation.

AI coding tools are a speed multiplier. Multiply a disciplined team and you get a disciplined team that ships faster. Multiply a sloppy one and you get sloppy at scale, with a codebase that looks great until the first real incident. Your job as the buyer is not to pick the tool. It is to require the gate, the reviewer, the policy and the readable repo, and then check that they exist.

If you want a build where the tests are in the pipeline and the reviewer is named before the first commit, let's talk.

Originally published on the Pykero blog.