Vibe Coding Without the Technical Debt: A Python Guardrail Pipeline That Catches What the AI Missed

Andrej Karpathy coined “vibe coding” in a post on X on February 2, 2025: you describe what you want, accept what the model writes, and “fully give in to the vibes.” By November, Collins had picked it as its Word of the Year. Adoption followed the hype: in the 2025 Stack Overflow Developer Survey, 84% of respondents said they use or plan to use AI tools, yet more of them distrust the accuracy of those tools (46%) than trust it (33%).

So is vibe coding bad? The research on AI generated code vulnerabilities gives a consistent answer: the code is not bad by default, but it is not safe by default either.

The fix is not to stop using the assistant, but to make every AI-written change pass the checks you would demand from a new hire. In this post we take one small, realistic “vibe-coded” Flask module, run a guardrail pipeline over it, look at what each tool caught and what it missed, fix the code, and wire everything into pre-commit and GitHub Actions.

Versions (pinned in requirements-dev.txt): Python 3.13.18, Flask 3.1.3, ruff 0.17.3, mypy 2.4.1, bandit 1.9.4, detect-secrets 1.5.0, vulture 2.16, pip-audit 2.10.1, pytest 9.2.0, pytest-cov 7.1.0, Hypothesis 6.170.13, pre-commit 4.7.0. Every output block below is copied from a real run.

The prompt was the kind you type in a hurry: “Flask API for orders: list a customer’s orders with pagination, apply a percentage discount, notify a webhook.” I wrote the result by hand to reproduce the patterns the studies above keep finding, so every issue here is deliberate, but none of it is exotic. It runs and it demos well:

import json import os import sqlite3 import requests from flask import Flask, jsonify, request app = Flask ( __name__ ) DB_PATH = " shop.db " PAYMENTS_API_KEY = " sk_live_51NzQ8vKp2ExampleKey9fT3 " WEBHOOK_SECRET = " whsec_3f9a1c7e2b " WEBHOOK_URL = os. getenv ( " WEBHOOK_URL " ) def get_db (): conn = sqlite3. connect ( DB_PATH ) conn. row_factory = sqlite3. Row return conn def apply_discount ( total, percent ): return round ( total - total * percent / 100, 2 ) def paginate ( items, page, per_page ): start = ( page - 1 ) * per_page return items [ start: start + per_page ] def legacy_total ( items ): total = 0 for i in items: total += i [ " price " ] * i [ " qty " ] return total @app.route ( " /orders " ) def list_orders (): customer = request. args. get ( " customer " ) page = request. args. get ( " page ", 1 ) db = get_db () rows = db. execute ( f " SELECT id, customer, total, status FROM orders WHERE customer = ' { customer } '" ). fetchall () orders = [ dict ( r ) for r in rows ] return jsonify ( paginate ( orders, page, 20 )) @app.route ( " /orders/<int:order_id>/discount ", methods = [ " POST " ]) def discount ( order_id ): data = request. get_json () percent = data [ " percent " ] db = get_db () row = db. execute ( " SELECT total FROM orders WHERE id =? ", ( order_id,)). fetchone () new_total = apply_discount ( row [ " total " ], percent ) db. execute ( " UPDATE orders SET total =?

WHERE id =? ", ( new_total, order_id )) db. commit () notify ( order_id, new_total ) return jsonify ({ " id ": order_id, " total ": new_total }) def notify ( order_id, total ): try: requests. post ( WEBHOOK_URL, json = { " id ": order_id, " total ": total }, headers = { " X-Signature ": WEBHOOK_SECRET }, verify = False, ) except: pass if __name__ == " __main__ ": app. run ( host = " 0.0.0.0 ", debug = True ) Enter fullscreen mode Exit fullscreen mode The assistant also wrote tests. They pass:

from shop.app import apply_discount, paginate def test_apply_discount (): assert apply_discount ( 100, 10 ) == 90 def test_paginate (): assert paginate ([ 1, 2, 3, 4, 5 ], 1, 2 ) == [ 1, 2 ] Enter fullscreen mode Exit fullscreen mode Try to count the problems before the tools do. The matrix near the end lists sixteen, and no tool in this post flags three of them.

Each tool answers one narrow question. None of them replaces the others, and none of them replaces a reviewer.

Pipeline diagram: commit-time hooks (ruff, bandit, detect-secrets, vulture, mypy, pip-audit on requirements change), pre-push tests with a 90% coverage gate, and the CI job that reruns everything with 1,000 Hypothesis examples

The project configuration that all tools read lives in one pyproject.toml:

[ project ] name = "shop" version = "0.1.0" requires-python = ">=3.12" [ tool. ruff ] line-length = 100 target-version = "py312" # vulture whitelists are bare expressions by design; force-exclude also applies to pre-commit extend-exclude = [ "vulture_whitelist.py" ] force-exclude = true [ tool. ruff. lint ] select = [ "E", "F", "B", "BLE", "UP", "SIM" ] [ tool. mypy ] strict = true packages = [ "shop" ] [ tool. bandit ] exclude_dirs = [ "tests", ".venv" ] [ tool. pytest. ini_options ] addopts = "--cov=shop --cov-branch --cov-report=term-missing --cov-fail-under=90" [ tool. coverage. run ] omit = [ "tests/*" ] Enter fullscreen mode Exit fullscreen mode Step 1. ruff: fast, but only the surface $ ruff check --output-format concise. shop/app.py:1:8: F401 [*] `json` imported but unused shop/app.py:64:5: SIM105 Use `contextlib.suppress(BaseException)` instead of `try`-`except`-`pass` shop/app.py:71:5: E722 Do not use bare `except` Found 3 errors. [*] 1 fixable with `--fix` (1 unsafe fix available with `--fix --unsafe-fixes`). [exit 1] Enter fullscreen mode Exit fullscreen mode ruff caught the unused json import and the bare except. That is useful hygiene, but notice SIM105: it proposes contextlib.suppress(BaseException). Accepting that “unsafe fix” (ruff labels it that way itself) keeps the real bug, a webhook failure that silently disappears, and just makes it shorter.

The right fix is to catch requests.RequestException and log it.

ruff also ships rules derived from flake8-bandit under the S prefix (the f-string query would be S608), and you can enable them. I keep bandit separate here so you can see exactly what each tool finds; in a large repo, ruff --select S is the faster option.

On the unannotated vibe code, strict mode returns twelve errors, and every one of them says the same thing:

$ mypy shop/app.py:16: error: Function is missing a return type annotation [no-untyped-def] shop/app.py:22: error: Function is missing a type annotation [no-untyped-def] shop/app.py:26: error: Function is missing a type annotation [no-untyped-def] shop/app.py:31: error: Function is missing a type annotation [no-untyped-def] shop/app.py:39: error: Function is missing a return type annotation [no-untyped-def] shop/app.py:42: error: Call to untyped function "get_db" in typed context [no-untyped-call] shop/app.py:47: error: Call to untyped function "paginate" in typed context [no-untyped-call] shop/app.py:51: error: Function is missing a type annotation [no-untyped-def] shop/app.py:54: error: Call to untyped function "get_db" in typed context [no-untyped-call] shop/app.py:56: error: Call to untyped function "apply_discount" in typed context [no-untyped-call] shop/app.py:59: error: Call to untyped function "notify" in typed context [no-untyped-call] shop/app.py:63: error: Function is missing a type annotation [no-untyped-def] shop/app.py:66: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" [arg-type] Found 13 errors in 1 file (checked 2 source files) [exit 1] Enter fullscreen mode Exit fullscreen mode By default, mypy does not type-check the bodies of unannotated functions, and AI assistants often skip annotations in quick scripts. So mypy has nothing to check until you add signatures. After adding only the signatures (no logic changes), the noise disappears and a real bug appears:

$ mypy shop/app.py:48: error: Argument 2 to "paginate" has incompatible type "str | int"; expected "int" [arg-type] shop/app.py:67: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" [arg-type] Found 2 errors in 1 file (checked 2 source files) [exit 1] Enter fullscreen mode Exit fullscreen mode request.args.get("page", 1) returns the string "2" when the parameter is present and the integer 1 when it is not. The happy-path test never sends page, so the code passed. In production, GET /orders?customer=alice&page=2 crashes with TypeError and returns a 500.

The second error is a deployment bug: os.getenv("WEBHOOK_URL") returns None when the variable is not set, and the code passes it straight to requests.post. Because of the bare except, that failure would also be silent.

bandit -q -c pyproject.toml -r shop -f custom \ --msg-template "{relpath}:{line}: {test_id}[{severity}/{confidence}] {msg}" shop/app.py:12: B105[LOW/MEDIUM] Possible hardcoded password: 'whsec_3f9a1c7e2b' shop/app.py:44: B608[MEDIUM/MEDIUM] Possible SQL injection vector through string-based query construction. shop/app.py:69: B501[HIGH/HIGH] Call to requests with verify=False disabling SSL certificate checks, security issue. shop/app.py:65: B113[MEDIUM/LOW] Call to requests without timeout shop/app.py:71: B110[LOW/HIGH] Try, Except, Pass detected. shop/app.py:76: B201[HIGH/MEDIUM] A Flask app appears to be run with debug=True, which exposes the Werkzeug debugger and allows the execution of arbitrary code. shop/app.py:76: B104[MEDIUM/MEDIUM] Possible binding to all interfaces. [exit 1] Enter fullscreen mode Exit fullscreen mode Seven findings, all real: the f-string SQL (B608, CWE-89), verify=False (B501), no timeout (B113), the swallowed exception (B110), debug=True that exposes the Werkzeug debugger (B201), and binding to all interfaces (B104).

Now look at what is missing. WEBHOOK_SECRET was flagged, but PAYMENTS_API_KEY = "sk_live_..." was not. B105 is name-based: it looks for variables whose names resemble password, pass, passwd, pwd, secret, token or secrete. “API key” is not on that list, so bandit let a live-looking payment key through.

This is why secure AI generated code needs a dedicated secret scanner. detect-secrets combines provider regexes, entropy checks and keyword rules:

$ detect-secrets-hook shop/app.py ERROR: Potential secrets about to be committed to git repo! Secret Type: Base64 High Entropy String Location: shop/app.py:11 Secret Type: Secret Keyword Location: shop/app.py:11 Secret Type: Stripe Access Key Location: shop/app.py:11 Secret Type: Secret Keyword Location: shop/app.py:12 [exit 1] # (help text with links removed) Enter fullscreen mode Exit fullscreen mode It flagged the Stripe-style key three ways (entropy, keyword, Stripe detector) and also caught the webhook secret.

The assistant pinned requests==2.31.0, an older release of the kind models reproduce from their training data. pip-audit resolves requirements.txt and checks every package against an advisory database:

$ pip-audit -r requirements.txt --aliases Found 6 known vulnerabilities in 1 package Name Version ID Fix Versions Aliases -------- ------- --------------- ------------ ----------------------------------- requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56 requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7 requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2 requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56 requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7 requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2 [exit 1] Enter fullscreen mode Exit fullscreen mode Three advisories, each printed twice in this run (the JSON output shows the same duplication; the unique set is three). The first one is directly relevant: CVE-2024-35195 means that once a Session makes its first request to a host with verify=False, later requests to that host keep skipping certificate verification for the life of the pooled connection. Our module disables verification on purpose, which makes this worse, not better.

The fix is requests==2.34.2 and removing verify=False.

If the advisory service is unreachable. pip-audit queries PyPI by default and also supports OSV (--vulnerability-service osv). In my sandbox, the PyPI service worked, but the OSV endpoint was blocked by the network proxy (api.osv.dev... Tunnel connection failed: 403 Forbidden). If your build agents cannot reach either service, do not drop the check. Run pip-audit in a CI job that does have network access, or rely on GitHub’s Dependabot alerts on the repository, and keep --strict so a dependency that cannot be resolved fails the job instead of being skipped.

One more caution from the pip-audit README: pip-audit -r is functionally equivalent to pip install -r, and its isolation is not a security boundary. Do not run it on requirements files you do not trust.

$ vulture shop --min-confidence 60 shop/app.py:1: unused import 'json' (90% confidence) shop/app.py:11: unused variable 'PAYMENTS_API_KEY' (60% confidence) shop/app.py:18: unused attribute 'row_factory' (60% confidence) shop/app.py:31: unused function 'legacy_total' (60% confidence) shop/app.py:38: unused function 'list_orders' (60% confidence) shop/app.py:50: unused function 'discount' (60% confidence) [exit 3] Enter fullscreen mode Exit fullscreen mode vulture found legacy_total, which nothing calls, and the unused PAYMENTS_API_KEY. It also reported list_orders and discount as unused, which is wrong: Flask calls them through @app.route. The vulture README says it directly: code that is only called implicitly may be reported as unused, and it suggests --ignore-decorators "@app.route" or a whitelist.

With 60% confidence for functions, you cannot simply raise the threshold without also hiding legacy_total. In the fixed version I generated a whitelist with vulture shop --make-whitelist.

The two vibe-coded tests pass, and the coverage gate (--cov-fail-under=90 with branch coverage) already says something is off:

$ pytest -q -p no:cacheprovider tests/test_app.py.. ERROR: Coverage failure: total of 44 is less than fail-under=90 shop/app.py 50 27 4 1 44% 17-19, 32-35, 40-47, 52-60, 64-72, 76 FAIL Required test coverage of 90% not reached. Total coverage: 44.44% 2 passed in 0.20s # (output filtered to the relevant lines) Enter fullscreen mode Exit fullscreen mode Coverage is a weak signal for AI code: an assistant can produce 90% coverage with assertions that check nothing. The real value is in tests that state what must always be true. First, API tests that describe the contract the prompt never mentioned:

from decimal import Decimal from flask.testing import FlaskClient def test_customer_filter_is_not_injectable ( client: FlaskClient ) -> None: resp = client. get ( " /orders ", query_string = { " customer ": " x ' OR ' 1 ' = ' 1 " }) assert resp. status_code == 200 assert resp. get_json () == [] def test_page_query_param_is_parsed ( client: FlaskClient ) -> None: resp = client. get ( " /orders ", query_string = { " customer ": " alice ", " page ": " 1 " }) assert resp. status_code == 200 assert [ o [ " id " ] for o in resp. get_json ()] == [ 1 ] def test_bad_page_is_a_client_error ( client: FlaskClient ) -> None: resp = client. get ( " /orders ", query_string = { " customer ": " alice ", " page ": " abc " }) assert resp. status_code == 400 def test_discount_applies ( client: FlaskClient ) -> None: resp = client. post ( " /orders/1/discount ", json = { " percent ": 10 }) assert resp. status_code == 200 assert Decimal ( str ( resp. get_json ()[ " total " ])) == Decimal ( " 90.00 " ) def test_discount_over_100_percent_is_rejected ( client: FlaskClient ) -> None: resp = client. post ( " /orders/1/discount ", json = { " percent ": 150 }) assert resp. status_code == 400 def test_non_numeric_percent_is_rejected ( client: FlaskClient ) -> None: resp = client. post ( " /orders/1/discount ", json = { " percent ": " ten " }) assert resp. status_code == 400 def test_missing_percent_is_rejected ( client: FlaskClient ) -> None: resp = client. post ( " /orders/1/discount ", json = { " pct ": 10 }) assert resp. status_code == 400 def test_unknown_order_is_404 ( client: FlaskClient ) -> None: resp = client. post ( " /orders/999/discount ", json = { " percent ": 10 }) assert resp. status_code == 404 Enter fullscreen mode Exit fullscreen mode Second, property-based tests with Hypothesis. Instead of picking examples, we describe the input space and an invariant, and Hypothesis searches for a counterexample and then shrinks it to the smallest one it can find:

from decimal import Decimal from hypothesis import given from hypothesis import strategies as st from shop.app import apply_discount, paginate money = st. decimals ( min_value = 0, max_value = 1_000_000, places = 2 ) percent = st. decimals ( min_value =- 1000, max_value = 1000, places = 2 ) @given ( total = money, pct = percent ) def test_discount_never_increases_total_or_goes_negative ( total: Decimal, pct: Decimal ) -> None: try: result = Decimal ( str ( apply_discount ( total, pct ))) except ValueError: return # rejecting bad input is allowed; returning garbage is not assert 0 <= result <= total @given ( items = st. lists ( st. integers (), max_size = 60 ), per_page = st. integers ( 1, 25 )) def test_pages_cover_every_item_exactly_once ( items: list [ int ], per_page: int ) -> None: pages = [ paginate ( items, p, per_page ) for p in range ( 1, len ( items ) // per_page + 2 )] assert [ x for page in pages for x in page ] == items @given ( items = st. lists ( st. integers (), min_size = 1 ), page = st. integers ( max_value = 0 )) def test_page_below_one_is_rejected ( items: list [ int ], page: int ) -> None: try: result = paginate ( items, page, 10 ) except ValueError: return raise AssertionError ( f " page= { page } returned { result!r} instead of raising " ) Enter fullscreen mode Exit fullscreen mode Against the vibe-coded module:

$ pytest -q -p no:cacheprovider --tb=short TypeError: unsupported operand type(s) for -: 'str' and 'int' TypeError: unsupported operand type(s) for -: 'str' and 'int' TypeError: can't multiply sequence by non-int of type 'float' KeyError: 'percent' TypeError: 'NoneType' object is not subscriptable E Failing test case: test_discount_never_increases_total_or_goes_negative( E total=Decimal('0.01'), E pct=Decimal('-50.00'), E AssertionError: page=0 returned [] instead of raising E Failing test case: test_page_below_one_is_rejected( E items=, E page=0, FAIL Required test coverage of 90% not reached. Total coverage: 77.78% FAILED tests/test_api.py::test_customer_filter_is_not_injectable - AssertionE... FAILED tests/test_api.py::test_page_query_param_is_parsed - assert 500 == 200 FAILED tests/test_api.py::test_bad_page_is_a_client_error - assert 500 == 400 FAILED tests/test_api.py::test_discount_over_100_percent_is_rejected - assert...

FAILED tests/test_api.py::test_non_numeric_percent_is_rejected - assert 500 =... FAILED tests/test_api.py::test_missing_percent_is_rejected - assert 500 == 400 FAILED tests/test_api.py::test_unknown_order_is_404 - assert 500 == 404 FAILED tests/test_properties.py::test_discount_never_increases_total_or_goes_negative FAILED tests/test_properties.py::test_page_below_one_is_rejected - AssertionE... 9 failed, 4 passed, 4 warnings in 1.09s # (output filtered to the relevant lines) Enter fullscreen mode Exit fullscreen mode Nine failures, each a real defect:

The output also contained ResourceWarning: unclosed database: the vibe-coded get_db() opens a new SQLite connection on every request and never closes it.

Coverage before and after: 44% with the assistant’s two tests, 78% with the guardrail tests against the vibe-coded module, 96% after the fixes, against a 90% gate

The fixed module keeps the same features. What changed:

Full fixed module: after/shop/app.py """ Orders API after the guardrail pass. """ import hashlib import hmac import json import logging import os import sqlite3 from decimal import ROUND_HALF_UP, Decimal, InvalidOperation from typing import Any import requests from flask import Flask, Response, abort, current_app, g, jsonify, request log = logging. getLogger ( __name__ ) CENT = Decimal ( " 0.01 " ) PER_PAGE = 20 MAX_CUSTOMER_LEN = 100 def apply_discount ( total: Decimal, percent: Decimal ) -> Decimal: if not total. is_finite () or total < 0: raise ValueError ( " total must be a non-negative amount " ) if not percent. is_finite () or not 0 <= percent <= 100: raise ValueError ( " percent must be between 0 and 100 " ) return ( total - total * percent / 100 ). quantize ( CENT, rounding = ROUND_HALF_UP ) def paginate [ T ]( items: list [ T ], page: int, per_page: int ) -> list [ T ]: if page < 1 or per_page < 1: raise ValueError ( " page and per_page must be >= 1 " ) start = ( page - 1 ) * per_page return items [ start: start + per_page ] def get_db () -> sqlite3. Connection: if " db " not in g: g. db = sqlite3. connect ( current_app. config [ " DATABASE " ]) g. db. row_factory = sqlite3.

Row conn: sqlite3. Connection = g. db return conn def close_db ( _exc: BaseException | None = None ) -> None: conn = g. pop ( " db ", None ) if conn is not None: conn. close () def parse_page ( raw: str | None ) -> int: if raw is None: return 1 if not raw. isdigit () or int ( raw ) < 1: abort ( 400, description = " page must be a positive integer " ) return int ( raw ) def parse_percent ( body: Any ) -> Decimal: value = body. get ( " percent " ) if isinstance ( body, dict ) else None if isinstance ( value, bool ) or not isinstance ( value, int | float | str ): abort ( 400, description = " percent must be a number " ) try: percent = Decimal ( str ( value )) except InvalidOperation: abort ( 400, description = " percent must be a number " ) if not percent. is_finite () or not 0 <= percent <= 100: abort ( 400, description = " percent must be between 0 and 100 " ) return percent def list_orders () -> Response: customer = request. args. get ( " customer ", "" ) if not customer or len ( customer ) > MAX_CUSTOMER_LEN: abort ( 400, description = " customer is required (max 100 chars) " ) page = parse_page ( request. args. get ( " page " )) rows = ( get_db (). execute ( " SELECT id, customer, total, status FROM orders WHERE customer =? ORDER BY id ", ( customer,), ). fetchall () ) return jsonify ( paginate ([ dict ( r ) for r in rows ], page, PER_PAGE )) def discount ( order_id: int ) -> Response: percent = parse_percent ( request. get_json ( silent = True )) db = get_db () row = db. execute ( " SELECT total FROM orders WHERE id =?

", ( order_id,)). fetchone () if row is None: abort ( 404, description = " order not found " ) new_total = apply_discount ( Decimal ( str ( row [ " total " ])), percent ) db. execute ( " UPDATE orders SET total =? WHERE id =? ", ( str ( new_total ), order_id )) db. commit () notify ( order_id, new_total ) return jsonify ({ " id ": order_id, " total ": str ( new_total )}) def notify ( order_id: int, total: Decimal ) -> None: url = current_app. config [ " WEBHOOK_URL " ] secret = current_app. config [ " WEBHOOK_SECRET " ] if not url or not secret: return body = json. dumps ({ " id ": order_id, " total ": str ( total )}). encode () signature = hmac. new ( secret. encode (), body, hashlib. sha256 ). hexdigest () try: requests. post ( url, data = body, headers = { " Content-Type ": " application/json ", " X-Signature-256 ": signature }, timeout = 5, ) except requests.

RequestException: log. warning ( " webhook delivery failed for order %s ", order_id, exc_info = True ) def create_app ( config: dict [ str, Any ] | None = None ) -> Flask: app = Flask ( __name__ ) app. config. update ( DATABASE = os. environ. get ( " SHOP_DB ", " shop.db " ), WEBHOOK_URL = os. environ. get ( " SHOP_WEBHOOK_URL ", "" ), WEBHOOK_SECRET = os. environ. get ( " SHOP_WEBHOOK_SECRET ", "" ), ) app. config. update ( config or {}) app. teardown_appcontext ( close_db ) app. add_url_rule ( " /orders ", view_func = list_orders ) app. add_url_rule ( " /orders/<int:order_id>/discount ", view_func = discount, methods = [ " POST " ]) return app Enter fullscreen mode Exit fullscreen mode

A third test file covers the webhook: the signature is a valid HMAC, the secret never appears in headers, there is a timeout, and a network error is logged instead of raised.

Webhook tests: after/tests/test_webhook.py import hashlib import hmac from decimal import Decimal from typing import Any import pytest import requests from shop import app as shop_app def test_webhook_is_signed_and_has_timeout ( monkeypatch: pytest. MonkeyPatch ) -> None: sent: dict [ str, Any ] = {} def fake_post ( url: str, ** kwargs: Any ) -> None: sent. update ( url = url, ** kwargs ) monkeypatch. setattr ( shop_app. requests, " post ", fake_post ) app = shop_app. create_app ({ " WEBHOOK_URL ": " hooks.test/orders ", " WEBHOOK_SECRET ": " s3cret " }) with app. app_context (): shop_app. notify ( 7, Decimal ( " 90.00 " )) expected = hmac. new ( b " s3cret ", sent [ " data " ], hashlib. sha256 ). hexdigest () assert sent [ " headers " ][ " X-Signature-256 " ] == expected assert " s3cret " not in str ( sent [ " headers " ]) assert sent [ " timeout " ] == 5 assert " verify " not in sent def test_webhook_failure_is_logged_not_raised ( monkeypatch: pytest. MonkeyPatch, caplog: pytest.

LogCaptureFixture ) -> None: def boom ( url: str, ** kwargs: Any ) -> None: raise requests. ConnectionError ( " down " ) monkeypatch. setattr ( shop_app. requests, " post ", boom ) app = shop_app. create_app ({ " WEBHOOK_URL ": " hooks.test/orders ", " WEBHOOK_SECRET ": " s " }) with app. app_context (): shop_app. notify ( 7, Decimal ( " 1.00 " )) assert " webhook delivery failed for order 7 " in caplog. text def test_webhook_skipped_without_config () -> None: app = shop_app. create_app ({ " WEBHOOK_URL ": "", " WEBHOOK_SECRET ": "" }) with app. app_context (): shop_app. notify ( 7, Decimal ( " 1.00 " )) Enter fullscreen mode Exit fullscreen mode

The same test files, run against the fixed module:

$ pytest -q -p no:cacheprovider --tb=short.............. [100%] ================================ tests coverage ================================ _______________ coverage: platform linux, python 3.13.18-final-0 _______________ Name Stmts Miss Branch BrPart Cover Missing -------------------------------------------------------------- shop/__init__.py 0 0 0 0 100% shop/app.py 89 2 24 3 96% 24, 38->41, 75 -------------------------------------------------------------- TOTAL 89 2 24 3 96% Required test coverage of 90% reached. Total coverage: 95.58% 14 passed in 0.72s [exit 0] Enter fullscreen mode Exit fullscreen mode And the static tools:

$ ruff check --output-format concise. All checks passed! [exit 0] $ mypy Success: no issues found in 2 source files [exit 0] $ bandit -q -c pyproject.toml -r shop -f custom --msg-template {relpath}:{line}: {test_id}[{severity}/{confidence}] {msg} [exit 0] $ detect-secrets-hook shop/app.py [exit 0] $ vulture shop vulture_whitelist.py --min-confidence 60 [exit 0] $ pip-audit -r requirements.txt --aliases No known vulnerabilities found [exit 0] Enter fullscreen mode Exit fullscreen mode Step 8. Lock it in: pre-commit and GitHub Actions Guardrails only work if nobody has to remember to run them. Fast checks go into pre-commit; the test suite runs on git push; CI reruns everything because hooks can be skipped with --no-verify.

# Every hook runs the tool pinned in requirements-dev.txt, so commit-time # checks and CI use exactly the same versions. Install: pre-commit install default_install_hook_types: [ pre-commit, pre-push ] repos: - repo: local hooks: - id: ruff-check name: ruff check entry: ruff check --force-exclude language: unsupported types: [ python ] - id: ruff-format name: ruff format entry: ruff format --check --force-exclude language: unsupported types: [ python ] - id: bandit name: bandit entry: bandit -c pyproject.toml -q language: unsupported types: [ python ] files: ^shop/ - id: detect-secrets name: detect-secrets entry: detect-secrets-hook --baseline.secrets.baseline language: unsupported exclude: ^\.secrets\.baseline$ - id: vulture name: vulture entry: vulture shop vulture_whitelist.py --min-confidence 60 language: unsupported pass_filenames: false types: [ python ] - id: mypy name: mypy --strict entry: mypy language: unsupported pass_filenames: false types: [ python ] - id: pip-audit name: pip-audit entry: pip-audit -r requirements.txt language: unsupported pass_filenames: false files: ^requirements.*\.txt$ - id: pytest name: pytest + hypothesis + coverage gate entry: pytest -q language: unsupported pass_filenames: false stages: [ pre-push ] Enter fullscreen mode Exit fullscreen mode Three choices here are deliberate:

name: guardrails on: pull_request: push: branches: [ main ] permissions: contents: read jobs: checks: runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@v6 - uses: actions/setup-python@v6 with: python-version: " 3.13" cache: pip cache-dependency-path: requirements-dev.txt - run: pip install -r requirements-dev.txt - name: Lint and format run: | ruff check --output-format github. ruff format --check. - name: Types run: mypy - name: Security lint run: bandit -c pyproject.toml -r shop - name: Secrets run: detect-secrets-hook --baseline.secrets.baseline $(git ls-files) - name: Dead code run: vulture shop vulture_whitelist.py --min-confidence 60 - name: Dependency audit run: pip-audit -r requirements.txt --strict - name: Tests, properties, coverage gate run: pytest -q --hypothesis-profile=ci Enter fullscreen mode Exit fullscreen mode The CI job loads a Hypothesis profile with 1,000 examples per property instead of the default 100, registered in tests/conftest.py. I validated both configs and then ran them:

$ pre-commit validate-config.pre-commit-config.yaml [ exit 0] $ actionlint.github/workflows/guardrails.yml [ exit 0] $ check-jsonschema --builtin-schema vendor.github-workflows.github/workflows/guardrails.yml ok -- validation done [ exit 0] $ pre-commit install pre-commit installed at.git/hooks/pre-commit pre-commit installed at.git/hooks/pre-push $ pre-commit run --all-files ruff check...............................................................Passed ruff format..............................................................Passed bandit...................................................................Passed detect-secrets...........................................................Passed vulture..................................................................Passed mypy --strict............................................................Passed pip-audit................................................................Passed [ exit 0] $ pre-commit run --all-files --hook-stage pre-push pytest pytest + hypothesis + coverage gate......................................Passed [ exit 0] Enter fullscreen mode Exit fullscreen mode actionlint and the GitHub workflow JSON schema both accept the workflow. To check the run: steps themselves, a small script (ci_local.py in the repo) executes each step of the workflow locally:

$ python ci_local.py Lint and format exit=0 7 files already formatted Types exit=0 Success: no issues found in 2 source files Security lint exit=0 Files skipped (0): Secrets exit=0 Dead code exit=0 Dependency audit exit=0 No known vulnerabilities found Tests, properties, coverage gate exit=0 14 passed in 5.50s [exit 0] Enter fullscreen mode Exit fullscreen mode Finally, the real test. Here is the kind of change an assistant might add in a follow-up prompt (“add a customer lookup helper”), committed on top of the clean repo:

STRIPE_KEY = " sk_live_51NzQ8vKp2ExampleKey9fT3 " def find_customer ( name: str ) -> list [ sqlite3. Row ]: return get_db (). execute ( f " SELECT * FROM orders WHERE customer = ' { name } '" ). fetchall () $ git commit - m " add customer lookup " ruff check............................................................... Passed ruff format..............................................................

Passed bandit................................................................... Failed >> Issue: [ B608: hardcoded_sql_expressions ] Possible SQL injection vector through string - based query construction. detect - secrets........................................................... Failed Secret Type: Base64 High Entropy String Secret Type: Stripe Access Key vulture..................................................................

Failed shop / app. py: 133: unused variable ' STRIPE_KEY ' ( 60 % confidence ) shop / app. py: 136: unused function ' find_customer ' ( 60 % confidence ) mypy -- strict............................................................ Passed pip - audit............................................( no files to check ) Skipped [ exit 1 ] # (output filtered to the relevant lines) Enter fullscreen mode Exit fullscreen mode The commit was rejected by three different tools for three different reasons. Notice that ruff and mypy passed: the code is well-formatted and correctly typed, and still dangerous.

Matrix of 16 issues in the vibe-coded module against seven tools: which tool caught each issue, and three issues that no tool flagged

Three problems in the original module were not flagged by any tool:

These are context problems: the model has no idea what your business rules are, and neither do linters. The best use of AI-generated code review tools is to automate the mechanical checks, so that human review time goes to the decisions only a person can make.

Is vibe coding bad? On this module, the AI-style code ran, demoed fine, and passed its own tests while hiding SQL injection, a leaked API key, a vulnerable dependency, a crash on a common query parameter and a discount that could raise prices. A pipeline that runs on every commit caught almost all of it. The parts it missed were design decisions, which is where your attention should go anyway.

What does your team run on AI-written code before it merges? I would like to hear which checks caught something real for you, and which ones turned out to be noise.