Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

Posted on Oct 16 • Originally published at georgeezejiofor.com

Self-Hosted Databricks Alternative: Homelab Data Lakehouse

Part of the homelab build series. Trino, Apache Iceberg, Project Nessie, dbt, Spark, MLflow and OpenMetadata on RKE2, with highly available state on Ceph and CloudNativePG, secrets from Vault and sign-in through Keycloak. Raw data comes in at one door, gets built and inspected on the line, and leaves as tables and models I can trust. Nobody sends an invoice.

A factory you rent by the hour, or one you own

Raw data is worth about as much as raw steel. Nobody wants it until someone has cut it, joined it, checked it and stamped it.

Snowflake, Databricks and BigQuery will rent you an excellent factory for that, and the meter runs for as long as the lights are on. I wanted the factory without the meter, so I built one on three mini-PCs in my homelab, on the same RKE2 cluster as everything else in this series. Every station is a real open-source tool doing a real job.

It's built to make one thing: machine-learning models trained on data I can name down to the exact version. One is already serving, a football predictor that a scheduled pipeline refits and re-tests every day.

Here is the floor, in the order the material moves:

The utilities were already in the building: Ceph, CloudNativePG, Vault, Keycloak, Istio and ArgoCD.

It's highly available where it holds state. Postgres runs three instances, one in each datacenter, the ledger runs two replicas and search runs three nodes. Trino, Superset, OpenMetadata and MLflow run a single replica each, and I come back to that below. One bay is still empty: a Hermes AI copilot that will read the lineage over OpenMetadata's MCP interface, waiting on a DGX Spark that hasn't arrived.

If you know Databricks, you already know the layout:

It's an open-source equivalent, not Databricks, which is a product and a trademark, and there's no affiliation. This is the second design: the first used Hive Metastore and Airflow, and I say below why I replaced both.

The brief I set myself: an MLOps and big-data platform on the existing homelab RKE2 cluster that a real data team could plausibly use. The homelab's existing backends (Rook-Ceph, CloudNativePG, Vault, Keycloak) had to thread through every component, and each piece had to land in GitOps and come back after a node reboot with nobody touching it.

What landed is two tiers. The lakehouse: Ceph RGW object storage, Apache Iceberg tables, a Project Nessie catalog (state in CNPG nessie_db ), Trino from Stackable's coordinated 25.3.0 release, dbt-trino for tested marts, and OpenMetadata for governance, with the Kubeflow spark-operator installed for the day the data outgrows SQL. MLOps on top: MLflow Tracking and the Model Registry, backed by CNPG mlflow_backend_db and the mlflow-artifacts bucket, behind oauth2-proxy for the browser and reachable directly over in-cluster DNS from Kubeflow notebooks.

The constraints that shaped every decision:

The rest of this article covers how each piece lands, what I cut, and which traps cost the most time.

If the word is new, the lakehouse is the third step in a short history. A warehouse gives fast SQL over the tables you planned for, in a format you rent. A lake keeps any file cheaply but has no table rules, so teams copy the data into a warehouse anyway and end up with two versions of the truth. A lakehouse puts the table rules (Iceberg) and the versions (Nessie) on the open files themselves, so every engine reads one copy.

Warehouse, lake, lakehouse: a warehouse sells storage and engine together, a lake plus a warehouse keeps two copies, and the lakehouse built here keeps one copy of open files in Ceph with Iceberg tables and Nessie versions read by Trino, dbt, Kubeflow and MLflow

The data factory:** dbt, Trino, Nessie, Kubeflow, MLflow and KServe on the line, Superset and OpenMetadata above it behind Keycloak, and Ceph, CloudNativePG, Vault, Istio and Argo CD underneath

With five guided views: the query path, the daily pipeline, governance, train-and-serve, and where secrets come from. Blog platforms can't embed the HTML, so host it and link it.

Kubeflow appears here as one station, the machine shop; how I changed it from upstream is in the SSO section. Read the diagram bottom-up, because that's the order the dependencies run:

Everything in the foundation layer predates this work. Most of my effort went into making sure the upper layers didn't reinvent any of it. The one deliberate exception is OpenMetadata's own OpenSearch; the going-live section explains why that stopped being optional.

Stackable ships a coordinated release of operators for Hadoop-ecosystem services. It's tempting to take all of them. I only took the parts that fit:

The Stackable operators pin to 25.3.0 and release as a set, so I bump them as a set (never Trino alone). Project Nessie and OpenMetadata are not Stackable products; they deploy from their own Helm charts, the price of a best-of-breed catalog and governance layer.

The secret-operator needs explaining. The workload operators' admission webhooks reference the SecretClass CRD, and without it workload CR creation fails. So I install the operator but create no SecretClass of my own. The chart's default tls SecretClass issues Trino's internal (autoTls) certificates, and the dbt and ingestion pods mount its CA to verify them. Credentials never go through it; they're plain v1/Secret s synced by VSO from Vault.

Stackable workloads also brought the first Istio trap. Under STRICT mTLS (the cluster default), the API server's webhook calls can't reach the operator pods, because the API server has no mesh client cert. The fix is a PeerAuthentication: PERMISSIVE in the mlops namespace. Don't turn off injection; relax the mode for that namespace and nothing else.

The homelab already runs Rook-Ceph with four storage classes: ceph-block (RBD, RWO, three copies, the default for notebook PVCs and most stateful pods), ceph-block-pg-local (RBD, one copy, pinned to a datacenter, for Postgres only), ceph-filesystem (CephFS, RWX, only when a workload really needs shared scratch) and ceph-bucket (RGW, S3), the data lake.

RGW is TLS-only. Clients use https://s3.georgehomelab.com, which terminates on the Istio gateway with a public Let's Encrypt certificate (no private CA to distribute), and path-style addressing still works. The old plain-HTTP …rook-ceph.svc:80 endpoint is gone.

Buckets come from Rook's ObjectBucketClaim CR. The OBC controller provisions the bucket on RGW and creates a per-bucket Secret with AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY keys, named after the claim: Spark reads mlops-data-lake, MLflow reads mlflow-artifacts, KFP reads kfp-artifacts. Nothing fans out from a single root key.

apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: mlops-data-lake namespace: mlops spec: bucketName: mlops-data-lake storageClassName: ceph-bucket Enter fullscreen mode Exit fullscreen mode Apply that and the bucket, its owner user and the Secret all exist, ready for envFrom.

The Ceph dashboard → Object Gateway → Buckets: mlops-data-lake, mlflow-artifacts and kfp-artifacts, each with its own OBC-issued owner.

The easy path is to leave every chart's postgresql.enabled: true on and accept the extra StatefulSets. I didn't: backups, upgrades, monitoring and HA belong on one tier, not nine. The homelab's CNPG pattern already runs Keycloak, SonarQube, Tekton and others, so one more cluster meant copying an existing manifest (trimmed here; the real one adds managed roles, affinity and resources):

apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: postgres-mlops-cnpg-cluster namespace: mlops spec: instances: 3 imageName: ghcr.io/cloudnative-pg/postgresql:17 bootstrap: initdb: database: mlops_placeholder # real DBs come from Database CRs owner: postgres secret: name: mlops-cnpg-bootstrap # VSO-synced from Vault storage: storageClass: ceph-block-pg-local # one Ceph copy, in the instance's own DC size: 40Gi postgresql: parameters: max_connections: " 200" backup: retentionPolicy: " 7d" barmanObjectStore: destinationPath: " s3://cnpg-backups/postgres-mlops-cnpg-cluster" # Ceph RGW Enter fullscreen mode Exit fullscreen mode Three instances, one per DC, automatic failover, and continuous WAL archival to RGW.

The storage class is worth a sentence. Postgres already keeps three copies of the data, one per instance. On the default ceph-block, Ceph kept three copies of each of those, nine in all, and every commit waited on a write to another datacenter. So Postgres now sits on ceph-block-pg-local: one Ceph copy, in the same datacenter as the instance. Postgres does the replicating and Ceph just holds the disk. A 4K synced write on a database node went from about 34 ms to between 15 and 24 ms. The price is that losing a drive loses that instance, and the operator has to clone a new one from the primary. CNPG's Database CR (its per-logical-DB declaration) provisions one logical database per app, each with its own owner role whose password lives at its own Vault path:

The lakehouse pivot added nessie_db and openmetadata_db with a short Database CR each, and no new stateful tier. The old hive_metastore_db and airflow_metadata_db (and their hive / airflow roles) are retained but no longer used: their Database CRs set databaseReclaimPolicy: retain, so nothing drops them by accident.

Each app sees only its own DB, with its own creds. The superuser never lands on an application Pod.

kubectl get database.postgresql.cnpg.io -n mlops: one Database per app on the single postgres-mlops-cnpg-cluster, all APPLIED true, including the retained (unused) hive_metastore_db and airflow_metadata_db.

Two Istio nuances are specific to CNPG. First, its istio-injected pods cycle their sidecars whenever Istio config changes, so the READY column bounces between 1/2 and 2/2 and the operator reports READY < INSTANCES. Postgres is fine; it's sidecar churn. Don't triage CNPG; bump the sidecar resources or ignore it.

Second, any istio-injected namespace must run at PSS=privileged, because istio-init needs NET_ADMIN/NET_RAW. CNPG's initdb Job lives in such a namespace. If PSS is baseline, the Job goes FailedCreate with no obvious cause. That's why the mlops namespace is PSS=privileged.

Vault is the single source of truth for credentials, and Terraform is the only thing that writes to it. The chain for any credential: Terraform reads the value from a gitignored.secrets.auto.tfvars and writes a Vault KV-v2 path; a VaultStaticSecret CR names that path, the Vault role and the Secret to produce; VSO reconciles it hourly into a plain v1/Secret (optionally reshaping keys with a template); the workload mounts it.

apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: mlops-mlflow-db namespace: mlops spec: vaultAuthRef: mlops-mlflow mount: secret type: kv-v2 path: homelab/mlops/mlflow-db refreshAfter: 1h destination: name: mlops-mlflow-db create: true overwrite: true Enter fullscreen mode Exit fullscreen mode Don't drop the overwrite: true. If the same Secret is referenced from both a kustomize secretGenerator (with placeholder data) and a VSO sync, every kubectl apply -k briefly stomps VSO's data with the kustomize placeholders. With overwrite: true on the VSS, VSO puts the real values back on its next reconcile. Without it, oauth2-proxy gets unauthorized_client from Keycloak and you spend two hours blaming the realm import.

Each app has its own ServiceAccount, its own Vault role (Kubernetes auth, policy scoped to its own paths) and its own VSS, so no credential crosses an app boundary.

The Vault UI → secret/ → homelab/mlops/: the list of per-app paths ( mlflow-db, nessie-db, openmetadata-db, …). ⚠️ Show the path list only;.

Each app gets its own realm too: MLflow's proxy uses realm mlflow, and OpenMetadata (which speaks OIDC natively) uses realm openmetadata.

Two more lessons came out of this. In oauth2-proxy's proxy mode (which I use for MLflow), the user email reaches the upstream as X-Forwarded-Email, not X-Auth-Request-Email. The latter is a response header for auth_request -style integrations. Get it wrong and the upstream app sees no user identity even though authentication "worked."

And Keycloak doesn't substitute environment variables at realm import. The realm JSON includes the oauth2-proxy client, but its secret is the literal placeholder string ${TOOL_OAUTH2_PROXY_CLIENT_SECRET}. Every new realm needs a post-import kcadm update clients/<id> -s secret=<actual> patch with the same value that's in Vault. post-deploy-fixes.sh automates it, but "Invalid client credentials" on a fresh realm still catches you out the first time.

The Keycloak sign-in page for the mlflow realm, with the "Sign in with Google" and "GitHub" buttons. Every UI in this stack sends you to a page like this first.

Upstream Kubeflow ships its own login stack: Dex as the identity provider, with oauth2-proxy acting as an Istio ext_authz check in front of it. I don't run Dex at all.

What changed, checked against the live cluster:

The catalog is the small service everything else depends on. It remembers that "table events_curated is an Iceberg table whose current metadata lives at s3://mlops-data-lake/warehouse/..., with these columns, these snapshots, these partitions." Replacing the Hive Metastore with Project Nessie was the main reason for the pivot.

Nessie deploys from its own Helm chart ( charts.projectnessie.org ), not Stackable. Its version store is Postgres (CNPG nessie_db ), and its warehouse is the same mlops-data-lake bucket Spark writes to:

versionStoreType: JDBC2 jdbc: jdbcUrl: " jdbc:postgresql://postgres-mlops-cnpg-cluster-rw.mlops.svc:5432/nessie_db" secret: name: mlops-nessie-db username: username password: password catalog: enabled: true iceberg: defaultWarehouse: warehouse warehouses: - name: warehouse location: s3://mlops-data-lake/warehouse storage: s3: defaultOptions: endpoint: https://s3.georgehomelab.com pathStyleAccess: true accessKeySecret: name: mlops-data-lake awsAccessKeyId: AWS_ACCESS_KEY_ID awsSecretAccessKey: AWS_SECRET_ACCESS_KEY Enter fullscreen mode Exit fullscreen mode Trino reaches it at http://nessie.mlops.svc:19120/iceberg. Spark uses org.apache.iceberg.nessie.NessieCatalog against.../api/v2 and can create and merge branches from a job.

The Nessie UI on main: the commit log and the namespaces ( raw, raw_football, marts_staging, marts_marts, marts_football ) that dbt and Trino have created.

Why Nessie and not lakeFS? Nessie already gives the Iceberg tables Git-style branches, tags and atomic multi-table commits; a second version-control layer over the same tables would be two sources of truth for one question. lakeFS earns its place for unstructured files (I have none yet), and its Iceberg REST catalog, SSO and RBAC are paid-edition features. So I'll revisit it as a replacement, never an addition, if Nessie stalls or unstructured data shows up.

Garbage collection. Versioning has a cost that's easy to miss: every dbt build and pipeline publish replaces table snapshots, and the old files stay forever unless something removes them. Nessie's separate GC tool runs as a weekly CronJob (Sundays, 13:00 UTC).

It marks the files still reachable (30 days of history per branch, and just the tagged commit for each train-football-… tag) and treats the rest of each table's files as candidates. It only looks under Iceberg table locations, so the landing Parquet is out of its reach. It runs with --defer-deletes for now, recording candidates in nessie_db until I've reviewed a few runs. The first walked 3 references and 131 commits and found 8 candidates, all old table metadata.

A daily tag-retention job sits in front of it, so the pins don't pile up either. It keeps any tag a Model Registry version names, plus the last 30 days, and is also in dry-run mode for now.

MLflow is the simplest part of the stack to deploy and the one I use most day to day. The chart is the official one from the MLflow repo, vendored at mlflow/charts/mlflow (chart 0.1.0, appVersion 3.12.0). It takes a Postgres backend and an S3 artifact store.

mlflow: backendStoreUriFrom: secretKeyRef: name: mlops-mlflow-db # VSO builds the `uri` key from Vault key: uri defaultArtifactRoot: mlflow-artifacts:/ # clients use the proxy scheme artifactsDestination: s3://mlflow-artifacts/ # the server does the S3 I/O env: - name: MLFLOW_SERVER_ENABLE_JOB_EXECUTION value: " false" - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: mlflow-artifacts key: AWS_ACCESS_KEY_ID - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: mlflow-artifacts key: AWS_SECRET_ACCESS_KEY - name: MLFLOW_S3_ENDPOINT_URL value: https://s3.georgehomelab.com - name: MLFLOW_S3_IGNORE_TLS value: " false" Enter fullscreen mode Exit fullscreen mode Artifacts are proxied: clients log to mlflow-artifacts:/ and the server writes to the bucket with its own OBC credentials, so notebooks only need the tracking URI, not S3 keys. The MLFLOW_S3_ENDPOINT_URL override points the server at RGW instead of AWS. The endpoint has a public certificate, so MLFLOW_S3_IGNORE_TLS stays false: TLS is genuinely verified.

Path-style addressing works because the host is in the CephObjectStore's hosting.dnsNames.

MLFLOW_SERVER_ENABLE_JOB_EXECUTION=false is there because MLflow 3.x's job subsystem is on by default and starts eight huey consumers, which ate about 1.5Gi while completely idle and kept OOM-killing the pod. I don't use server-side jobs, so turning it off beat raising the limit.

import mlflow mlflow. set_tracking_uri ( " http://mlflow.mlops.svc.cluster.local:5000 " ) mlflow. set_experiment ( " homelab-iris-v1 " ) with mlflow. start_run (): mlflow. log_param ( " lr ", 1e-3 ) mlflow. log_metric ( " val_acc ", 0.97 ) mlflow. sklearn. log_model ( model, " model " ) Enter fullscreen mode Exit fullscreen mode There's no SSO round-trip on the SDK path; any pod the Network Policy allow-lists can reach the in-cluster service. Only the browser UI goes through Keycloak.

The MLflow UI → experiment football-match-outcome: the daily runs with wf_logloss and deploy_approved as columns.

v1 had an Airflow tier here, which the pivot deleted. Airflow isn't bad, but on a homelab with one operator a persistent scheduler is a lot of surface area for what's really three jobs a day. In its place is a query/transform/govern tier that does more and needs less babysitting.

Trino: SQL over Iceberg. The Stackable TrinoCluster runs coordinator-only: workers are scaled to 0 and node-scheduler.include-coordinator is true, so the coordinator runs the splits itself. No app node has room for a 4Gi/1-CPU worker right now; workers come back with more hardware. Trino queries Iceberg through the Nessie REST catalog. In 25.3.0 the first-class iceberg connector models the Hive metastore, so the REST wiring goes through the generic connector.

Nessie vends credentials for reads, but Trino also needs its own S3 keys to create tables (more on that in the going-live section). They come from the Rook bucket Secret via valueFromSecret, never as literals:

spec: connector: generic: connectorName: iceberg properties: iceberg.catalog.type: value: " rest" iceberg.rest-catalog.uri: value: " http://nessie.mlops.svc:19120/iceberg/" iceberg.rest-catalog.vended-credentials-enabled: value: " true" s3.endpoint: value: " https://s3.georgehomelab.com" s3.path-style-access: value: " true" s3.aws-access-key: valueFromSecret: name: mlops-data-lake key: AWS_ACCESS_KEY_ID s3.aws-secret-key: valueFromSecret: name: mlops-data-lake key: AWS_SECRET_ACCESS_KEY Enter fullscreen mode Exit fullscreen mode dbt: tested transformations. dbt-trino turns raw Iceberg tables into modeled, tested, documented marts. It runs as a CronJob at 09:30 UTC. The project ships as a kustomize-generated dbt-project ConfigMap (no git clone at runtime), and a non-root container runs dbt build. dbt's not_null, unique, accepted_values and accepted_range tests are the data-quality gate; there's no separate Soda/GE.

The dbt profile needs no secret at all. dbt proves who it is with a short-lived Kubernetes token for its own service account, mounted into the pod and sent as a JWT, and Trino's rules then decide what dbt-runner may read and write (how that works is the subject of the RBAC article in this series). It also verifies Trino's certificate against the Stackable tls CA, mounted per pod. After each run it publishes manifest.json, catalog.json and run_results.json to a small CephFS volume for OpenMetadata.

dbt writes on a branch, then publishes (write-audit-publish). At first, each dbt build landed in main as roughly 15 separate Nessie commits ( __dbt_tmp renames, __dbt_backup drops), so a query at the wrong moment could catch a half-built mart. Now the 09:30 CronJob does three things:

main now gets one atomic "Merged dbt-audit" commit per build. A failure leaves main untouched and the branch sitting there for me to inspect. A normal run ends PASS=23 (3 models and 20 tests) and merges cleanly.

spec: connector: generic: connectorName: iceberg properties: iceberg.catalog.type: value: " rest" iceberg.rest-catalog.uri: value: " http://nessie.mlops.svc:19120/iceberg/dbt-audit/" Enter fullscreen mode Exit fullscreen mode A few details kept this from leaking into the rest of the stack. OpenMetadata's manifest still comes from dbt docs generate --target prod, so lineage keeps its iceberg.* names, and its Trino crawl only includes ^iceberg$, so the audit catalog never shows up as a duplicate. The dbt runner got an Istio AuthorizationPolicy rule that allows only /api/v2/trees*, not the Iceberg REST path that vends S3 credentials. And because adding a Trino catalog restarts the coordinator under Stackable, I rolled it out between the scheduled runs, not during one.

Bronze, silver, gold. The tables fall into the three layers Databricks calls a medallion architecture: bronze is data as it lands, silver is typed and cleaned, gold is tested and ready to use. Here every layer is a set of Iceberg tables, and the gate between them is the dbt-audit branch. OpenMetadata labels each table with its layer (its Certification field), so a reader can filter the whole catalog down to Gold.

This is ELT rather than the warehouse's ETL: data is loaded first, almost as it arrives, and transformed afterwards inside the lakehouse. Keeping the raw layer is what makes that safe: when a dbt model is wrong, I fix the model and rebuild from bronze, without fetching anything again.

Bronze, silver, gold: raw_football.matches and raw.events are bronze, stg_events is silver, the two marts are gold; dbt builds silver and gold on the dbt-audit branch, the football model trains on bronze at a pinned tag, and people read gold through Trino and Superset, with lineage in OpenMetadata.

Two details I'd rather show than smooth over. Football has no silver step: dbt builds the gold mart straight from the raw table. And the football model trains on bronze, at a pinned tag, because it needs the exact rows it saw, not a cleaned view that could change underneath it. So raw_football.matches is Bronze and Tier 1 at the same time: the layer says how refined a table is, and the tier says how much depends on it.

OpenMetadata → Explore → Data Assets: Table, Certification: Gold. The two gold marts with their badge, tier and owner.

OpenMetadata: governance, lineage and MCP. The top layer is a metadata knowledge graph that crawls Trino and ingests dbt's artifacts to produce end-to-end lineage, descriptions and data-quality results. It runs 2.0.2, with CNPG for its store, its own 3-node OpenSearch 3.4.0 for search (operator-managed, on the database node pool), and Keycloak realm openmetadata for OIDC; login redirects straight to Keycloak's Google/GitHub page.

Ingestion runs as two CronJobs ( om-ingest-trino at 10:00, om-ingest-dbt at 10:15), not the bundled Airflow (I'm not bringing back what I just removed). They authenticate with the ingestion-bot token Terraform writes to Vault. OpenMetadata is in this homelab at all because it ships an MCP server, which lets the Hermes agents query certified, lineage-aware data in natural language. The rest of the stack is building towards that.

OpenMetadata → marts_marts.daily_active_users → Data Quality tab: the dbt tests with their latest results.

Every lakehouse diagram has Spark in the middle, so I should be straight about mine. The Kubeflow spark-operator is installed and has run test jobs, but nothing on the daily line uses it. My tables are small and Trino builds them in seconds, so Spark on that line would be a forklift carrying one box. It joins in its own article, on a job that needs it: real-time fraud detection with Kafka, once the cluster has more memory. It will read and write the same Iceberg tables through Nessie, so nothing about the catalog or the storage has to change.

A factory that stops when one machine does isn't much of a factory. Here's what this one survives, and how I know:

kubectl get cluster.postgresql.cnpg.io -n mlops: postgres-mlops-cnpg-cluster with 3 instances, 3 ready, and its current primary.

Everything above was designed before it ran. Once it ran, it met a cluster that powers off at night and defaults that fail without an error. Three traps shaped the platform enough to belong here. The rest, from an operator chart pointing at an image that was never published to a catalog that would hand storage credentials to any pod that asked, have their own article in this series.

Trino could query, but never create a table. Every CREATE TABLE failed:

Failed checking new table's location: s3://mlops-data-lake/warehouse/raw/... -> Failed to list location -> SdkClientException: Unable to load credentials from any of the providers in the chain Enter fullscreen mode Exit fullscreen mode Trino checks a new table's location with its own credentials before Nessie gets involved, and it had none. That gap alone is why dbt had never built anything. The fix gives Trino the same bucket key Nessie already vends (so no new access).

The cron that never fired. dbt-build had been scheduled for 03:00 UTC for eleven days and never ran once, with no error, event or Job. Thanos had the answer: the lab powers off overnight, and there isn't a single metric sample at 03:00 in the retention window. Every CronJob moved into daytime (09:30 dbt → 10:00 Trino crawl → 10:15 dbt lineage), with startingDeadlineSeconds so a missed slot is skipped, not replayed at boot.

A quota on limits can block a model rollout. Every Kubeflow profile namespace gets a ResourceQuota, generated by a Kyverno policy, and mine capped CPU limits at 8. A KServe rollout runs the old and the new model server side by side until the new one is ready.

Each asks for 2.5 CPU of limit, and one of those is the Istio sidecar's. Add one running notebook and the pipeline step itself and the total is 8.2. So the new revision was refused, the deploy step waited fifteen minutes and timed out, and the run failed with a perfectly good model sitting in the bucket.

The symptom is a revision stuck on FailedCreate with exceeded quota in its events and no pod to look at. I raised the limit quota to 16 and left the request quota at 4, because requests are what reserve real capacity. Then the change hit a second trap: ArgoCD reported the policy synced while the live object was still on its first generation with the old value.

Check metadata.generation on a policy before believing a sync.

One day on the factory floor: every schedule in the lakehouse in the order it fires, from the morning power-on to the afternoon backups

From a data scientist's seat, the platform looks like this:

mlflow. set_tracking_uri ( " http://mlflow.mlops.svc.cluster.local:5000 " ) mlflow. sklearn. log_model ( model, " model ", registered_model_name = " event-clf " ) Enter fullscreen mode Exit fullscreen mode Because the data lives on a Nessie tag, the run is reproducible: months later you can point back at the exact catalog state. That works through Nessie (pyiceberg reads the table at the tag). Trino's catalogs are each pinned to one branch, so a SQL query sees the current version, not last month's.

From match results to a served model: football-data.org to landing Parquet, published to an Iceberg table through a Nessie branch, then Katib, the promotion gate, KServe, MLflow and the predictions site

A KFP RecurringRun fires at 10:00 London time every day. A Job in mlops, the only namespace with internet egress, pulls Premier League results into s3://mlops-data-lake as Parquet. The lakehouse part is the publish step:

Katib and the training step read exactly that snapshot, and MLflow and the Kubeflow Model Registry record the tag, the commit and the Iceberg snapshot id beside the model. dbt's football_matches model reads the same table.

The model is modest, and I'd distrust anything better: a walk-forward log-loss of 1.050 against 1.082 for always guessing the usual split, over 587 matches, and the right result 47% of the time. A new model has to pass a promotion gate, a smoke test and a canary at 10% then 50% of traffic before it serves. The full story, from the features to the rollout, is its own article in this series.

epl-predict.georgehomelab.com: the matchday predictions with the scorecard above them, and the goals tables with their "does not beat that base rate" line.

This workflow is how I know the integration points actually work: S3 endpoints resolve, Nessie answers Iceberg REST, dbt tests gate the marts, MLflow logs to Postgres and S3, and a notebook in a different namespace reaches everything over in-cluster DNS.

Apache Superset is best known for charts and dashboards, but it also ships SQL Lab: a SQL editor in the browser. Point it at Trino and it becomes a SQL client for the whole lakehouse, with no desktop tool to install and no password to hand out.

Here it signs you in through Keycloak, then sends each query to Trino as you, not as a shared Superset account. Trino applies the same per-person rules it applies everywhere else, and the query lands in the audit table under your email.

Apache Superset client connected to Trino, running SELECT * FROM iceberg.marts_marts.daily_active_users and returning rows: the query v1 couldn't run without Spark.

A list, with reasons. Most of these deserve an article of their own.

This build cost about three weekends of focused work, and most of the hours went on the traps, not the happy paths.

What I'd tell anyone trying this on their own metal:

The part I lean on most is Git-for-data. dbt publishes each build to main as one tested commit, and the football model trains on a tagged snapshot that MLflow and the Model Registry both remember. It stopped being a slide.

The ArgoCD UI filtered to mlops-* & kubeflow-*: the whole lakehouse (Trino, Nessie, dbt, OpenMetadata, MLflow, CNPG) Synced and Healthy.

The next milestones, and probably the next few articles:

If you want to see how those go, including the parts that break, follow along.

Follow me on LinkedIn for the next posts in this series.

Expand DEV DEV DEV Follow Joined Oct 16, 2026 • Oct 16 Dropdown menu Copy link Hide Dev-verify