Bewerbungstracker mit Rails 8, pgvector und Claude prueft Anforderungen gegen Belege

A typical job ad lists ten or more requirements. When I apply, the hard part isn't the cover letter; it's answering honestly, for each requirement, can I prove this? And if I can, where is the evidence: which project, which line of my CV?

So I built Evidex, a job application tracker that checks every requirement in an ad against your own CV and project documents, and cites the passages that support each verdict. It's live with a public demo (no sign-up): evidex.henockmullissa.dev, and the code is on GitHub.

This post walks through how it works: the stack, the retrieval pipeline, how one structured Claude call judges every requirement, and the part I'd keep in any LLM project: checking the model's answer on the server before trusting it.

The API is documented with OpenAPI generated from the request specs (rswag), and the frontend's TypeScript types are generated from that same file. CI fails if either drifts, so the backend and frontend can't quietly disagree about a response shape.

When you upload a document (PDF, Word, Markdown or text), a background job:

Why a local embedding model instead of an API? Three reasons. Claude has no embeddings model of its own, so it would mean another provider. A CV is personal data, and with a local model no document text leaves the server just to be indexed. And it costs nothing per document. The quantized model is about 118 MB and understands both English and Finnish, which matters for jobs in Finland.

One detail that's easy to miss with e5 models: they expect a prefix that says whether the text is a passage or a search query.

def embed_passages ( texts ) texts. each_slice ( BATCH_SIZE ). flat_map { | batch | run ( batch. map { | text | "passage: #{ text } " }) } end def embed_query ( text ) run ([ "query: #{ text } " ]). first end Enter fullscreen mode Exit fullscreen mode Step 2: hybrid search Vector search is great at meaning ("background job retries" finds "moved slow integrations into Sidekiq jobs with retries"), but it can miss exact terms like a product name or an acronym. Full-text search is the opposite. So Evidex runs both and merges them:

rankings = { semantic: semantic_ids, keyword: keyword_ids } scores = Hash. new ( 0.0 ) rankings. each do | kind, ids | ids. each_with_index do | id, rank | scores [ id ] += 1.0 / ( RRF_K + rank + 1 ) # RRF_K = 60, the standard constant end end Enter fullscreen mode Exit fullscreen mode Every query is scoped to the signed-in user's own, fully processed documents. That scope lives in one method, so no search path can forget it.

For a match report, Evidex runs the hybrid search once per requirement, keeps the best passages (de-duplicated, at most 60), and numbers them P1, P2, … in a stable order. Then it makes one Claude call for the whole posting, not one per requirement, which keeps cost and latency down and lets the model see all the evidence at once.

The prompt puts the requirements and passages in clearly delimited blocks:

<requirements> R1 [required] 5+ years of professional Ruby on Rails experience R2 [required] Strong PostgreSQL skills, including query tuning... </requirements> <passages> <passage id="P1" document="Alex Example – CV" section="Summary"> Backend developer with six years of experience building Ruby on Rails applications... </passage>... </passages> Enter fullscreen mode Exit fullscreen mode The system prompt says plainly that both blocks are data to assess, never instructions, even if they contain text addressed to an AI. Job ads and CVs are untrusted input, and the model has no tools, so an injected instruction has nothing to act on.

The response must follow a JSON schema: for every requirement, one verdict ( evidenced, transferable or gap ), one or two sentences of reasoning, and the ids of the passages it relied on, plus a short list of strengths and gaps. With structured outputs, I never parse free text.

This is the part I'd keep in any LLM feature. A model can say "evidenced" and cite a passage that doesn't exist, or cite nothing at all. So before saving, the server:

cited = Array ( match [ "passage_ids" ]). map ( &:to_s ). uniq. filter_map { | id | passages [ id ] } verdict = RequirementMatch:: VERDICTS. include? ( match [ "verdict" ])? match [ "verdict" ]: "gap" if verdict!= "gap" && cited. empty? verdict = "gap" reasoning = " #{ reasoning } (Downgraded: no supporting passage was cited.)" end Enter fullscreen mode Exit fullscreen mode Cited passages are stored as snapshots (document title, section and excerpt), so a report still reads correctly after the CV is edited or deleted. Each run creates a new version, and when you override a verdict or add a note, that override carries over to the next version as long as the requirement's text hasn't changed.

The result is a report I can actually rely on: every "evidenced" line comes with the passage that proves it, one click away.

Every AI call goes through one wrapper that records the model, input and output tokens, latency and whether it succeeded. Each account has a daily AI limit, and the admin area shows calls and estimated cost per account.

The public demo needed extra care, because anyone can open it without signing up:

Production is one Docker image that runs Rails, the built SPA and the background jobs. Solid Queue runs inside Puma in async mode, so the embedding model is loaded once per process instead of once per worker. With the model loaded, the app uses roughly 620–670 MB of memory, which fits on a small Arm server at Hetzner. Kamal handles zero-downtime deploys, HTTPS through kamal-proxy with Let's Encrypt, and PostgreSQL as an accessory on the same machine.

Security got the same attention as the AI parts: two-factor sign-in with recovery codes, re-authentication before sensitive changes, security emails for new devices, a personal activity log, SSRF-safe fetching when importing a job ad from a link, and request specs that prove one user can never read another user's data, admins included.

I'd love to hear how you validate LLM output in your own projects, or what you'd add to a tool like this.