How I built a Reddit lead finder with RSS, an LLM scorer and a Node worker

Every time I shipped a side project, I lost evenings doing the same thing: searching Reddit for posts like "is there a tool that does X?" or "any alternative to Y?". Those posts are some of the warmest early leads a solo founder can get, but they are scattered across dozens of subreddits and fall off "new" within hours.

So I built a small system to do the finding for me. This post is about how it's put together, what broke along the way, and what it still can't do. No growth hacks, just architecture.

The app never talks to Reddit directly. The dashboard writes workspaces (a product description, subreddits, keywords) to Mongo; the worker reads them, fetches posts, stores matches and scores them; the dashboard reads matches back. When the app needs something right now (validating a subreddit, triggering a backfill for a new workspace), it calls the worker's search API with a shared secret header.

Why split it? Next.js on a container is great at serving requests and terrible at long-running loops. Keeping the loops in a separate process means a deploy of the web app never interrupts a polling cycle, and a stuck Reddit request never ties up a web worker.

The obvious route is Reddit's.json listings. In my setup those got 403s once traffic went through proxies, so the worker reads the public Atom feeds instead:

RSS gives you less than JSON (no scores, no comment counts), but for "has someone just asked for a thing like mine?" you mostly need title, body, author, link and timestamp, and Atom has all of those.

Requests go out through a proxy pool with a configurable number of "lanes", a minimum gap between requests and a bounded retry count on 403s. All of that is configured through env variables, which brings me to the first lesson.

worker/index.js is basically a handful of timers:

Splitting fetching from scoring was one of the better decisions. Fetching is cheap and bursty; scoring costs money and has rate limits. With separate loops, a slow LLM call never delays the next poll, and a Reddit outage never leaves the scorer idle with a backlog it can't reach.

Keyword matching alone is noisy. "Looking for a CRM" matches a buyer, but so does "I hate every CRM I've tried, rant incoming" and "I built a CRM, roast it". The scorer sends each match plus the workspace's product description to an OpenAI model and asks how likely it is that this post is someone looking for a solution like this product.

The interesting part isn't the prompt, it's the guardrails around it:

For a lead worth answering, the app drafts a reply. The user edits it and posts it themselves. The tool never posts to Reddit on anyone's behalf.

That's a product decision, not a technical limitation. Reddit communities are very quick to spot bots, and I didn't want to build a spam machine that gets its users banned. Drafting saves time; posting stays human.

1. Read env at call time, not at import. The worker is ESM, and imports are evaluated before the body of the entry file. If a module reads process.env.SOMETHING at the top level, it can run before the.env.local loader does, and you get undefined in production but not on your laptop. Now the env loader is the very first import, and modules read config inside functions ( getWebshareKey(), pollExcludeUserIds() ), with a test that fails if a value gets captured at import time.

2. Duplicated schemas will bite you. The app and the worker each have their own Mongoose schemas for the same collections. It keeps the worker deployable without the app's code, but every field change has to be made twice. I've left myself a loud note about it; a shared package would be the real fix.

3. Anything that deletes data gets a kill switch. The cleanup job exports old matches and then deletes them. It ships with a disable flag and refuses to delete if email isn't configured (so the export can't silently fail first). Boring, and exactly what you want at 2am.

4. Make one command the definition of "done". npm run verify runs a syntax check, the worker tests (including the lead-quality fixture and a mocked backfill pipeline test), next build and Playwright end-to-end tests. CI runs the same command on every PR. Solo projects need this more than team ones, because nobody else is going to catch it.

5. Build the web app as an image, not on the server. GitHub Actions builds a Docker image (Next standalone output, Node 22 Alpine, non-root) and the host just pulls it. One gotcha: NEXT_PUBLIC_* variables get inlined at build time, so changing them in the hosting dashboard does nothing until you rebuild.

If you want to build something similar, the core is small: an RSS fetcher, a keyword matcher, a queue of unscored items and an LLM call with strict caps. The rest is making it boring and reliable.

I turned this into a product called IndiePilot (there's a free plan) for founders who'd rather not run the plumbing themselves. Either way, I'm happy to answer questions about the architecture in the comments: what would you do differently?