Command Palette

Search for a command to run...

Back to Projects
AI Lead Discovery Engine
In-progressNext.jsTypeScriptExpress.js+8 more

AI Lead Discovery Engine

Conversational lead sourcing: describe your ideal customer in plain English and an LLM pipeline turns it into a verified, provider-validated search returning hundreds of thousands of prospects.

Timeline

2025 - Present

Role

Senior Full Stack Engineer

Team

Second Brain Labs

Status
In-progress

Technology Stack

Next.js
TypeScript
Express.js
LLM APIs
Langfuse
PostgreSQL
Drizzle ORM
Redis
BullMQ
AWS ECS
Docker

Key Challenges

  • Translating vague human intent into a valid structured query
  • Preventing the model from silently approximating unsupported criteria
  • Keeping extraction reliable when upstream accounts fail mid-job

Key Learnings

  • A verifier loop beats a bigger prompt
  • What the verifier grades against matters more than the generator
  • Ship prompt changes behind measured quality gates

Overview

Most lead tools hand you a wall of filters and expect you to already know how your ideal customer maps onto them. This engine removes that step: you describe the audience you want in your own words, and the system produces a validated search, shows you a scored sample of who it would find, and only then spends your credits extracting the full list.

The interesting part is not the chat interface. It is everything that has to happen between a loose sentence like "founders at small B2B SaaS companies in the US who care about outbound" and a query a data provider will actually accept.

To date the engine has sourced hundreds of thousands of unique prospects.

From conversation to a validated query

A mapping step converts the conversation into a structured search against the parameters the provider genuinely supports.

The failure mode that matters here is not an invalid query. It is a plausible one: a search that looks reasonable, runs without error, and returns the wrong audience. Most of the design work went into making those cases visible rather than letting them pass silently.

The verifier loop

A single prompt, however carefully written, was not reliable enough. Instead, every generated query is graded by a separate verifier and rewritten until it passes.

Getting that loop to converge took several iterations, most of them spent on what the verifier should be grading against rather than on the generation prompt itself. The result moved first-pass acceptance from single digits to near-total on the evaluation set.

Reliability during extraction

Extraction runs as a background job that pulls leads in bounded rounds while tracking cursor state, so a long job resumes rather than restarts.

Sourcing accounts are a limited, shared resource, so the runtime tracks each account's rate budget, fails over to a healthy account when one errors, and takes unhealthy ones out of rotation. Account lifecycle is the subtle part: get the locking wrong and the failure is not a retry you can shrug off, it is two jobs using one account at the same time.

Shipping prompt changes safely

Prompts are versioned and evaluated outside the codebase, so a prompt or model change is released against measured quality rather than a hunch, and can be rolled back the same way.

Tech Stack

Next.js, TypeScript, Express.js, LLM APIs, Langfuse for prompt versioning and evaluation, PostgreSQL with Drizzle ORM, Redis, BullMQ, AWS ECS, and Docker.