Most data pipeline blog posts are written by people who ran a script once and got 100 records. This isn't one of those. We've operated the Kairos Signal data pipeline for over 36,000 aggregate hours — scraping, classifying, enriching, and serving structured intelligence to autonomous AI agents. Along the way we've learned exactly what breaks, what doesn't, and what nobody tells you about running AI infrastructure in production.

Here's the architecture, the war stories, and the lessons.

The Pipeline, Step by Step

Every signal in our corpus follows the same path. Understanding this flow is the key to understanding why some pipelines survive contact with reality and others don't.

Layer 1: Collection

Our scraper (propintel-scraper) runs 30 workers in parallel. Each worker is assigned a metro area. Within that metro, the worker queries 12 distinct distress sources:

That's 30 × 12 = 360 concurrent collection streams. The worker cycles through each source, extracting structured signals and dumping them into our ingestion database. Every 5 minutes, a guard process checks that signals are flowing and disk space isn't critical. Lesson 1: Source diversity matters more than source quality. A single source can go down, change its API, or start rate-limiting you. If you have 12 sources per city, losing one is an inconvenience. If you have one source, losing it is a catastrophe.

Layer 2: Ingestion

Raw signals land in ingest.db, a SQLite database running in WAL mode (Write-Ahead Logging). We chose SQLite over PostgreSQL for this layer because:

The ingest layer also normalizes: every record gets a source tag, a timestamp, and a unique ID. Nothing enters the enrichment pipeline without passing through ingest first. Lesson 2: Don't enrich at ingest time. The scraper's job is to get data in. The enricher's job is to make it valuable. Separating these concerns means the scraper can run indefinitely even if the enrichment pipeline is down for maintenance — and vice versa.

Layer 3: Enrichment

This is where the magic happens — and where most pipelines fail.

Our enrichment engine reads raw signals from the ingest database and runs them through a local LLM for classification. We use granite4.1:3b via Ollama, running on an NVIDIA RTX 3070. The model receives a prompt like:

> "You are a distress signal classifier. Analyze this social media post and extract: intent category, location, urgency level, and estimated property value."

The response is parsed into structured JSON. If the intent is real estate distress, the signal is enriched with:

GOLD leads are written to enriched_leads.jsonl and immediately become available in the MCP marketplace. As of today, that file contains 1,007,725 records. Lesson 3: Classification breaks everything. We went through 2,953 enrichment crashes before finding the root cause: the fallback model was qwen3.5:4b, which wasn't loaded in Ollama. Every time our primary model timed out, the enricher tried qwen3.5, got a connection refused, and crashed. The fix was simple — point the fallback at granite4.1:3b — but discovering it took weeks of log analysis. Lesson 4: Local inference is not just a cost play. At 70,000 signals per day, sending each one to a cloud API would cost approximately $420/day at current rates. Local inference costs approximately $0.07/day in electricity. But the real advantage isn't cost — it's latency and privacy. Cloud APIs add 200-800ms per request. Local inference is 50-100ms. And your data never leaves your machine.

Layer 4: Distribution

Enriched leads are served through two channels:

  • Data Hound (/api/v3/): A traditional REST API with Stripe integration (pending live key configuration)
  • Fast Checkout (/mcp/): Our MCP-native terminal that AI agents use to discover and purchase data autonomously

    • Both run behind nginx, which routes traffic through a Cloudflare tunnel. The tunnel provides SSL termination, DDoS protection, and hides our origin server.

    Lesson 5: Serve AI agents first, humans second. Human interfaces require dashboards, forms, email confirmations, and support. AI interfaces require structured JSON, clear pricing, and reliable uptime. When you build for machines first, the human experience improves automatically — because the data is clean, the pricing is transparent, and the delivery is instant.

    What Broke (And How We Fixed It)

    No production pipeline runs without incidents. Here are the ones that taught us the most:

    The qwen3.5 crash loop. We mentioned this above, but it deserves emphasis: 2,953 restarts over a model that wasn't even loaded. The fix was a one-line configuration change. The lesson is that model fallback logic needs to validate model availability before attempting inference. We now run a pre-flight check on startup. The disk space spiral. Our enriched_leads.jsonl file hit 554MB. With only 15GB free on the root partition, the guard process started warning. The fix was implementing log rotation and archiving historical signals to the 1TB data drive. But the real lesson is that JSONL files grow faster than you expect — plan for 5-10x your initial volume estimate. The recon broker dead-end. For months, our recon service was submitting leads to localhost:5002, a mock endpoint that didn't exist. Every single ping-post returned HTTP 400. We fixed it by redirecting recon to our own MCP terminal — now recon-enriched leads feed directly into the marketplace. The lesson: mock endpoints in production configs are landmines. Audit them regularly. The port collision. Fast Checkout and Data Hound both tried to bind port 5007. Fast Checkout crashed on startup. The fix was giving each service its own port (6007 and 5007 respectively) and documenting the port map.

    The Stack: What We Use

    If you're building your own pipeline, here's our stack in full:

    | Component | Technology | Why | |---|---|---| | Scraper | Python, 30 multiprocessing workers | Python's ecosystem for HTTP and parsing is unmatched | | Ingestion DB | SQLite (WAL mode) | Zero-config, single-file, fast enough for 100 writes/second | | Enrichment | Python + Ollama + granite4.1:3b | Local inference, fast classification, no API costs | | Vault | PostgreSQL (hermes_vault) | 48 tables, relational integrity, complex queries | | Web servers | Nginx + Cloudflare Tunnel | SSL termination, caching, DDoS protection | | Process management | PM2 | Auto-restart, log management, zero-config clustering | | Model serving | Ollama (12 models) | Local LLMs without Docker or Kubernetes | | MCP terminal | Custom Python HTTP server | Lightweight, single-file, MCP protocol compliant | | Blog | Custom Python server | Static-ish markdown blog with RSS and sitemap |

    Total cost: approximately $140/month (server + Cloudflare + domain). Compare that to any managed data pipeline service, which starts at $2,000/month and doesn't include the data itself.

    Building Yours

    If you're starting a data pipeline, here's our advice:

  • Start with one source, one city. Get the full pipeline working end-to-end before you scale horizontally. Our first iteration only scraped Dallas, TX foreclosure records. Once that ran for a week without errors, we added cities.
  • Separate collection from enrichment. These are different workloads with different failure modes. Treat them as independent services.
  • Use local LLMs for classification. Cloud APIs are fine for prototyping, but 70,000 API calls per day adds up fast. A local model on commodity hardware handles the same volume for pennies.
  • Build for machine consumption first. JSON over MCP. Clean schemas. Clear pricing. The human interface is secondary.
  • Monitor disk space obsessively. JSONL files grow. Databases grow. Logs grow. Set alerts at 80% usage, not 95%.
  • Audit your configuration files. Mock endpoints, placeholder API keys, and stale URLs accumulate over time. Every quarter, grep your configs for localhost, mock, and placeholder and verify each one.

    • We've open-sourced components of our pipeline and documented the architecture in depth. If you're building something similar, we'd love to hear about it.

    And if you'd rather not build it yourself — we sell the output.