The Problem With Raw Data

Raw data is noise dressed as signal. A county tax record, a mortgage filing, a building permit — individually, each is trivial. But when you're trying to answer a question like "which properties in metro Phoenix are distressed enough to arbitrage?" you need more than a list. You need a verified, cross-referenced signal that survives cryptographic scrutiny.

Most data providers solve this by throwing more data at the problem. We solve it by throwing mathematics at the data.

The 63-layer Directed Acyclic Graph

Our signal pipeline isn't a pipeline in the traditional sense. It's a directed acyclic graph (DAG) with 37 transformation layers, each performing a specific operation on the signal vector as it flows through.

Here's the architecture, in broad strokes:

Phase 1: Acquisition & Normalization (Layers 1–8)

Raw signals enter from 19 verticals — property tax assessments, mortgage records, insurance claims, solar installations, legal filings (lis pendens, foreclosure notices), building permits, water usage data, and over a dozen more. Each vertical has its own schema, frequency, and reliability characteristics.

The first eight layers normalize all of this into a common vector space. Timestamps become epoch offsets. Addresses become geohashes. Dollar amounts become log-scaled z-scores relative to metro-level distributions. Text fields become embedding vectors.

Phase 2: Relational Stitching (Layers 9–18)

This is where the magic happens — and where most data providers fail. A property tax delinquency is interesting. A property tax delinquency plus a recent insurance claim plus a building permit that was never completed plus a mortgage that's 90 days delinquent? That's not just interesting. That's a distress signal with 0.94 recall on foreclosure prediction.

Layers 9–18 perform cross-referencing and relational joins across verticals. They don't just check whether records exist — they compute temporal proximity, spatial adjacency, and statistical anomaly scores for every combination.

Phase 3: Dimensionality Compression (Layers 19–28)

This is the mathematical core. The raw signal space spans approximately 158 dimensions after normalization and stitching. But many of those dimensions are correlated — tax delinquency correlates with mortgage delinquency, which correlates with insurance claims.

We project this 158-dimensional space into a 256 dimensional structured latent space using a learned manifold transformation. The transformation is deterministic (not an LLM — those hallucinate) and preserves the geometric relationships that matter most for distress prediction while eliminating noise dimensions.

The 256-dimension output might seem counterintuitive — why expand? Because the latent space isn't just a compressed version of the input. It's a structured space where each axis corresponds to a well-defined economic feature. Think of it as converting polar coordinates to Cartesian: same information, but expressed in a basis that makes operations (classification, retrieval, comparison) dramatically more efficient.

Phase 4: Cryptographic Footprinting (Layers 29–35)

Every enriched record receives a SHA-256 hash over its 256 dimensional vector. This produces a 32-byte fixed-length fingerprint — a FixedString(32) in ClickHouse terms.

Why hash the vector instead of the raw record? Because the vector is the derived signal — it's what downstream models consume. If two records produce the same vector, they're economically identical regardless of superficial differences in the source data. The hash proves that a specific signal was produced by a specific pipeline at a specific time.

This is also how we verify integrity. Any consumer can recompute the hash and confirm that the data hasn't been tampered with.

Phase 5: Schema Enforcement (Layers 36–37)

The final two layers are Pydantic projection lattices — rigid type boundaries that every record must pass through. If a field can't be coerced to its expected type, the record is discarded. If a footprint doesn't verify, the record is discarded. If a required cross-reference is missing, the record is discarded.

This is the Golod-Shafarevich bypass in practice: by enforcing strict type constraints at the output boundary, we prevent discriminant blowup — the phenomenon where small input noise propagates through a pipeline and magnifies into large output errors. Every record that survives layers 36–37 is guaranteed to satisfy the schema.

Why ClickHouse?

We store all 2B+ footprints in ClickHouse as FixedString(32) with ORDER BY (dag_vector_hash). This gives us O(1) binary lookups — sub-millisecond, regardless of table size.

When you have 853 million market ticks and billions of potential footprints, PostgreSQL's B-tree indexes become a bottleneck. ClickHouse's sparse index architecture, combined with binary-ordered hashes, stays constant-time even as the table grows.

What We Don't Reveal

Some things are competitive advantage. We don't publish the exact projection matrix. We don't reveal which features correspond to which latent-space axes. We don't disclose the training methodology for the manifold transform.

What we do tell you: every record that leaves our pipeline is mathematically guaranteed to satisfy a rigid, published schema, and every record carries a cryptographic proof of provenance. That's more than any other data provider offers.

Why This Matters for Your AI Agent

When your agent queries our MCP endpoint, it doesn't get a CSV with a disclaimer about data quality. It gets a structured, validated, footprinted tensor — a piece of economic intelligence that can be fed directly into a model with zero hallucination risk.

That's the difference between buying data and buying intelligence.

---

Kairos Signal: Verified intelligence for AI agents. 2B+ rows. Millions of signals daily. MCP-native. Explore data products →