Search Diagram

Search & Indexing Architecture#

The diagram below shows the “Ingest & Index Pluggable” → “Query & Rank Stable API” flow, and the table that follows maps each piece to the matrix-hub modules that implement it (as well as the native client and SDK).

flowchart LR
  subgraph Ingest & Index
    GH[GitHub] --> N[Normalize]
    N --> DB[Postgres]
    N --> Ch[Chunk]
    Ch --> Em[Embed]
    Em --> VI[Vector Index]
    N --> BS[Blob Store]
  end

  subgraph Query & Rank
    U[User Query] --> S[Search API]
    S --> L[Lexical]
    S --> V[Vector]
    DB --> L
    VI --> V
    L --> H[Hybrid Rank]
    V --> H
    H --> R[RAG Optional]
    R --> O[Response]
    H --> O
  end

Implementation Mapping#

Flow Node	Package / Module	File(s)
GitHub Ingestor (manifests + READMEs)	`matrix-hub` (Companion service)	`src/services/ingest.py`
Normalizer (validate + enrich + version)	`matrix-hub`	`src/services/validate.py` + parts of `ingest.py`
Catalog DB (Postgres or SQLite)	`matrix-hub`	`src/db.py`, `src/models.py`
Chunker (name/desc/README/examples)	`matrix-hub` (search internals)	`src/services/search/chunking.py`
Embedder (MiniLM model; workers)	`matrix-hub` (search backends)	`src/services/search/backends/embedder.py`
Vector Index (pgvector → Milvus)	`matrix-hub` (search backends)	`src/services/search/backends/vector.py`
BlobStore (local disk → S3/MinIO)	`matrix-hub` (search backends)	`src/services/search/backends/blobstore.py`
GET /catalog/search (Stable API)	`matrix-hub` (API routes)	`src/routes/catalog.py`
Lexical Search (pg_trgm BM25 → OpenSearch)	`matrix-hub` (search backends)	`src/services/search/backends/lexical.py`
Hybrid Ranker (weights in config)	`matrix-hub` (search logic)	`src/services/search/ranker.py`
RAG (optional) (fetch best chunks + summarize)	`matrix-hub` (search logic)	`src/services/search/rag.py`
Python SDK (client library)	`matrix-python-sdk`	`matrix_sdk/client.py`, `cache.py`, `types.py`
Agent Creator (native CLI client)	`matrix-cli`	`matrix_cli/__main__.py`, `matrix_cli/commands/*.py`
Agent Generator Plugin (reuse‑first)	`agent-generator-matrix`	`agent_generator_matrix/plugin.py`

Note: • All ingestion, indexing, embedding and search‑and‑rank logic lives in matrix-hub under its services/ and routes/ directories. • The Python SDK (matrix-python-sdk) and CLI (matrix-cli aka “agent‑creator”) are the native clients of the platform. • The agent-generator-matrix plugin hooks your planning_agent.py to call Matrix Hub first (reuse‑first), then fall back to code generation only if no suitable agent is found.