100.0%Mean precision@5
100%Hit rate (24/24)
4Golden suites
Last exported: 8/2/2026, 8:49:21 PM UTC
Ranking
Graph + BM25
Lexical, structural expansion, call edges, cluster-aware ranking, and BM25 over symbols and module docs.
Privacy
100% local
No cloud API. Index and eval run entirely on your machine from the open-source engine.
Quality
CI gated
GitHub Actions runs python eval/run_eval.py --all on every push; regressions fail the build.
fixture_auth_python
Mean precision@5: 100.0% · hit rate 100%
| Query | Expected | P@5 |
|---|---|---|
| password hashing salt | app/auth.py | 1.00 |
| get user from database | app/db.py | 1.00 |
| login handler typescript | web.ts | 1.00 |
fixture_go_service
Mean precision@5: 100.0% · hit rate 100%
| Query | Expected | P@5 |
|---|---|---|
| JWT login handler | internal/auth/login.go | 1.00 |
| validate password credentials | internal/auth/login.go | 1.00 |
| list users handler | internal/store/users.go | 1.00 |
| health check endpoint | cmd/server/main.go | 1.00 |
fixture_ts_api
Mean precision@5: 100.0% · hit rate 100%
| Query | Expected | P@5 |
|---|---|---|
| Stripe payment intent checkout | src/payments/stripe.tssrc/handlers/checkout.ts | 1.00 |
| validate cart items total | src/cart/validate.ts | 1.00 |
| webhook signature verification | src/payments/stripe.ts | 1.00 |
| order service place order | src/handlers/checkout.ts | 1.00 |
vinemap_self
Mean precision@5: 100.0% · hit rate 100%
| Query | Expected | P@5 |
|---|---|---|
| MCP server | engine/vinemap/mcp/server.py | 1.00 |
| password hashing | app/auth.py | 1.00 |
| license activation | engine/vinemap/license.py | 1.00 |
| context pack | engine/vinemap/pack/packer.py | 1.00 |
| session memory | engine/vinemap/memory/session.py | 1.00 |
| symbol audit | engine/vinemap/pro/audit.py | 1.00 |
| connect cursor agent | engine/vinemap/connect.pyengine/vinemap/agents/cursor.py | 1.00 |
| file watcher debounced re-index | engine/vinemap/watch.py | 1.00 |
| SQLite incremental content hash store | engine/vinemap/graph/sqlite_store.py | 1.00 |
| tree-sitter go imports parser | engine/vinemap/scanner/parsers/treesitter_parser.py | 1.00 |
| federated shared graph OIDC teams server | teams/server/federated.pyteams/server/auth.py | 1.00 |
| Stripe checkout webhook license | api/main.py | 1.00 |
| monorepo package clusters boundaries | engine/vinemap/scanner/monorepo.py | 1.00 |
Methodology
- Index each fixture or the Vinemap monorepo with tree-sitter parsers.
- Run natural-language queries through the production ranker.
- Score precision@k: fraction of expected files appearing in top-k ranked paths.
- Fail CI if mean precision drops below each suite's threshold.
Reproduce locally
git clone https://github.com/rohit0x62/vinemap.git cd vinemap/engine pip install -e ".[dev,treesitter]" python eval/run_eval.py --all python eval/export_results.py # refresh website JSON