obsidian:~/vault$ search --hybrid obsidian

Obsidian MCP Guide: AI Search & Retrieval (2026)

# Wire Obsidian into Claude and other agents over MCP: server setup, hybrid BM25 + vector retrieval, and indexing a 16,894-file vault — with working configs.

author: words: 29572 read_time: 148m updated: 2026-11-11 00:00
$ retriever search --hybrid obsidian

Obsidian is not a note-taking app. It is a local-first, plaintext, graph-structured markdown corpus that becomes an AI context reservoir when you add retrieval infrastructure. 16,894 files. 49,746 chunks. 23ms queries. Zero API calls. One 83 MB SQLite file. This guide covers the complete system: from vault architecture to hybrid retrieval to MCP integration to operational workflows.


Key Takeaways

Context engineering, not note-taking. The value of an Obsidian vault for AI is not the notes themselves but the retrieval layer that makes them queryable. A 16,000-file vault without retrieval is a write-only database. A 200-file vault with hybrid search and MCP integration is an AI knowledge base. The retrieval infrastructure is the product. The notes are the raw material.

Hybrid retrieval beats pure keyword or pure semantic search. BM25 catches exact identifiers and function names. Vector search catches synonyms and conceptual matches across different terminology. Reciprocal Rank Fusion (RRF) merges both without requiring score calibration. Neither method alone covers both failure modes. The paper that introduced RRF reported the same effect for fused rankings: across its TREC experiments, RRF beat the Condorcet Fuse baseline every time and the best individual system in every case but one (TREC 9, where the top run used a human in the loop).1 The hybrid retriever deep dive covers the RRF math, worked examples with real numbers, failure mode analysis, and an interactive fusion calculator.

MCP gives AI tools direct vault access. Model Context Protocol (MCP) servers expose the retriever as a tool that Claude Code, Codex CLI, Cursor, and other AI tools can call directly. The agent queries the vault, receives ranked results with source attribution, and uses the context without loading entire files. The MCP server is a thin wrapper around the retrieval engine.

Local-first means zero API costs and full privacy. The entire stack runs on a single machine: SQLite for storage, Model2Vec for embeddings, FTS5 for keyword search, sqlite-vec for vector KNN. No cloud services, no API calls, no network dependency. Personal notes never leave the machine. The full re-embed of 49,746 chunks would cost about $0.04 at OpenAI API prices, but the real costs are latency, privacy exposure, and the network dependency for a system that should work offline.2

Incremental indexing keeps the system current in under 10 seconds. File modification time comparison detects changes. Only modified files are re-chunked and re-embedded. A full reindex takes about four minutes on Apple M-series hardware. Incremental updates on a typical day’s edits run in under ten seconds. The system stays current without manual intervention.

The architecture scales from 200 to 20,000+ notes. The same three-layer design (intake, retrieval, integration) works at any vault size. Start with BM25-only search over a small vault. Add vector search when keyword collisions become a problem. Add RRF fusion when you need both exact and semantic matches. Each layer is independently useful and independently removable.


How to Use This Guide

This guide covers the complete system. Your starting point depends on where you are:

You are… Start here Then explore
New to Obsidian + AI Why Obsidian for AI Infrastructure, Obsidian MCP Setup Vault Architecture, MCP Server Architecture
Existing vault, want AI access MCP Server Architecture, Claude Code Integration Embedding Models, Full-Text Search
Building a retrieval system The Complete Retrieval Pipeline, Reciprocal Rank Fusion Performance Tuning, Troubleshooting
Team or enterprise context Decision Framework, Knowledge Graph Patterns Developer Workflow Recipes, Migration Guide

Sections marked Contract include implementation details, configuration blocks, and failure modes. Sections marked Narrative focus on concepts, architecture decisions, and the reasoning behind design choices. Sections marked Recipe provide step-by-step workflows.


Why Obsidian for AI Infrastructure

The thesis of this guide: Obsidian vaults are the best substrate for personal AI knowledge bases because they are local-first, plaintext, graph-structured, and the user controls every layer of the stack.

What Obsidian gives AI that alternatives do not

Plaintext markdown files. Every note is a .md file on your filesystem. No proprietary format, no database export, no API required to read the content. Any tool that reads files can read your vault. grep, ripgrep, Python’s pathlib, SQLite FTS5 — they all work directly on the source files. When you build a retrieval system, you are indexing files, not API responses. The index is always consistent with the source because the source is the file system.

Local-first architecture. The vault lives on your machine. No server, no cloud sync dependency, no API rate limits, no terms of service governing how you process your own content. You can embed, index, chunk, and search your notes without any external service. This matters for AI infrastructure because the retrieval pipeline runs as fast as your disk allows, not as fast as an API endpoint responds. It also matters for privacy: personal notes containing credentials, health data, financial information, and private reflections never leave your machine.

Graph structure through wiki-links. Obsidian’s [[wiki-link]] syntax creates a directed graph across notes. A note about OAuth implementation links to notes about token rotation, session management, and API security. The graph structure encodes human-curated relationships between concepts. Vector embeddings capture semantic similarity, but wiki-links capture intentional connections that the author made while thinking about the topic. The graph is a signal that embeddings cannot replicate.

Plugin ecosystem. Obsidian lists 10,000+ community plugins (10,232 entries in the official registry on November 10, 2026, up from about 2,700 in March 2026).25 Dataview queries your vault like a database. Templater generates notes from templates with JavaScript logic. Git integration syncs your vault to a repository. Linter enforces formatting consistency. The Bases core plugin (public since 1.9.10, in early access from 1.9.0) adds database-like views (table, list, cards and kanban, plus map with the Maps plugin) over vault files using frontmatter properties as fields, saved as .base files.13 These plugins add structure to the vault without changing the underlying plaintext format. The retrieval system indexes the output of these plugins, not the plugins themselves.

A large, active community. Obsidian’s users produce templates, workflows, plugins, and documentation. When you encounter a problem with vault organization or plugin configuration, someone has likely documented a solution. The community also produces Obsidian-adjacent tools: MCP servers, indexing scripts, publishing pipelines, and API wrappers.

What a filesystem alone does not give you

A directory of markdown files has the plaintext advantage but lacks three things that Obsidian adds:

  1. Bidirectional links. Obsidian tracks backlinks automatically. When you link from Note A to Note B, Note B shows that Note A references it. The graph panel visualizes connection clusters. This bidirectional awareness is metadata that a raw filesystem does not provide.

  2. Live preview with plugin rendering. Dataview queries, Mermaid diagrams, and callout blocks render in real-time. The writing experience is richer than a text editor while the storage format remains plaintext. You write and organize in a rich environment; the retrieval system indexes the raw markdown.

  3. Community infrastructure. Plugin discovery, theme marketplace, sync service (optional), publish service (optional), and a documentation ecosystem. You can replicate any individual feature with standalone tools, but Obsidian packages them into a coherent workflow.

What Obsidian does not do (and what you build)

Obsidian does not include retrieval infrastructure. It has basic search (full-text, filename, tag) but no embedding pipeline, no vector search, no fusion ranking, no MCP server, no credential filtering, no chunking strategy, and no integration hooks for external AI tools. This guide covers the infrastructure you build on top of Obsidian. The vault is the substrate. The retrieval pipeline, the MCP server, and the integration hooks are the infrastructure.

The architecture described here is markdown-first, not Obsidian-exclusive. If you use Logseq, Foam, Dendron, or a plain directory of markdown files, the retrieval pipeline works identically. The chunker reads .md files. The embedder processes text strings. The indexer writes to SQLite. None of these components depend on Obsidian-specific features. Obsidian’s contribution is the writing and organizational environment that produces the markdown files the retriever indexes.


Obsidian MCP Setup

Model Context Protocol (MCP) is the standard interface that gives Claude Code, Codex CLI, Cursor, and other AI tools direct access to an Obsidian vault. This section gets a vault connected to an AI tool in five minutes. You will install Obsidian, create a vault, install an MCP server, and run your first query. The quick start uses a community MCP server for immediate results. Later sections cover building a custom retrieval pipeline for production use.

Prerequisites

  • macOS, Linux, or Windows
  • Node.js 22+ (obsidian-mcp 2.x requires Node 22 and MCPVault 0.18 requires 22.12; cyanheads’ obsidian-mcp-server requires Node 24)26
  • Obsidian 1.12+ (for CLI integration; 1.15.0 is the current public desktop release, promoted November 2, 2026, with 1.15.1 in Catalyst early access; earlier versions work for MCP-only setups)24
  • Claude Code, Codex CLI, or Cursor installed

Step 1: Create a vault

Download Obsidian from obsidian.md and create a new vault. Choose a location you will remember — the MCP server needs the absolute path.

# Example vault location
~/Documents/knowledge-base/

Add a few notes to give the retriever something to work with. Even 10-20 notes are enough to see results. Each note should be a .md file with a meaningful title and at least one paragraph of content.

Step 2: Install an MCP server

Several community MCP servers provide immediate vault access. The ecosystem has grown significantly through 2025-2026. A notable one is MCPVault (npm @bitbonsai/mcpvault, repo bitbonsai/mcpvault), now at v0.18.3 (as of November 11, 2026; needs Node 22.12 or newer since 0.18.0), a separate project from MarkusPfundstein/mcp-obsidian below, not a rename of it. Three releases matter to this guide: 0.15.0 adds --read-only, which exposes only the eleven non-mutating tools and rejects every vault mutation, the right flag for a retrieval-only agent; 0.16.0 moves to the official MCP TypeScript SDK v2 and serves 2026-07-28 and 2025-era clients from the same process; 0.17.0 adds a repeatable --exclude <subtree> option for persistent search and listing exclusions. Do not stay on 0.17.0 itself: it can write outside the vault through an unresolved symlink path (advisory GHSA-97wj-cq79-2crq, medium severity, listed against 0.17.0; the advisory says earlier versions were not assessed), and 0.17.1 closes that by rejecting unresolved symlinks. Two more security fixes followed: 0.17.2 treats a confirmed system-trash delete path as a literal filename (bracket characters in a note name no longer select neighbouring files), and 0.18.2 disables gray-matter’s executable JavaScript frontmatter engine.124156

Earlier releases, for anyone pinned below 0.15: v0.11.0 (March 2026) added list_all_tags, which scans frontmatter tags and inline hashtags and returns them sorted by frequency (.base and .canvas support is older, from 0.8.2). The July 23, 2026 patches are worth taking: the wiki_link tool (changelog entry 0.12.3, first published to npm in 0.12.4) resolves [[Document Name]], [[Name|Display]], table-escaped [[Name\|Display]], and #fragment forms, returning note content plus the resolved path and any ambiguous alternatives (the retrieval primitive that lets an agent follow a vault’s own link graph instead of re-searching for it), and the same entry excludes .trash/ from every tool through the default path filter; v0.12.4 also extends wiki_link to path-qualified links like [[folder/Note]]; v0.12.2 stops patch_note from corrupting insertions that contain $ replacement patterns and normalizes paths that accidentally carry the vault prefix. Two medium-severity advisories (GHSA-9c83-rr99-vfwj and GHSA-j99q-93c9-h869) were disclosed against its path-filter restricted-directory deny-list; both were fixed well before the 0.12 line, in 0.11.4 and 0.11.5, so every release from 0.12.0 on is clear of them. A third medium-severity advisory, GHSA-97wj-cq79-2crq, is the 0.17.0 symlink write described above; it is listed against 0.17.0 and fixed in 0.17.1.1256

The Obsidian CLI as the preferred bridge: Obsidian 1.12.0 introduced the first-class CLI, and the public 1.12.7 installer (March 23, 2026) bundled the standalone binary, TUI and socket-file improvements that made terminal workflows easier to install and run.14 The 1.13 line reached the public channel as 1.13.4 on July 30, 2026 (a settings, image and URI-security release); 1.14 followed on October 5 (Bases kanban view, opening Markdown files from outside the vault) and 1.15 on November 2, 2026, which added heading-targeted append and prepend with line-break control to the CLI and URI surface.2324

The CLI needs no community plugin, API key or local HTTPS port; the Local REST API route does (it is the plugin behind mcp-obsidian). That plugin is now named Local REST API with MCP and, since 4.0.0 (May 15, 2026), serves MCP itself over Streamable HTTP at /mcp/, so a REST-API setup no longer needs a separate bridge process.46 The MarkusPfundstein/mcp-obsidian repo is still maintained (commits through August 2026: a pin of the MCP Python SDK below 2.0 on August 20 and Docker packaging on August 31; the May 2026 work added search_by_tag and get_frontmatter), though it ships no tagged releases (install from a pinned commit). Two open issues report that on plugin 5.x its patch_content tool fails with error 40084: the server does not send the Markdown-Patch-Version header that plugin 5.0.0 began to require, and the fixes are unmerged. It remains Local-REST-API-based; for a new setup, prefer the CLI bridge or one of the file-based servers listed below, which have fewer moving parts.18 See the “Obsidian CLI for AI Workflows” section later in this guide for the recommended setup.

On the Local REST API route, run plugin 5.4.0 or later (5.5.2 is current, and later than every fix below). Three high-severity advisories cover earlier builds, and each is open to any client that holds the plugin’s API key, which is what a bridge server or agent is given. GHSA-62gx-5q78-wrvx lets an authenticated client read, write or delete arbitrary files on the host through the /vault/{path} endpoints (affected: 4.1.2 and earlier; patched: 4.1.3). GHSA-j9qv-qgpv-345g lets an authenticated MCP client write files outside the vault through the vault_write tool (listed against 5.1.0, with no patched version in the advisory; the 5.2.0 release notes record the fix). GHSA-66m9-r757-qvq7 lets an authenticated caller write a plugin into .obsidian/plugins/, enable it and run code as the user (listed against 5.3.1, again with no patched version; the 5.4.0 release notes record the fix, which closes the configuration folder to the API by default, so leave “Allow access to the configuration directory” off). Releases from 5.1.0 on need Obsidian 1.13.1 or newer, so update the app first if it is older.54

Server Author Transport Requires Plugin Key Feature
obsidian-mcp (npm obsidian-mcp) StevenStavrakis STDIO No Lightweight, file-based
mcp-obsidian MarkusPfundstein STDIO Local REST API with MCP (run 5.4.0 or later)54 Full vault CRUD via REST, plus search_by_tag/get_frontmatter; actively maintained (commits through August 2026); no tagged releases, pin a commit; patch_content is reported failing on plugin 5.x (open issues)18
obsidian-mcp-tools jacksteamdev STDIO Yes (its own plugin, no longer in the community registry) Semantic search + Templater. Archived May 2026: read-only reference, not for new setups27
obsidian-claude-code-mcp iansinnott WebSocket (plus legacy HTTP+SSE) Yes (its own plugin, not in the community registry; manual install) Auto-discovery for Claude Code. No commits since June 2025: not recommended for new setups27
obsidian-mcp-server (npm obsidian-mcp-server) cyanheads STDIO or Streamable HTTP Local REST API with MCP (run 5.4.0 or later)54 Tags, frontmatter management; configured via OBSIDIAN_API_KEY/OBSIDIAN_BASE_URL, not CLI flags; Node 24+26
obsidian-hybrid-search (npm obsidian-hybrid-search) flowing-abyss STDIO or Streamable HTTP No (a companion Hybrid Search plugin is optional) BM25 + fuzzy title + semantic search fused with RRF; ships a CLI (ohs) and an MCP server (obsidian-hybrid-search-mcp); Node 22.22+45

For the quick start, the simplest option is a file-based server that reads .md files directly. Watch the npm name collision: the file-based server is npm obsidian-mcp (StevenStavrakis); npm obsidian-mcp-server is cyanheads’ REST-API-backed server, which needs the Local REST API with MCP plugin (5.4.0 or later, per the note above) and an API key, a common mixup that leaves readers with a server that cannot start. The configs below install and run the file-based server in one step: npx -y obsidian-mcp@2 fetches the package into the npm cache on first use, pins the 2.x line (Node 22 or newer) and picks up compatible updates.28 If you prefer a global binary:

npm install -g obsidian-mcp@2

Step 3: Configure your AI tool

Claude Code — register the server with claude mcp add (Claude Code stores MCP servers in ~/.claude.json for user scope or a project’s .mcp.json — not ~/.claude/settings.json, which silently ignores an mcpServers block):

# User scope (all your projects)
claude mcp add obsidian -s user -- npx -y obsidian-mcp@2 serve --vault notes=/absolute/path/to/your/vault

# Or project scope, shared with the repo (writes .mcp.json)
claude mcp add obsidian -s project -- npx -y obsidian-mcp@2 serve --vault notes=/absolute/path/to/your/vault

Codex CLI — add to ~/.codex/config.toml:

[mcp_servers.obsidian]
command = "npx"
args = ["-y", "obsidian-mcp@2", "serve", "--vault", "notes=/absolute/path/to/your/vault"]

Cursor — add to .cursor/mcp.json:

{
  "mcpServers": {
    "obsidian": {
      "command": "npx",
      "args": ["-y", "obsidian-mcp@2", "serve", "--vault", "notes=/absolute/path/to/your/vault"]
    }
  }
}

Step 4: Run your first query

Open your AI tool and ask a question that your vault notes can answer:

Search my Obsidian vault for notes about [topic you wrote about]

The AI tool calls the MCP server, which searches your vault and returns matching content. You should see results with file paths and relevant excerpts.

What Claude can do once connected

Exact tool names vary by server, but the core capability surface is consistent across implementations:

Capability Typical tool What the agent does with it
Search the vault obsidian_search / search Finds notes matching a query and returns ranked excerpts with file paths and source attribution
Read a full note obsidian_read_note / read_note Pulls complete note content when a search excerpt is not enough
List and browse obsidian_list_notes / list_notes Explores notes by folder, tag, or date range when there is no specific query
Get formatted context obsidian_get_context Returns a topic-shaped context block sized to a token budget, ready for injection into the conversation

In practice: Claude answers questions from your notes with source attribution, pulls prior decisions and reference material into coding sessions, and explores vault structure without loading entire files into context. Some community servers also expose write operations (create, append, tag and frontmatter management); the custom server built later in this guide is deliberately read-only, with note creation handled by hooks instead.

Deep dives: MCP Server Architecture for tool and permission design, Claude Code Integration for hooks and the bridge pattern, Codex CLI Integration and Cursor and Other Tools for other agents.

What you just built

You connected a local knowledge base to an AI tool through a standard protocol. The MCP server reads your vault files, performs basic search, and returns results. This is the minimal viable version. Note that obsidian-mcp 2.x also exposes write tools (obsidian_create_note, obsidian_edit_note, obsidian_move_note, obsidian_delete_note), so back up the vault or keep your client’s permission prompts on.28

What this quick start does not give you:

  • Hybrid retrieval (BM25 + vector search + RRF fusion)
  • Embedding-based semantic search
  • Credential filtering
  • Incremental indexing
  • Hook-based automatic context injection

The rest of this guide covers building each of these capabilities. The quick start proves the concept. The full pipeline delivers production-quality retrieval.


Obsidian CLI for AI Workflows

Obsidian 1.12 (February 2026) introduced a built-in command line interface that opens a new integration surface for AI workflows. It remains current through 1.15.0 (public since November 2, 2026), which added heading-targeted append and prepend (the heading= parameter) to the CLI, and heading and spacing parameters to the matching obsidian:// URI actions. The CLI’s inline flag (append without a line break) is older than 1.15. Version 1.15.1, in Catalyst early access, adds top-level await to eval.142429 The CLI acts as a remote control for the Obsidian GUI: Obsidian must be running (or will launch automatically on first command). Enable it in Settings > General > Command line interface. Every command takes named parameter=value arguments plus bare-word flags, never positional arguments: obsidian create name=Note content="Hello world" open. Quote a value that contains spaces, and write \n inside a value for a newline.2957

Why the CLI matters for AI infrastructure

The CLI provides programmatic access to Obsidian-native operations that previously required the GUI or plugin APIs. For AI workflows, the key capabilities are:

  • Search from scripts and hooks. obsidian search query="..." returns matching file paths and obsidian search:context query="..." returns grep-style path:line: text matches, from any shell script, hook, or automation pipeline. Add format=json for machine-readable output, path=<folder> to scope a folder and limit=<n> to cap the file count. The search:context output is the form to feed into AI prompts.
  • Daily notes automation. obsidian daily opens today’s daily note (the Daily notes plugin creates it if it does not exist), and obsidian daily:append content="..." appends to it without knowing its path; add heading="..." to target a section and inline to skip the newline. daily:prepend and daily:read complete the set, so a hook can append AI-generated summaries to the daily note with one command.
  • Template-based note creation. obsidian templates lists the core Templates plugin’s templates, obsidian create path="03-resources/x.md" template="Resource" creates a note from one, and obsidian template:insert name="..." inserts one into the active file, so an AI agent can create structured vault entries without writing frontmatter by hand.
  • Property management. obsidian property:set name=status value=archived file="..." and obsidian property:read name=status file="..." write and read frontmatter properties without parsing YAML (type= sets the property type; path= targets by exact vault path instead of name). obsidian properties counts lists the vault’s property names with occurrence counts (without the counts flag it lists the names only).
  • Plugin control. obsidian plugin:enable id=dataview, obsidian plugin:disable id=dataview, obsidian plugins and obsidian plugins:enabled manage plugins programmatically, useful for toggling indexing plugins during batch operations.
  • Task management. obsidian tasks todo lists open tasks (done, daily, file= and status= filter further) and obsidian task ref="Recipe.md:8" done completes one (todo and toggle are the other forms). The CLI reads and updates tasks; it does not create them, so a new task goes in through append.

CLI vs MCP for AI access

The CLI and MCP servers serve different roles and are complementary, not competing:

Aspect Obsidian CLI MCP Server
Caller Shell scripts, hooks, cron jobs AI agents (Claude Code, Codex, Cursor)
Protocol POSIX process (stdin/stdout/stderr) MCP (JSON-RPC over STDIO or HTTP)
Strength Obsidian-native operations (templates, plugins, properties) Custom retrieval (embeddings, BM25, RRF fusion)
Limitation No vector search, no embedding pipeline No access to Obsidian-internal operations
Best for Automation scripts, intake pipelines, hook actions Real-time AI agent queries during sessions

Recommendation: Use the CLI for intake automation (creating notes, managing properties, running Obsidian-native search) and MCP for retrieval (hybrid search with embeddings). A UserPromptSubmit hook can call obsidian search:context as a fast pre-check before the heavier hybrid retrieval runs (tool-scoped hook events do not add plain stdout to the conversation; to inject from PostToolUse, print JSON with hookSpecificOutput.additionalContext).

Example: CLI-powered intake hook

#!/bin/bash
# Hook: append the start of a Bash tool's output to today's daily note via the CLI.
# Register on PostToolUse with "matcher": "Bash". The payload arrives as stdin JSON,
# and for Bash the tool_response is an object whose text is in .stdout.
OUTPUT=$(jq -r '.tool_response.stdout // ""' | head -c 2000)
[ -z "$OUTPUT" ] && exit 0
obsidian daily:append content="## Tool output\n${OUTPUT}"

daily:append resolves today’s note from the Daily notes settings, so the script never builds a Daily Notes/${DATE}.md path by hand; the CLI turns the \n inside the value into a newline. To write to an arbitrary note instead, use obsidian append path="06-daily/${DATE}.md" content="...", which also accepts heading= to land the text under a section.57 The script copies output; it does not summarize it. To append a model-written summary, generate the text first and pass it as content=.

Without the desktop app. The CLI drives a running Obsidian. Obsidian Headless (open beta; npm install -g obsidian-headless, Node.js 22 or later) is a separate command-line client for Obsidian Sync that keeps a vault synced on a server or in a container with no desktop app. Obsidian lists giving “agentic tools access to a vault without access to your full computer” among its uses.47


Obsidian Agent Plugins

A growing category of Obsidian plugins embeds AI coding agents directly in the vault UI, providing an alternative to external MCP server configuration. These plugins run the AI agent inside Obsidian’s sidebar rather than connecting from an external tool.

Claudian

Claudian embeds coding agents in the vault: Claude Code, Codex, Grok Build, Antigravity CLI, OpenCode, Pi and DeepSeek Harness (Obsidian 1.13 or newer, desktop only). The vault directory becomes the agent’s working directory, giving it full agentic capabilities: file read/write, search, bash commands, and multi-step workflows.15

Key features for AI infrastructure:

  • Vault-aware prompts. @ mentions vault files, folders and other Claudian sessions; inline edit works from a selection or the cursor with a word-level diff preview.
  • Slash commands and skills. / and $ call reusable prompt templates and Skills from user- and vault-level scopes.
  • Projects and cross-session messaging. Sessions grouped into a project share instructions and a memory file the agents keep current, and agents can find, read, message and steer other Claudian sessions.
  • Permission modes. Each agent’s own modes, surfaced in the toolbar; for Claude Code that is Auto, Manual, acceptEdits and YOLO (which maps to bypassPermissions).
  • MCP servers. External tools connect through each agent’s own CLI-managed MCP configuration, so the vault server built later in this guide works inside Claudian too.

Agent Client

Agent Client brings Claude Code, Codex, Gemini CLI, Hermes Agent, OpenCode, Pi, Grok Build, Kiro and Mistral Vibe (nine presets, plus any ACP-compatible agent as a custom entry) into a unified Obsidian sidebar via the Agent Client Protocol (ACP).16

Key features:

  • Multi-agent switching. Chat with any of the nine preset agents, or a custom ACP agent, from the same panel, switching between agents as needed.
  • Note mentions. Use @notename to include note contents in prompts, similar to Claudian but agent-agnostic.
  • Shell execution. Execute terminal commands inline in the chat — build scripts, git commands, or any terminal operation without leaving the conversation.
  • Action approval. Every permission request surfaces for approval or rejection, from the banner or with a hotkey; auto-allow is off by default.

When to use agent plugins vs external MCP

Scenario Agent plugin External MCP
Writing and editing vault notes with AI assistance Better — agent sees the editor context Works but no editor awareness
Code development across multiple repos Possible: Agent Client can open a chat in any directory; Claudian stays vault-scoped Better: project-scoped with full filesystem
Retrieval from a large indexed corpus Basic search only Full hybrid retrieval pipeline
Quick vault Q&A during note-taking sessions Ideal — no context switching Requires switching to terminal

Recommendation: Use agent plugins for vault-centric workflows (writing, organizing, summarizing notes). Use external MCP servers for development workflows where the AI agent needs the full retrieval pipeline and access to codebases outside the vault. The two approaches can coexist — run Claudian inside Obsidian for note work and Claude Code with MCP externally for development.


Decision Framework: Obsidian vs Alternatives

Not every use case needs Obsidian. This section maps when Obsidian is the right substrate, when it is overkill, and when something else fits better.

Decision Tree

START: What is your primary content type?
│
├─ Structured data (tables, records, schemas)
│  → Use a database. SQLite, PostgreSQL, or a spreadsheet.
│  → Obsidian is for prose, not tabular data.
│
├─ Ephemeral context (current project, temporary notes)
│  → Use CLAUDE.md / AGENTS.md in the project repo.
│  → These travel with the code and reset per project.
│
├─ Team wiki (shared documentation, onboarding)
│  → Evaluate Notion, Confluence, or a shared git repo.
│  → Obsidian vaults are personal-first. Team sync is possible
│    but not native.
│
└─ Growing personal knowledge corpus
   │
   ├─ < 50 notes
   │  → A folder of markdown files + grep is sufficient.
   │  → Obsidian adds value mainly through the link graph,
   │    which needs density to be useful.
   │
   ├─ 50 - 500 notes
   │  → Obsidian adds value. Wiki-links create a navigable graph.
   │  → BM25-only search (FTS5) is sufficient at this scale.
   │  → Skip vector search and RRF until keyword collisions appear.
   │
   ├─ 500 - 5,000 notes
   │  → Full hybrid retrieval becomes valuable. Keyword collisions
   │    increase. Semantic search catches queries that BM25 misses.
   │  → Add vector search + RRF fusion at this scale.
   │
   └─ 5,000+ notes
      → Full pipeline is essential. BM25-only returns too much noise.
      → Credential filtering becomes critical (more notes = more
        accidentally pasted secrets).
      → Incremental indexing matters (full reindex takes minutes).
      → MCP integration pays dividends on every AI interaction.

Comparison Matrix

Criterion Obsidian Notion Apple Notes Plain Filesystem CLAUDE.md
Local-first Yes No (cloud) Partial (iCloud) Yes Yes
Plaintext Yes (markdown) No (blocks) No (proprietary) Yes Yes
Graph structure Yes (wiki-links) Partial (mentions) No No No
AI indexable Direct file access API required Export required Direct file access Already in context
Plugin ecosystem 10,000+ plugins Integrations None N/A N/A
Offline capable Full Downloaded pages (offline mode)48 Partial Full Full
Scales to 10K+ notes Yes Yes (with API) Degrades Yes No (single file)
Cost Free (core) Free plan; Plus $10 per member/mo48 Free Free Free

When Obsidian is overkill

  • Single-project context. If the AI only needs context about the current codebase, put it in CLAUDE.md, AGENTS.md, or project-level documentation. These files travel with the repo and are automatically loaded.
  • Structured data. If the content is tables, records, or schemas, use a database. Obsidian notes are prose-first. Dataview can query frontmatter fields, but a real database handles structured queries better.
  • Temporary research. If the notes will be discarded after the project ends, a scratch directory with markdown files is simpler. Do not build retrieval infrastructure for ephemeral content.

When Obsidian is the right choice

  • Accumulating knowledge over months or years. The value compounds as the corpus grows. A 200-note vault queried daily for six months provides more value than a 5,000-note vault queried once.
  • Multiple domains in one corpus. A vault containing notes on programming, architecture, security, design, and personal projects benefits from cross-domain retrieval that a project-specific CLAUDE.md cannot provide.
  • Privacy-sensitive content. Local-first means the retrieval pipeline never sends content to external services. The vault contains whatever you put in it, including content you would not upload to a cloud service.

Mental Model: Three Layers

The system has three layers that operate independently but compound when combined. Each layer has a different concern and a different failure mode.

┌─────────────────────────────────────────────────────┐
│                 INTEGRATION LAYER                     │
│  MCP servers, hooks, skills, context injection        │
│  Concern: delivering context to AI tools              │
│  Failure: wrong context, too much context, stale      │
└──────────────────────┬──────────────────────────────┘
                       │ query + ranked results
┌──────────────────────┴──────────────────────────────┐
│                  RETRIEVAL LAYER                      │
│  BM25, vector KNN, RRF fusion, token budget           │
│  Concern: finding the right content for any query     │
│  Failure: wrong ranking, missed results, slow queries │
└──────────────────────┬──────────────────────────────┘
                       │ chunked, embedded, indexed
┌──────────────────────┴──────────────────────────────┐
│                   INTAKE LAYER                        │
│  Note creation, signal triage, vault organization     │
│  Concern: what enters the vault and how it's stored   │
│  Failure: noise, duplicates, missing structure        │
└─────────────────────────────────────────────────────┘

Intake determines what enters the vault. Without curation, the vault accumulates noise: screenshots of tweets, copy-pasted articles with no annotation, half-finished thoughts with no context. The intake layer is responsible for quality control at the point of entry. A scoring pipeline, tagging convention, or manual review process — any mechanism that ensures the vault contains content worth retrieving.

Retrieval makes the vault queryable. This is the engine: chunking notes into search units, embedding chunks into vector space, indexing for keyword and semantic search, fusing results with RRF. The retrieval layer transforms a directory of files into a queryable knowledge base. Without this layer, the vault is navigable through manual browsing and basic search but not programmatically accessible to AI tools.

Integration connects the retrieval layer to AI tools. An MCP server exposes retrieval as a callable tool. Hooks inject context automatically. Skills capture new knowledge back into the vault. The integration layer is the interface between the knowledge base and the AI agents that consume it.

The layers are decoupled by design. The intake scoring pipeline knows nothing about embeddings. The retriever knows nothing about signal routing rules. The MCP server knows nothing about how notes were created. This decoupling means you can improve any layer independently. Replace the embedding model without changing the intake pipeline. Add a new MCP capability without modifying the retriever. Change the signal scoring heuristics without touching the index.


Vault Architecture for AI Consumption

A vault optimized for AI retrieval follows different conventions than a vault optimized for personal browsing. This section covers folder structure, note schema, frontmatter conventions, and the specific patterns that improve retrieval quality.

Folder Structure

Use numbered prefixes for top-level folders to create a predictable organizational hierarchy. The numbers do not imply priority — they group related domains and make the structure scannable.

vault/
├── 00-inbox/              # Unsorted captures, pending triage
├── 01-projects/           # Active project notes
├── 02-areas/              # Ongoing areas of responsibility
├── 03-resources/          # Reference material by topic
│   ├── programming/
│   ├── security/
│   ├── ai-engineering/
│   ├── design/
│   └── devops/
├── 04-archive/            # Completed projects, old references
├── 05-signals/            # Scored signal intake
│   ├── ai-tooling/
│   ├── security/
│   ├── systems/
│   └── ...12 domain folders
├── 06-daily/              # Daily notes (if used)
├── 07-templates/          # Note templates (excluded from index)
├── 08-attachments/        # Images, PDFs (excluded from index)
├── .obsidian/             # Obsidian config (excluded from index)
└── .indexignore            # Paths to exclude from retrieval index

Folders that should be indexed: Everything containing markdown prose — projects, areas, resources, signals, daily notes.

Folders that should be excluded from indexing: Templates (they contain placeholder variables, not content), attachments (binary files), Obsidian configuration, and any folder containing sensitive content you do not want in the retrieval index.

The .indexignore file

Create a .indexignore file at the vault root to explicitly exclude paths from the retrieval index. The syntax matches .gitignore:

# Obsidian internal
.obsidian/

# Templates contain placeholders, not content
07-templates/

# Binary attachments
08-attachments/

# Personal health/medical notes
02-areas/health/

# Financial records
02-areas/finance/personal/

# Career documents (resumes, salary data)
02-areas/career/private/

The indexer reads this file before scanning and skips matching paths entirely. Files in excluded paths are never chunked, never embedded, and never appear in search results.

Note Schema

Every note should have YAML frontmatter. The retriever uses frontmatter fields for filtering and context enrichment:

---
title: "OAuth Token Rotation Patterns"
type: note           # note | signal | project | moc | daily
domain: security     # primary domain for routing
tags:
  - authentication
  - oauth
  - token-management
created: 2026-01-15
updated: 2026-02-28
source: ""           # URL if captured from external source
status: active       # active | archived | draft
---

Required fields for retrieval:

  • title — Used in search result display and heading context for BM25
  • type — Enables type-filtered queries (“show me only MOCs” or “only signals”)
  • tags — Indexed in FTS5 heading context with 0.3 weight, providing keyword matches even when the body uses different terminology

Optional but valuable fields:

  • domain — Enables domain-scoped queries (“search security notes only”)
  • source — Attribution for captured content; the retriever can include source URLs in results
  • status — Allows excluding archived or draft notes from active search

Chunking Conventions

The retriever chunks at H2 (##) heading boundaries. This means your note structure directly affects retrieval granularity:

Good for retrieval:

## Token Rotation Strategy

The rotation interval depends on the threat model...

## Implementation with refresh_token

The OAuth 2.0 refresh token flow requires...

## Error Handling: Expired Tokens

When a token expires mid-request...

Three H2 sections produce three independently searchable chunks. Each chunk has enough context for the embedding to capture its meaning. A query about “expired token handling” matches the third chunk specifically.

Poor for retrieval:

# OAuth Notes

Token rotation depends on threat model. The OAuth 2.0 refresh
token flow requires storing the refresh token securely. When a
token expires mid-request, the client should retry after refresh.
The rotation interval is typically 15-30 minutes for access tokens
and 7-30 days for refresh tokens...

One long section with no H2 headings produces one large chunk. The embedding averages across all topics in the section. A query about any subtopic matches the entire note equally.

Rule of thumb: If a section covers more than one concept, split it into H2 subsections. The chunker handles the rest.

What Not to Put in Notes

Content that degrades retrieval quality:

  • Raw copy-pastes of entire articles without annotation. The retriever indexes the original article’s keywords, diluting your vault with content you did not write. Add a summary, extract key points, or link to the source URL instead.
  • Screenshots without text description. The retriever indexes markdown text. An image without alt text or surrounding description is invisible to both BM25 and vector search.
  • Credential strings. API keys, tokens, passwords, connection strings. Even with credential filtering, the safest approach is to never paste secrets into notes. Reference them by name (“the Cloudflare API token in ~/.env”) instead.
  • Auto-generated content without curation. If a tool generates a note (meeting transcript, Readwise highlights, RSS import), review and annotate it before it enters the permanent vault. Uncurated auto-imports add volume without adding retrievable value.

Plugin Ecosystem for AI Workflows

Obsidian plugins that improve vault quality for AI retrieval fall into three categories: structural (enforce consistency), querying (expose metadata), and sync (keep the vault current).

Essential Plugins

Dataview. Queries your vault like a database using frontmatter fields. Create dynamic indexes: “all notes tagged security updated in the last 30 days” or “all project notes with status active.” Dataview does not directly help retrieval, but it helps you identify gaps in your vault’s coverage and find notes that need updating.

TABLE type, domain, updated
FROM "03-resources"
WHERE status = "active"
SORT updated DESC
LIMIT 20

Templater. Creates notes from templates with dynamic fields. Ensure every new note starts with correct frontmatter by using a template that pre-fills created, type, and domain fields. Consistent frontmatter improves retrieval filtering.

---
title: "<% tp.file.cursor() %>"
type: note
domain: <% tp.system.suggester(["programming", "security", "ai-engineering", "design", "devops"], ["programming", "security", "ai-engineering", "design", "devops"]) %>
tags: []
created: <% tp.date.now("YYYY-MM-DD") %>
updated: <% tp.date.now("YYYY-MM-DD") %>
source: ""
status: active
---

## Key Points

## Details

## References

Keep --- as the first line of the template. Obsidian reads properties only when the block starts at the very beginning of the file, and Templater leaves the line break after a command tag in place. A tag on its own line above the block (a <%* ... %> script or comment, for example) therefore pushes the frontmatter down to line 2, and the note is created with no properties. If a template needs such a tag first, close it with -%>, which trims the line break that follows.43

Linter. Enforces formatting rules across the vault. Consistent heading hierarchy (H1 for title, H2 for sections, H3 for subsections) ensures the chunker produces predictable results. Linter rules that matter for retrieval:

  • Heading increment: enforce sequential heading levels (no jumping from H1 to H3)
  • YAML title: match the filename
  • Trailing spaces: remove (avoids FTS5 tokenization artifacts)
  • Consecutive blank lines: limit to 1 (cleaner chunks)

Git integration. Version control for your vault. Track changes over time, sync between machines, and recover from accidental deletions.

Plugins That Help Indexing

Smart Connections. An Obsidian plugin that provides AI-powered semantic search within Obsidian itself. Smart Connections v4 creates local embeddings by default: once your vault is indexed, semantic connections and lookup work entirely offline with no API calls.9 v4.5.0 (May 5, 2026) makes footer connections part of Smart Connections Core, so every install can show related-note connections in the footer without opening a side panel. Before that, v4.3.0 (April 1, 2026) added a graph view for connection lists, a setting for where the Connections view opens, and better recovery when embedding work is interrupted. It shipped with the “Substrate Update,” an update to Smart Environment, the shared setup that Smart Connections, Smart Context, Smart Chat, Smart Templates and companion plugins load once and reuse.19 v4.7 (August 2026; current release 4.7.2) moves onto Smart Environment v3: a larger catalog of built-in local embedding models including lighter and multilingual ones, model switching without restarting Obsidian or deleting the embeddings you may want to return to, and a target menu that re-anchors Connections on a recent note, a single block, or a vault file dropped onto the view.19

While the retrieval system in this guide is external to Obsidian (runs as a Python pipeline), Smart Connections is useful for exploring semantic relationships while writing. The two systems index the same content but serve different use cases: Smart Connections for in-editor discovery, the external retriever for AI tool integration via MCP.

Community plugins built for agent workflows. Three that are in the official registry under exactly these names (search the plugin browser for the name shown):45

Plugin Repository What it does
Hybrid Search flowing-abyss/obsidian-hybrid-search-plugin In-app search over the obsidian-hybrid-search engine (npm obsidian-hybrid-search): BM25 + fuzzy title + semantic search fused with RRF. The engine also ships a CLI and an MCP server
Drift ryanbbrown/obsidian-drift Detects file changes made outside Obsidian (an agent’s edits, for example) and shows side-by-side diffs with per-chunk accept or reject
EngramQuest bahfahh/engram-quest Turns notes into quest maps, memory maps and flashcard review decks through AI Skills for Claude Code, Gemini CLI and Cursor

The closest in philosophy to this guide’s external retriever is Hybrid Search: its engine keeps the index in SQLite with FTS5 and sqlite-vec and fuses the rankings with RRF, so it is the packaged version of the pipeline built below. The registry labels all three as not manually reviewed by Obsidian staff; read the code, or try them on a copy of the vault, before giving one write access.45

Dataview note: Dataview (the long-standing Obsidian query plugin) last released 0.5.70 in April 2025 and has been effectively dormant since. For new work, Obsidian’s built-in Bases feature (public since 1.9.10) is the implicit successor and the recommended path.

Metadata Menu. Provides structured frontmatter editing with autocomplete for field values. Reduces typos in type, domain, and tags fields. Consistent metadata improves retrieval filtering accuracy.

Plugins That Hurt Indexing

Excalidraw. Stores drawings as JSON embedded in markdown files. The JSON is syntactically valid markdown but produces garbage when chunked and embedded. Exclude Excalidraw files from the index via .indexignore or filter by file extension.

Kanban. Stores board state as specially-formatted markdown. The format is designed for Kanban rendering, not for prose retrieval. The chunker produces fragments of card titles and metadata that do not embed well. Exclude Kanban boards from the index.

Calendar. Creates daily notes with minimal content (often just a date header). Empty or near-empty notes produce low-quality chunks. If you use daily notes, write substantive content in them or exclude the daily notes folder from the index.

Plugin Configuration That Matters

File recovery → Enabled. Protects against accidental note deletion. Not directly related to retrieval but critical for a knowledge base you depend on.

Strict line breaks → Either. This is a reading-view rendering setting: on, single line breaks are ignored, as the Markdown spec says; off, they show. It does not change the files on disk, so it has no effect on chunking.44

Default location for new notes → In the folder specified below. Route new files to 00-inbox/ so uncategorized notes do not pollute domain folders. The inbox is a staging area; files move to domain folders after triage.

New link format → Shortest path when possible. Shorter link targets are easier for the retriever to resolve when indexing link structure.


Embedding Models: Choosing and Configuring

The embedding model converts text chunks into numerical vectors for semantic search. The model choice determines retrieval quality, index size, embedding speed, and runtime dependencies. This section explains why Model2Vec’s potion-base-8M is the default choice and when to choose alternatives.

Why Model2Vec potion-base-8M

Model: minishlab/potion-base-8M Parameters: 7.6 million Dimensions: 256 Size: ~30 MB Dependencies: model2vec (numpy, tokenizers, safetensors, huggingface-hub and a few small helpers; no PyTorch for inference) Inference: CPU-only, static word embeddings (no attention layers)

Model2Vec distills a sentence transformer’s knowledge into static token embeddings. Instead of running attention layers over the input (as BERT, MiniLM, and other transformer models do), Model2Vec computes one fixed, frequency-weighted vector per token ahead of time and produces a sentence vector by averaging them. The practical consequence: with no attention layers to run at query time, embedding is up to 500x faster on CPU than the transformer the model was distilled from (and the model file up to 50x smaller).3

On the current Model2Vec results page, potion-base-8M reaches about 92% of all-MiniLM-L6-v2’s all-task score (51.32 vs 55.80) while staying orders of magnitude faster.4 The remaining quality gap is the trade-off for the speed and simplicity advantages. For short markdown chunks (at most 2,000 characters in this pipeline, and 166 on average in the vault measured here2), the quality difference is less pronounced than on longer documents because both models converge on similar representations for short, focused text.

Configuration

# embedder.py
import glob
import os
import sys

DEFAULT_MODEL = "minishlab/potion-base-8M"
EMBEDDING_DIM = 256
VENV = os.path.expanduser("~/.claude/venvs/memory")

def _activate_venv():
    """Add the isolated venv's site-packages to sys.path."""
    for site_packages in glob.glob(os.path.join(VENV, "lib", "python*", "site-packages")):
        if site_packages not in sys.path:
            sys.path.insert(0, site_packages)

class Model2VecEmbedder:
    def __init__(self, model_name=DEFAULT_MODEL):
        self._model_name = model_name
        self._model = None

    def _ensure_model(self):
        if self._model is not None:
            return
        _activate_venv()  # Add isolated venv to sys.path
        from model2vec import StaticModel
        self._model = StaticModel.from_pretrained(self._model_name)

    def embed_batch(self, texts):
        self._ensure_model()
        vecs = self._model.encode(texts)
        return [v.tolist() for v in vecs]

Lazy loading. The model loads on first use, not at import time. Importing the embedder module costs nothing when the retriever operates in BM25-only fallback mode (e.g., when the embedding venv is not installed).

Isolated virtual environment. The model runs in a dedicated venv (e.g., ~/.claude/venvs/memory/) to avoid dependency conflicts with the rest of the toolchain. The _activate_venv() function adds the venv’s site-packages to sys.path at runtime.

# Create isolated venv
python3 -m venv ~/.claude/venvs/memory
~/.claude/venvs/memory/bin/pip install model2vec sqlite-vec

Batch processing. The embedder processes texts in batches of 64 to amortize Model2Vec’s overhead. The indexer feeds chunks to embed_batch() rather than embedding one chunk at a time.

When to Choose Alternatives

Model Dim Size Speed Quality score Best for
potion-base-8M 256 30 MB 500x 51.32 Default: local, fast, no GPU
potion-base-32M 512 129 MB 400x 52.83 Higher quality, still static
potion-retrieval-32M 512 129 MB 400x 35.06 (retrieval) Retrieval-optimized static
potion-multilingual-128M 256 ~500 MB 300x — Multilingual vaults (101 languages)
all-MiniLM-L6-v2 384 91 MB 1x 55.80 Higher quality, still local
nomic-embed-text-v1.5 768 547 MB 0.5x 62.28 Best local quality
text-embedding-3-small 1536 API N/A 62.3 API-based, highest quality

The quality scores come from two scales. The potion and all-MiniLM-L6-v2 figures are the “Avg (All)” column of Model2Vec’s own results table (the potion-retrieval-32M figure is from the same page’s separate retrieval table). The nomic and OpenAI figures are MTEB averages published by those vendors. Compare within each group, not across them. Sizes are the float32 weight files.4649

Choose potion-base-32M when you want better quality than potion-base-8M without leaving the static embedding family. It uses a larger vocabulary distilled from baai/bge-base-en-v1.5, achieving a 52.83 all-task score (about 3% higher than potion-base-8M) while staying PyTorch-free at inference.6 Its output is 512-dimensional, not 256: either change embedding float[256] to float[512] in the vec0 schema (double the vector storage) or load it with dimensionality=256 to keep the existing schema. Either way the switch is a full reindex, because vectors from different models never mix (see Model Hash Tracking below). The 4x larger model file increases memory usage but embedding speed remains orders of magnitude faster than transformer models.

Choose potion-retrieval-32M when your primary use case is retrieval (which vault search is). This variant is fine-tuned from potion-base-32M specifically for retrieval tasks, scoring 35.06 on Model2Vec’s retrieval benchmark table versus 32.67 for potion-base-32M.6 The trade-off is that it is optimized for retrieval rather than general-purpose embedding quality. It is also 512-dimensional, so the same float[512] or dimensionality=256 choice applies.6

Choose potion-multilingual-128M when your vault contains notes in multiple languages. Released May 2025, this 101-language model is the best performing static embedding model for multilingual tasks, generating embeddings for any text in any language with the same PyTorch-free inference as the other potion models.10 The larger model file (~500 MB) is the trade-off for cross-lingual capability. Use this when you have notes in Japanese, Chinese, German, or other non-English languages alongside English content.

Choose all-MiniLM-L6-v2 when retrieval quality matters more than speed and you have PyTorch installed. The 384-dimensional vectors need 50% more vector storage than 256-dimensional ones. Vectors are about 51 MB of the 83 MB database measured here (49,746 x 256 x 4 bytes), so the whole file grows by about 30%. Embedding speed drops from <1 minute to ~10 minutes for a full reindex of 15,000 files on M-series hardware.

Choose nomic-embed-text-v1.5 when you need the best possible local retrieval quality and accept slower indexing. The 768-dimensional vectors triple the vector storage, which a little more than doubles the database. Requires PyTorch and a modern CPU or GPU.

Choose text-embedding-3-small when network latency and privacy are acceptable trade-offs. The API produces the highest-quality embeddings but introduces a cloud dependency, per-token cost ($0.02/million tokens), and sends your content to OpenAI’s servers.

Stay with potion-base-8M in all other cases. The speed advantage is critical for iterative indexing (reindex during development), the PyTorch-free install avoids a heavyweight dependency, and the 256-dimensional vectors keep the database compact.

Quantization and Dimensionality Reduction

Model2Vec v0.5.0+ supports loading models with reduced precision and dimensions.8 This is useful for deployment on constrained hardware or reducing database size without switching models:

from model2vec import StaticModel

# Load with int8 weights (about a quarter of the float32 size)
model = StaticModel.from_pretrained("minishlab/potion-base-8M", quantize_to="int8")

# Load with reduced dimensions (e.g., 128 instead of 256)
model = StaticModel.from_pretrained("minishlab/potion-base-8M", dimensionality=128)

Quantized models retain near-identical retrieval quality at a fraction of the memory footprint: on model2vec 0.10.0, potion-base-8M loads as a 29,528 x 256 int8 matrix of 7.6 MB against 30.2 MB in float32. The parameter is quantize_to ("int8", "float16", "float32" or "float64").30 Dimensionality reduction follows Matryoshka-style truncation, where the first N dimensions carry the most information. Reducing from 256 to 128 dimensions halves the vector storage with minimal quality loss for short-text retrieval.

Model2Vec 0.8.x refactored the tokenizer and persistence internals, dropped Python 3.9 and refreshed the published results to the newer MTEB tables; 0.9.0 (August 2026) removed the Lightning dependency from training and rewrote inference; 0.10.0 (October 2026, the current release) replaced the PCA step, added ONNX export for classifier and regressor pipelines, a pairwise trainer with InfoNCE loss, token dropout during training, and faster encoding. Pin or test model2vec before upgrading a production indexer, because library upgrades can change model-loading paths even when the embedding model name stays the same.8

Fine-Tuning for Vault-Specific Embeddings

Model2Vec v0.4.0+ supports training custom classification models on top of static embeddings, v0.7.0 adds vocabulary quantization and configurable pooling for distillation, and v0.10.0 adds a pair-similarity trainer.8 This is relevant for vaults with specialized vocabulary (medical notes, legal references, domain-specific jargon) where the default potion models may not capture semantic nuances:

# pip install "model2vec[train]"  (training needs PyTorch; inference does not)
from model2vec.train import StaticModelForClassification

# Fine-tune a classifier head on vault-specific data
classifier = StaticModelForClassification.from_pretrained(path="minishlab/potion-base-8M")
classifier.fit(train_texts, train_labels)

# Convert to a PyTorch-free inference pipeline, then save it
pipeline = classifier.to_pipeline()
pipeline.save_pretrained("./vault-classifier")

The model2vec.train package exports model classes (StaticModelForClassification, StaticModelForSimilarity, StaticModelForRegression, StaticModelForPairSimilarity), not a standalone training function, and importing it without the train extra fails with an ImportError that names the missing torch dependency. The trained classifier has no save_pretrained of its own: to_pipeline() converts it to a StaticModelPipeline, which saves to a folder and loads back with StaticModelPipeline.from_pretrained("./vault-classifier") on a machine that has no PyTorch. For retrieval-oriented tuning, StaticModelForPairSimilarity (0.10.0) trains on text pairs with InfoNCE loss, the objective that matches a query-to-chunk search task.30

For most vaults, the default potion-base-8M produces sufficient retrieval quality. Fine-tuning is worthwhile only when retrieval consistently misses domain-specific connections that a general-purpose model cannot capture.

Model Hash Tracking

The indexer stores a hash derived from the model name, the vocabulary size (embedding.shape[0], the number of token rows) and the output dimension (dim). If you change the embedding model, the indexer detects the mismatch on the next incremental run and triggers a full reindex automatically.

import hashlib

def _compute_model_hash(self):
    """Hash model name, vocabulary size and dimension for compatibility tracking."""
    self._ensure_model()  # the hash is compared before anything is embedded
    key = f"{self._model_name}:{self._model.embedding.shape[0]}:{self._model.dim}"
    return hashlib.sha256(key.encode()).hexdigest()[:16]

StaticModel exposes dim, embedding and tokens. The method calls _ensure_model() first because an incremental run compares hashes before it embeds anything, when the lazily loaded model is still None. Including dim means a dimensionality=128 load or a 512-dimensional model also forces the full reindex.30

This prevents mixing vectors from different models in the same database, which would produce nonsensical cosine similarity scores.

Failure Modes

Model download failure. The first run downloads the model from Hugging Face. If the download fails (network issue, corporate firewall), the retriever falls back to BM25-only mode. The model is cached locally after the first download.

Dimension mismatch. If you switch models without clearing the database, the stored vectors have a different dimension than new embeddings. The indexer detects this via the model hash and triggers a full reindex. If the hash check fails (custom model without proper hash), sqlite-vec will error on KNN queries with mismatched dimensions.

Memory pressure on large vaults. Embedding 50,000+ chunks in a single batch can consume significant memory. The indexer processes in batches of 64 to limit peak memory usage. If memory is still an issue, reduce the batch size.


Full-Text Search with FTS5

SQLite’s FTS5 extension provides full-text search with BM25 ranking. FTS5 is the keyword search component of the hybrid retrieval pipeline. This section covers the FTS5 configuration, when BM25 excels, and its specific failure modes.

FTS5 Virtual Table

CREATE VIRTUAL TABLE chunks_fts USING fts5(
    chunk_text,
    section,
    heading_context,
    content=chunks,
    content_rowid=id
);

Content-sync mode. The content=chunks parameter tells FTS5 to reference the chunks table directly rather than storing a duplicate copy of the text. This halves the storage requirement but means FTS5 must be manually synced when chunks are inserted, updated, or deleted.

Columns. Three columns are indexed:

  • chunk_text — The primary content of each chunk (BM25 weight: 1.0)
  • section — The H2 heading text (BM25 weight: 0.5)
  • heading_context — Note title, tags, and metadata (BM25 weight: 0.3)

BM25 Ranking

BM25 ranks documents by term frequency, inverse document frequency, and document length normalization. The bm25() auxiliary function in FTS5 accepts per-column weights:

SELECT
    c.id, c.file_path, c.section, c.chunk_text,
    bm25(chunks_fts, 1.0, 0.5, 0.3) AS score
FROM chunks_fts
JOIN chunks c ON chunks_fts.rowid = c.id
WHERE chunks_fts MATCH ?
ORDER BY score
LIMIT 30;

The column weights (1.0, 0.5, 0.3) mean:

  • A keyword match in chunk_text contributes the most to the score
  • A match in section (heading) contributes half as much
  • A match in heading_context (title, tags) contributes 30% as much

These weights are tunable. If your vault has descriptive headings that strongly predict content quality, increase the section weight. If your tags are comprehensive and accurate, increase the heading_context weight.

Building the MATCH Query

The string bound to MATCH is not plain text. FTS5 parses it as a query expression: a bare term may contain only letters, digits, underscores and non-ASCII characters, anything else has to sit inside double quotes, and terms separated only by spaces are joined with an implicit AND.42 Bind what a user typed and three things go wrong. --incremental, AGENTS.md and what's new? raise syntax errors. sqlite-vec configuration raises no such column: vec. A natural-language question returns nothing unless a single chunk contains every one of its words. Quote each term and join the terms with OR:

import sqlite3
import sys

def fts_query(text):
    """Turn free text into an FTS5 query: every term quoted, joined with OR."""
    return " OR ".join('"' + t.replace('"', '""') + '"' for t in text.split())

def _bm25_search(self, query_text, limit=30):
    match = fts_query(query_text)
    if not match:
        return []
    try:
        return self.db.execute("""
            SELECT
                c.id, c.file_path, c.section, c.chunk_text,
                bm25(chunks_fts, 1.0, 0.5, 0.3) AS score
            FROM chunks_fts
            JOIN chunks c ON chunks_fts.rowid = c.id
            WHERE chunks_fts MATCH ?
            ORDER BY score
            LIMIT ?
        """, [match, limit]).fetchall()
    except sqlite3.OperationalError as e:  # FTS5 missing, or no index yet
        print(f"BM25 skipped: {e}", file=sys.stderr)
        return []

fts_query("sqlite-vec configuration") returns "sqlite-vec" OR "configuration". Inside the quotes the tokenizer still splits on punctuation, so a quoted identifier matches as a phrase of adjacent tokens: "AGENTS.md" finds AGENTS.md, "_rrf_fuse" finds _rrf_fuse, and "--incremental" finds the flag. OR means a chunk needs only one of the terms to qualify; BM25 then ranks the chunks that match more of them, and the rarer ones, first. FTS5’s bm25() gives better matches numerically lower values, which is why the query sorts ascending.42

When BM25 Wins

BM25 excels at queries containing exact identifiers:

  • Function names: _rrf_fuse, embed_batch, get_stale_files
  • CLI flags: --incremental, --vault, --model
  • Configuration keys: bm25_weight, max_tokens, batch_size
  • Error messages: SQLITE_LOCKED, ConnectionRefusedError
  • Specific terms of art: PostToolUse, PreToolUse, AGENTS.md

For these queries, BM25 finds the exact match immediately. Vector search would return semantically related content but might rank the exact match lower than a conceptual discussion.

When BM25 Fails

BM25 fails at queries that use different terminology than the stored content:

  • Query: “how to handle authentication failures” → Vault contains notes about “login error recovery” and “session expiration handling.” BM25 has no shared content word to rank on: it matches only filler such as “how” and “to”, so those words decide the ranking and relevance does not.
  • Query: “what is the best way to manage state” → Vault contains notes about “Redux store patterns” and “context providers.” BM25 misses because “state management” is expressed through specific technology names.

BM25 also fails with keyword collision at scale. In a 15,000-file vault, a search for “configuration” matches hundreds of notes because nearly every project note mentions configuration. The results are technically correct but practically useless — the ranking cannot determine which “configuration” note is relevant to the current query.

FTS5 Tokenizer

FTS5 uses the unicode61 tokenizer by default, which handles ASCII and Unicode text. For vaults with significant CJK (Chinese, Japanese, Korean) content, consider the trigram tokenizer:

-- For CJK-heavy vaults
CREATE VIRTUAL TABLE chunks_fts USING fts5(
    chunk_text, section, heading_context,
    content=chunks, content_rowid=id,
    tokenize='trigram'
);

The default unicode61 tokenizer splits on word boundaries, which works poorly for languages without spaces between words. The trigram tokenizer splits every three characters, enabling substring matching at the cost of index size (roughly 3x larger).

Maintenance

FTS5 requires explicit sync when the underlying chunks table changes:

# After inserting chunks
cursor.execute("""
    INSERT INTO chunks_fts(chunks_fts)
    VALUES('rebuild')
""")

The rebuild command reconstructs the FTS5 index from the content table. Run it after bulk inserts (full reindex) but not after individual incremental updates. For those, use INSERT INTO chunks_fts(rowid, chunk_text, section, heading_context) to sync individual rows, and issue the matching 'delete' row before a chunk is removed or replaced (both are shown under Content Hashing and Change Detection).


Vector Search with sqlite-vec

The sqlite-vec extension brings vector KNN (K-Nearest Neighbors) search into SQLite. This section covers the sqlite-vec configuration, the embedding pipeline from note to searchable vector, and the specific query patterns.

sqlite-vec Virtual Table

CREATE VIRTUAL TABLE chunk_vecs USING vec0(
    id INTEGER PRIMARY KEY,
    embedding float[256] distance_metric=cosine
);

The vec0 module stores 256-dimensional float vectors as packed binary data. The id column maps 1:1 to the chunks table, enabling joins between vector results and chunk metadata. distance_metric=cosine matters: vec0 defaults to L2 (Euclidean) distance, and the distance values the rest of this guide quotes are cosine distances (0 = identical, 2 = opposite). The potion models emit unit-length vectors, so the two metrics rank identically (squared L2 equals twice the cosine distance), but the numbers differ, and a distance threshold tuned against one metric is wrong for the other.31

Migrating an existing index. The metric is fixed when the table is created. If your chunk_vecs table was created without distance_metric=cosine, either keep it and read its distance as L2 (for normalized vectors, cosine = distance * distance / 2), or change the schema and run a full reindex (python index_vault.py --full), which drops and recreates the table. The dimension stays 256, so nothing else changes.

Embedding Pipeline

The pipeline flows from note to searchable vector:

Note (.md file)
  → Chunker: split at H2 boundaries
    → Chunks (30-2000 chars each)
      → Credential filter: scrub secrets
        → Embedder: Model2Vec encode
          → Vectors (256-dim float arrays)
            → sqlite-vec: store as packed binary
              → Ready for KNN queries

Vector Serialization

Python’s struct module serializes float vectors for sqlite-vec storage:

import struct

def _serialize_vector(vec):
    """Pack float list into binary for sqlite-vec."""
    return struct.pack(f"{len(vec)}f", *vec)

def _deserialize_vector(blob, dim=256):
    """Unpack binary blob to float list."""
    return list(struct.unpack(f"{dim}f", blob))

KNN Query

A vector search query embeds the input query, then finds the K nearest chunks by cosine distance:

def _vector_search(self, query_text, limit=30):
    query_vec = self.embedder.embed_batch([query_text])[0]
    packed = _serialize_vector(query_vec)

    results = self.db.execute("""
        SELECT
            cv.id,
            cv.distance,
            c.file_path,
            c.section,
            c.chunk_text
        FROM chunk_vecs cv
        JOIN chunks c ON cv.id = c.id
        WHERE embedding MATCH ?
            AND k = ?
        ORDER BY distance
    """, [packed, limit]).fetchall()

    return results

The MATCH operator in sqlite-vec runs an exact, brute-force KNN scan over the vec0 chunks; approximate indexes exist only in the 0.1.10 alpha line, covered below. The k parameter controls how many results to return. With distance_metric=cosine declared on the column, the distance column contains the cosine distance (0 = identical, 2 = opposite).31

KNN Pagination with Distance Constraints

As of sqlite-vec v0.1.7, KNN queries accept >, >=, < and <= constraints on the distance column. Asking for rows with distance > the last distance you saw returns the next page.11 The release notes caution that this is not real pagination: rows that tie exactly at the boundary distance are skipped, so use it to walk further down a ranking, not as a stable cursor. The later v0.1.8 and v0.1.9 stable releases are packaging and DELETE bug-fix releases, so v0.1.7 remains the feature boundary for this pattern.21

On the horizon, the v0.1.10-alpha line (March 31 – May 18, 2026) is the first to move sqlite-vec beyond brute-force KNN: it introduces approximate-nearest-neighbor index types — rescore, an experimental ivf (inverted-file) index that is not enabled by default, and a disk-based DiskANN index for vaults too large to keep vectors resident in memory.21 These would change the scaling story for very large vaults, but the 0.1.10 line is still pre-release (alpha) — treat ANN indexing as experimental and continue building on the stable v0.1.9 brute-force KNN path for production vaults until a stable 0.1.10 ships.

def _paginated_vector_search(self, query_vec, page_size=20, after_distance=None):
    """Page through KNN results. Pass the previous page's cursor as after_distance."""
    packed = _serialize_vector(query_vec)
    params = [packed, page_size]
    constraint = ""
    if after_distance is not None:  # 0.0 is a valid cursor
        constraint = "AND distance > ?"
        params.append(after_distance)

    results = self.db.execute(f"""
        SELECT cv.id, cv.distance, c.file_path, c.chunk_text
        FROM chunk_vecs cv
        JOIN chunks c ON cv.id = c.id
        WHERE embedding MATCH ?
            AND k = ?
            {constraint}
        ORDER BY distance
    """, params).fetchall()

    # The last distance on this page is the cursor for the next one
    next_cursor = results[-1]["distance"] if results else None
    return results, next_cursor

This replaces the previous pattern of fetching a large k and slicing in Python, reducing memory usage for exploratory queries over large vaults.

DELETE Support in vec0 Tables

DELETE has always removed rows from a vec0 table. What changed is the cleanup: v0.1.7 made a delete reclaim space (a storage chunk of about 1,024 vectors is freed once enough of its rows are gone), and v0.1.9 fixed a false error that DELETE reported on tables with a metadata text column longer than 12 characters.1121 The indexer’s file-removal path deletes vectors directly:

db.execute("DELETE FROM chunk_vecs WHERE id = ?", [chunk_id])

On a release older than 0.1.7 the same statement works, but the space those vectors used is never released as notes are deleted or moved. Upgrade to 0.1.9 so the space comes back and schemas with metadata columns delete cleanly.

When Vector Search Wins

Vector search excels at queries where the concept matters more than the specific words:

  • Query: “how to handle authentication failures” → Finds notes about “login error recovery” (same semantic space, different keywords)
  • Query: “what patterns exist for caching” → Finds notes about “memoization,” “Redis TTL strategies,” and “HTTP cache headers” (related concepts, diverse terminology)
  • Query: “approaches to testing asynchronous code” → Finds notes about “pytest-asyncio fixtures,” “mock event loops,” and “async test patterns” (same concept expressed through implementation details)

When Vector Search Fails

Vector search struggles with exact identifiers:

  • Query: _rrf_fuse → Returns notes about “fusion algorithms” and “rank merging” but may rank the actual function definition lower than conceptual discussions
  • Query: PostToolUse → Returns notes about “tool lifecycle hooks” and “post-execution handlers” rather than the specific hook name

Vector search also struggles with structured data. JSON configuration files, YAML blocks, and code snippets produce embeddings that capture structural patterns rather than semantic meaning. A JSON file with "review": true embeds differently than a prose discussion of code review.

Graceful Degradation

If sqlite-vec fails to load (missing extension, incompatible platform, corrupted library), the retriever falls back to BM25-only search:

import sqlite3
import sys

class VectorIndex:
    def __init__(self, db_path):
        self.db = sqlite3.connect(db_path)
        self.db.row_factory = sqlite3.Row  # read columns by name: r["file_path"]
        self._vec_available = False
        _activate_venv()  # from embedder.py: sqlite-vec is installed in the same venv
        try:
            import sqlite_vec  # inside the try, so a missing package degrades too
            self.db.enable_load_extension(True)
            sqlite_vec.load(self.db)  # finds vec0 inside the pip package
            self.db.enable_load_extension(False)
            self._vec_available = True
        except Exception as e:
            print(f"BM25-only mode: sqlite-vec failed to load: {e}", file=sys.stderr)

    @property
    def vec_available(self):
        return self._vec_available

Load the extension through sqlite_vec.load(). The pip package installs vec0.dylib or vec0.so inside site-packages/sqlite_vec/, not on the dynamic loader’s search path, so a bare db.load_extension("vec0") fails with “no such file” and, inside a silent except, leaves the retriever permanently in BM25-only mode with nothing in the logs. Print the failure so the fallback is visible.31 The import sits inside the try for the same reason: when the package is not installed at all, the import is the line that fails. row_factory = sqlite3.Row makes every row readable by column name, which _rrf_fuse and the MCP wrapper later rely on.

The retriever checks vec_available before attempting vector queries. When it is false, every search is BM25-only. The fusion step still runs, over that one list, so results carry the same fields in both modes (see The Search API).


Reciprocal Rank Fusion (RRF)

RRF merges two ranked lists without requiring score calibration. This section covers the algorithm, a worked query trace, tuning the k parameter, and why RRF is chosen over alternatives. For an interactive calculator with editable ranks, scenario presets, and a visual architecture explorer, see the hybrid retriever deep dive.

The Algorithm

RRF assigns each document a score based only on its rank position in each list:

score(d) = Σ (weight_i / (k + rank_i))

Where:

  • k is a smoothing constant (60, following Cormack et al.1)
  • rank_i is the document’s 1-based rank in result list i
  • weight_i is an optional per-list multiplier (default 1.0)

Documents that rank well in multiple lists receive higher fused scores. Documents that appear in only one list receive a score from that single source.

Why RRF Over Alternatives

Weighted linear combination requires calibrating BM25 scores against cosine distances. BM25 scores are unbounded and scale with corpus size. Cosine distances are bounded [0, 2]. Combining them requires normalization, and the normalization parameters are dataset-dependent. RRF uses only rank positions, which are always integers starting at 1 regardless of the scoring method.

Learned fusion models require labeled training data — query-document relevance pairs. For a personal knowledge base, this training data does not exist. You would need to manually judge hundreds of query-document pairs to train a useful model. RRF works without any training data.

Voting-based methods (Borda count, and Condorcet methods such as Schulze) are theoretically elegant but more complex to implement and tune. In the original RRF paper’s TREC experiments, RRF scored higher than Condorcet Fuse in every case and higher than CombMNZ in all but one.1

Fusion in Practice

Query: “how does the review aggregator handle disagreements”

BM25 ranks review-aggregator.py at position 3 (exact keyword matches on “review,” “aggregator,” “disagreements”) but places two config files higher (they match “review” more prominently). Vector search ranks the same chunk at position 1 (semantic match on conflict resolution). After RRF fusion:

Chunk BM25 Vec Fused Score
review-aggregator.py “Disagreement Resolution” #3 #1 0.0323
code-review-patterns.md “Multi-Reviewer” #4 #2 0.0318
deliberation-config.json “Review Weights” #1 — 0.0164

Chunks that rank well in both lists surface to the top. Chunks that only appear in one list get a single-source score and drop below dual-ranked results. The actual disagreement resolution logic wins because both methods found it — BM25 through keywords, vector search through semantics.

For the full step-by-step trace with per-rank RRF math, try different k values in the interactive RRF calculator.

Implementation

RRF_K = 60

def _rrf_fuse(self, bm25_results, vec_results,
              bm25_weight=1.0, vec_weight=1.0):
    """Fuse BM25 and vector results using Reciprocal Rank Fusion."""
    scores = {}

    for rank, r in enumerate(bm25_results, start=1):
        cid = r["id"]
        if cid not in scores:
            scores[cid] = {
                "rrf_score": 0.0,
                "file_path": r["file_path"],
                "section": r["section"],
                "chunk_text": r["chunk_text"],
                "bm25_rank": None,
                "vec_rank": None,
            }
        scores[cid]["rrf_score"] += bm25_weight / (self._rrf_k + rank)
        scores[cid]["bm25_rank"] = rank

    for rank, r in enumerate(vec_results, start=1):
        cid = r["id"]
        if cid not in scores:
            scores[cid] = {
                "rrf_score": 0.0,
                "file_path": r["file_path"],
                "section": r["section"],
                "chunk_text": r["chunk_text"],
                "bm25_rank": None,
                "vec_rank": None,
            }
        scores[cid]["rrf_score"] += vec_weight / (self._rrf_k + rank)
        scores[cid]["vec_rank"] = rank

    fused = sorted(
        scores.values(),
        key=lambda x: x["rrf_score"],
        reverse=True,
    )
    return fused

Tuning k

The k constant controls how much weight is given to top-ranked results versus lower-ranked results:

  • Lower k (e.g., 10): Top-ranked results dominate. Rank 1 scores 1/11 = 0.091, rank 10 scores 1/20 = 0.050 (1.8x difference). Good when you trust the individual rankers to get the top result right.
  • Default k (60): Balanced. Rank 1 scores 1/61 = 0.0164, rank 10 scores 1/70 = 0.0143 (1.15x difference). Rank differences are compressed, giving more weight to appearing in multiple lists.
  • Higher k (e.g., 200): Appearing in both lists matters much more than rank position. Rank 1 scores 1/201, rank 10 scores 1/210 — nearly identical. Use when the individual rankers produce noisy rankings but cross-list agreement is reliable.

Start with k=60. The original RRF paper fixed k = 60 after a pilot experiment found it near-optimal and the exact choice not critical.1 Tune only after measuring failure cases on your own query distribution.

Tie-Breaking

When two chunks have identical RRF scores (rare but possible with the same rank in one list and no appearance in the other), break ties by:

  1. Prefer chunks that appear in both lists over chunks that appear in only one
  2. Among chunks in both lists, prefer the one with the lower combined rank
  3. Among chunks in only one list, prefer the one with the lower rank in that list

The Complete Retrieval Pipeline

This section traces a query from input to output through the entire pipeline: BM25 search, vector search, RRF fusion, token budget truncation, and context assembly.

End-to-End Flow

User query: "PostToolUse hook for context compression"
  │
  ├─ BM25 Search (FTS5)
  │    → MATCH '"PostToolUse" OR "hook" OR "for" OR "context" OR "compression"'
  │    → Top 30 results ranked by BM25 score
  │    → 12ms
  │
  ├─ Vector Search (sqlite-vec)
  │    → Embed query with Model2Vec
  │    → KNN k=30 on chunk_vecs
  │    → Top 30 results ranked by cosine distance
  │    → 8ms
  │
  └─ RRF Fusion
       → Merge 60 candidates (may overlap)
       → Score by rank position
       → Top 10 results
       → 3ms
       │
       └─ Token Budget
            → Truncate to max_tokens (default 4000)
            → Estimate at 4 chars per token
            → Return results with metadata
            → <1ms

Total latency: ~23ms for a 49,746-chunk database on Apple M3 Pro hardware.

The Search API

import os
import sys

class HybridRetriever:
    def __init__(self, db_path, vault_path=None, rrf_k=RRF_K):
        if not os.path.exists(db_path):
            raise FileNotFoundError(
                f"{db_path} not found. Run: python index_vault.py --full"
            )
        self.index = VectorIndex(db_path)    # from Graceful Degradation
        self.db = self.index.db
        self.embedder = Model2VecEmbedder()  # lazy: loads on the first vector query
        self.vault_path = vault_path
        self._rrf_k = rrf_k

    # _bm25_search, _vector_search and _rrf_fuse from the sections
    # above are methods of this class.

    def search(self, query, limit=10, max_tokens=4000,
               bm25_weight=1.0, vec_weight=1.0):
        """
        Search the vault using hybrid BM25 + vector retrieval.

        Args:
            query: Search query text
            limit: Maximum results to return
            max_tokens: Token budget for total result text
            bm25_weight: Weight for BM25 results in RRF
            vec_weight: Weight for vector results in RRF

        Returns:
            List of dicts with file_path, section, chunk_text,
            rrf_score, bm25_rank, vec_rank
        """
        if not query.strip():
            return []

        # BM25 search
        bm25_results = self._bm25_search(query, limit=30)

        # Vector search (if sqlite-vec loaded and the model can be loaded)
        vec_results = []
        if self.index.vec_available:
            try:
                vec_results = self._vector_search(query, limit=30)
            except Exception as e:
                print(f"BM25-only for this query: vector search failed: {e}",
                      file=sys.stderr)

        # Fuse. With one list empty this is the single-source fallback,
        # and every result still carries rrf_score, bm25_rank and vec_rank.
        fused = self._rrf_fuse(
            bm25_results, vec_results,
            bm25_weight, vec_weight,
        )

        # Token budget truncation
        results = []
        token_count = 0
        for r in fused[:limit]:
            chunk_tokens = len(r["chunk_text"]) // 4
            if token_count + chunk_tokens > max_tokens:
                break
            results.append(r)
            token_count += chunk_tokens

        return results

Token Budget Truncation

The max_tokens parameter prevents the retriever from returning more context than the AI tool can use. The estimate uses 4 characters per token (a reasonable approximation for English prose). Results are truncated greedily: add results in ranked order until the budget is exhausted.

This is a conservative strategy. A more sophisticated approach would consider per-result quality scores and prefer shorter, higher-quality results over longer, lower-quality results. The greedy approach is simpler and works well in practice because RRF ranking already orders results by relevance.

Database Schema (Complete)

-- Chunk content and metadata
CREATE TABLE chunks (
    id INTEGER PRIMARY KEY,
    file_path TEXT NOT NULL,
    section TEXT NOT NULL,
    chunk_text TEXT NOT NULL,
    heading_context TEXT DEFAULT '',
    mtime_ns INTEGER NOT NULL,
    embedded_at REAL NOT NULL
);

CREATE INDEX idx_chunks_file ON chunks(file_path);
CREATE INDEX idx_chunks_mtime ON chunks(mtime_ns);

-- FTS5 for BM25 search (content-synced to chunks table)
CREATE VIRTUAL TABLE chunks_fts USING fts5(
    chunk_text, section, heading_context,
    content=chunks, content_rowid=id
);

-- sqlite-vec for vector KNN search (cosine; the default is L2)
CREATE VIRTUAL TABLE chunk_vecs USING vec0(
    id INTEGER PRIMARY KEY,
    embedding float[256] distance_metric=cosine
);

-- Model metadata for compatibility tracking
CREATE TABLE model_meta (
    key TEXT PRIMARY KEY,
    value TEXT
);

Create the schema once per database with index.db.executescript(...), after VectorIndex has loaded the extension: the vec0 module does not exist on a connection until sqlite_vec.load() has run. In BM25-only mode, leave out the chunk_vecs statement.

Graceful Degradation Path

Full pipeline:     BM25 + Vector + RRF  →  Best results
No sqlite-vec:     BM25 only            →  Good results (no semantic)
No model download:  BM25 only            →  Good results (no semantic)
No FTS5:           Vector only           →  Decent results (no keyword)
No database:       Error                 →  Prompt user to run indexer

The retriever checks capabilities at initialization and adapts its query strategy. A missing component degrades quality but does not cause errors: a failed vector search or a missing FTS5 table is reported on stderr, and the other half still answers. The only hard failure is a missing database file, which HybridRetriever raises as FileNotFoundError with the indexer command in the message.

Production Stats

Measured on a vault of 16,894 files, 49,746 chunks, 83 MB SQLite database, Apple M3 Pro:

Metric Value
Total files 16,894
Total chunks 49,746
Database size 83 MB
BM25 query latency (p50) 12ms
Vector query latency (p50) 8ms
RRF fusion latency 3ms
End-to-end search latency (p50) 23ms
Full reindex time ~4 minutes
Incremental reindex time <10 seconds
Embedding model potion-base-8M (256-dim)
BM25 candidate pool 30
Vector candidate pool 30
Default result limit 10
Default token budget 4,000 tokens

Content Hashing and Change Detection

The indexer needs to know which files have changed since the last index run. This section covers the change detection mechanism and the hashing strategy.

File Modification Time Comparison

The indexer stores mtime_ns (file modification time in nanoseconds) for every chunk in the chunks table. On an incremental run, the indexer:

  1. Scans the vault for all .md files in allowed folders
  2. Reads the mtime_ns for each file from the filesystem
  3. Compares against the stored mtime_ns in the database
  4. Identifies three categories:
  5. New files: path exists in filesystem but not in database
  6. Changed files: path exists in both but mtime_ns differs
  7. Deleted files: path exists in database but not in filesystem
def get_stale_files(self, vault_mtimes):
    """Find files whose mtime changed or are new."""
    stored = dict(self.db.execute(
        "SELECT DISTINCT file_path, mtime_ns FROM chunks"
    ).fetchall())

    stale = []
    for path, mtime in vault_mtimes.items():
        if path not in stored or stored[path] != mtime:
            stale.append(path)
    return stale

def get_deleted_files(self, vault_paths):
    """Find files in database that no longer exist in vault."""
    stored_paths = set(r[0] for r in self.db.execute(
        "SELECT DISTINCT file_path FROM chunks"
    ).fetchall())
    return stored_paths - set(vault_paths)

Why mtime, Not Content Hash

Content hashing (SHA-256 of file contents) would be more reliable than mtime comparison: it would detect cases where a file was touched without changing (e.g., touch, or a sync tool rewriting an identical file). However, hashing requires reading every file on every incremental run. For 16,894 files, reading file contents takes 2-3 seconds. Reading mtimes from the filesystem takes <100ms.

The trade-off: mtime comparison occasionally triggers unnecessary re-indexing of unchanged files (false positives), and it misses a change only when a tool writes new content while keeping the old timestamp (some sync tools do; see Index Drift under Troubleshooting). False positives cost a few extra embedding calls per run. The speed difference (100ms vs 3 seconds) makes mtime the pragmatic choice for a system that runs on every AI interaction.

Handling Deletions

When a file is deleted from the vault, the indexer removes all its chunks from the database:

def remove_file(self, file_path):
    """Remove a file's chunks, their FTS5 entries and their vectors."""
    rows = self.db.execute(
        "SELECT id, chunk_text, section, heading_context "
        "FROM chunks WHERE file_path = ?",
        [file_path],
    ).fetchall()

    for r in rows:
        # External-content FTS5: hand back the indexed text to delete its entries
        self.db.execute(
            "INSERT INTO chunks_fts(chunks_fts, rowid, chunk_text, section, heading_context) "
            "VALUES('delete', ?, ?, ?, ?)",
            [r["id"], r["chunk_text"], r["section"], r["heading_context"]],
        )
        if self.vec_available:
            self.db.execute("DELETE FROM chunk_vecs WHERE id = ?", [r["id"]])

    self.db.execute("DELETE FROM chunks WHERE file_path = ?", [file_path])
    self.db.commit()

The 'delete' rows are not optional. chunks_fts is an external-content table, so FTS5 keeps its own index entries and can remove them only when it is handed the row’s original text. Delete from chunks alone and keyword searches keep returning the dead rowids, and the FTS5 integrity-check command fails.42 Issue the 'delete' while the row is still in chunks, as the code does.

DELETE FROM chunk_vecs works on every sqlite-vec release; from v0.1.7 it also reclaims the space, and v0.1.9 fixes a false DELETE error on tables with longer metadata text columns.1121 If you run a release older than 0.1.9, upgrade before relying on deletes in a schema with metadata columns.

If the indexer ever ran while sqlite-vec failed to load, run a full reindex once it loads again. A removal made in BM25-only mode leaves the file’s vectors behind, and the next vector insert that reuses one of those ids fails with a UNIQUE constraint error; chunks indexed in that mode have no vectors at all.

Indexing a Changed File

Re-indexing a file is the same removal followed by three inserts per chunk: the row, its FTS5 entry and its vector. chunks is the chunker’s output, a list of dicts with section, text and heading_context, already passed through the credential filter; vectors is embedder.embed_batch([c["text"] for c in chunks]).

import time

def index_file(self, file_path, chunks, vectors, mtime_ns):
    """Replace a file's rows: each chunk, its FTS5 entry and its vector."""
    self.remove_file(file_path)
    for chunk, vec in zip(chunks, vectors):
        cur = self.db.execute(
            "INSERT INTO chunks(file_path, section, chunk_text, heading_context, "
            "mtime_ns, embedded_at) VALUES(?, ?, ?, ?, ?, ?)",
            [file_path, chunk["section"], chunk["text"],
             chunk["heading_context"], mtime_ns, time.time()],
        )
        self.db.execute(
            "INSERT INTO chunks_fts(rowid, chunk_text, section, heading_context) "
            "VALUES(?, ?, ?, ?)",
            [cur.lastrowid, chunk["text"], chunk["section"], chunk["heading_context"]],
        )
        if self.vec_available:
            self.db.execute(
                "INSERT INTO chunk_vecs(id, embedding) VALUES(?, ?)",
                [cur.lastrowid, _serialize_vector(vec)],
            )
    self.db.commit()

get_stale_files, get_deleted_files, remove_file and index_file are methods of VectorIndex, next to the connection they use.


Incremental vs Full Reindex

The indexer supports two modes: incremental (fast, daily use) and full (slow, occasional). This section covers when to use each, the idempotency guarantees, and corruption recovery.

Incremental Reindex

When to use: Daily indexing after editing notes. The default mode.

What it does: 1. Scan vault for file changes (mtime comparison) 2. Delete chunks for deleted files 3. Re-chunk and re-embed changed files 4. Insert new chunks for new files 5. Sync FTS5 index

Typical duration: <10 seconds for a day’s edits on a 16,000-file vault.

python index_vault.py --incremental

Full Reindex

When to use:

  • After changing the embedding model (model hash mismatch detected)
  • After schema migration (new columns, changed indexes)
  • After database corruption (integrity check fails)
  • When incremental indexing produces unexpected results

What it does: 1. Drop all existing data (chunks, vectors, FTS5 entries) 2. Scan entire vault 3. Chunk all files 4. Embed all chunks 5. Build FTS5 index from scratch

Typical duration: ~4 minutes for 16,894 files on Apple M3 Pro.

python index_vault.py --full

Idempotency

Both modes are idempotent: running the same command twice produces the same result. The indexer deletes existing chunks for a file before inserting new ones, so a re-run of incremental indexing on an already-current database produces zero changes. A re-run of full indexing produces an identical database.

Corruption Recovery

If the SQLite database becomes corrupted (power loss during write, disk error, killed process mid-transaction):

# Check integrity
sqlite3 vectors.db "PRAGMA integrity_check;"

# If corruption detected, full reindex rebuilds from source files
python index_vault.py --full

The source of truth is always the vault files, not the database. The database is a derived artifact that can be rebuilt at any time. This is a critical design property: you never need to back up the database.

The --incremental Flag

When the indexer runs with --incremental:

  1. Model hash check. Compare stored model hash against current model. If different, automatically switch to full reindex mode and warn the user.
  2. File scan. Walk allowed folders, collect file paths and mtimes.
  3. Change detection. Compare against stored data.
  4. Batch processing. Re-chunk and re-embed changed files in batches of 64.
  5. Progress reporting. Print count of processed files and elapsed time.
  6. Graceful shutdown. Handle SIGINT by finishing the current file before stopping.

Credential Filtering and Data Boundaries

Personal notes contain secrets: API keys, bearer tokens, database connection strings, private keys pasted during debugging sessions. The credential filter prevents these from entering the retrieval index.

The Problem

A note about debugging an OAuth integration might contain:

The token was: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...
I used this curl command:
  curl -H "Authorization: Bearer sk-ant-api03-abc123..."

Without filtering, both the JWT and the API key would be chunked, embedded, and stored in the database. A search for “authentication” would return the chunk containing real secrets. Worse, if the retriever feeds results to an AI tool through MCP, the secrets appear in the AI’s context window and potentially in the tool’s logs.

Pattern-Based Filtering

The credential filter runs on every chunk before storage, matching 25 vendor-specific patterns plus generic patterns:

Vendor-Specific Patterns:

Pattern Example Regex
Anthropic API key sk-ant-api03-... sk-ant-api\d{2}-[a-zA-Z0-9_-]{20,}
OpenAI API key sk-... sk-[a-zA-Z0-9_-]{20,}
GitHub token ghp_..., github_pat_... (?:gh[pousr]_[A-Za-z0-9._-]{36,}\|github_pat_[A-Za-z0-9_]{22,})
AWS Access Key AKIA... AKIA[0-9A-Z]{16}
Stripe key sk_live_... [sr]k_(live\|test)_[a-zA-Z0-9]{24,}
Cloudflare token ... Various patterns

Order matters. The patterns run top to bottom, and the OpenAI pattern also matches an Anthropic key, so the Anthropic row comes first; in the other order every Anthropic key is logged as an OpenAI one. The GitHub pattern covers the five gh?_ prefixes (ghp_, gho_, ghu_, ghs_, ghr_) and fine-grained github_pat_ tokens. Its character class allows dots and underscores because the longer installation tokens GitHub began rolling out in April 2026 contain both.53

Generic Patterns:

Pattern Detection
JWT tokens eyJ[a-zA-Z0-9_-]+\.eyJ[a-zA-Z0-9_-]+(?:\.[a-zA-Z0-9_-]+)?
Bearer tokens Bearer\s+[a-zA-Z0-9_\-\.]+
Private keys -----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z ]*PRIVATE KEY-----
High-entropy base64 Strings with >4.5 bits/char entropy, 40+ chars
Password assignments password\s*[:=]\s*(?:"[^"]*"\|'[^']*'\|\S+)

The private-key pattern runs from the BEGIN line to the matching END line, so the key body is replaced along with its header, and [A-Z ]* covers PKCS#8 (BEGIN PRIVATE KEY) and encrypted keys as well as RSA, EC and OpenSSH ones. The JWT pattern takes the signature segment when there is one. The password pattern accepts a quoted value (spaces included) or a bare one, so password: hunter2 is caught; it also redacts the word after password: in an ordinary sentence, which is the safer side to err on. In both tables, \| stands for a plain | (the table syntax needs the backslash); drop the backslash when you copy a pattern into code.

Filter Implementation

def clean_content(text):
    """Scrub credentials from text before indexing."""
    result = ScanResult(is_clean=True, match_count=0, patterns=[])

    for pattern in CREDENTIAL_PATTERNS:
        matches = pattern.regex.findall(text)
        if matches:
            text = pattern.regex.sub(
                f"[REDACTED:{pattern.name}]", text
            )
            result.is_clean = False
            result.match_count += len(matches)
            result.patterns.append(pattern.name)

    return text, result

Key design choices:

  1. Filter before embedding. The cleaned text is what gets embedded. The vector representation never encodes credential patterns. A query for “API key” returns notes that discuss API key management, not notes that contain actual keys.

  2. Replace, not remove. The [REDACTED:pattern-name] token preserves the semantic context of the surrounding text. The embedding captures that “something credential-like was here” without encoding the credential itself.

  3. Log patterns, not values. The filter logs which patterns matched (e.g., “Scrubbed 2 credential(s) from oauth-debug.md [jwt, bearer-token]”) but never logs the credential value.

Path-Based Exclusion

The .indexignore file provides coarse-grained exclusion by path. The credential filter provides fine-grained scrubbing within indexed files. Both are necessary:

  • .indexignore for entire folders you know contain sensitive content (health notes, financial records, career documents)
  • Credential filter for secrets accidentally embedded in otherwise-indexable content

Data Classification

For vaults containing diverse content, consider classifying notes by sensitivity:

Level Examples Index? Filter?
Public Blog drafts, technical notes Yes Yes
Internal Project plans, architecture decisions Yes Yes
Sensitive Salary data, health records No (.indexignore) N/A
Restricted Credentials, private keys No (.indexignore) N/A

MCP Server Architecture

Model Context Protocol (MCP) servers expose the retriever as a tool that AI agents can call. This section covers the server design, capability surface, and permission boundaries.

Protocol Choice: STDIO vs HTTP

MCP supports two transport modes:

STDIO — The AI tool spawns the MCP server as a child process and communicates over stdin/stdout. This is the standard mode for local tools. Claude Code, Codex CLI, and Cursor all support STDIO MCP servers.

In this guide’s configs and commands, python stands for the interpreter that has the server’s dependencies installed. A Mac has no python command unless you add one (macOS ships only python3), so in a config give that interpreter’s absolute path, for example /path/to/venv/bin/python.55

{
  "mcpServers": {
    "obsidian": {
      "command": "python",
      "args": ["/path/to/obsidian_mcp.py"],
      "env": {
        "VAULT_PATH": "/path/to/vault",
        "DB_PATH": "/path/to/vectors.db"
      }
    }
  }
}

HTTP — The MCP server runs as a standalone HTTP service. Useful for remote access, multi-client setups, or team configurations where the vault is on a shared server.

{
  "mcpServers": {
    "obsidian": {
      "type": "http",
      "url": "http://localhost:3333/mcp"
    }
  }
}

In Claude Code the type is required on a url entry. An entry with no type is read as a stdio server, so Claude Code skips it and reports a configuration error; claude mcp add --transport http obsidian http://localhost:3333/mcp writes the entry for you.40 Cursor’s documented remote entry is url alone (plus optional headers).50

Recommendation: Use STDIO for personal vaults. It is simpler, more secure (no network exposure), and the server lifecycle is managed by the AI tool. Use HTTP only when multiple tools or multiple machines need concurrent access to the same vault.

MCP Spec Evolution. The March 2025 revision (2025-03-26) replaced HTTP+SSE with the Streamable HTTP transport, added an authorization framework based on OAuth 2.1, and added tool annotations that describe a tool’s behavior, such as whether it is read-only or destructive. The June 2025 revision (2025-06-18) classed MCP servers as OAuth resource servers and required clients to send RFC 8707 resource indicators; it also added structured tool outputs (typed return schemas) and elicitation (server-initiated user prompts). The November 2025 revision (2025-11-25) added OpenID Connect discovery for authorization servers, a .well-known fallback for protected-resource metadata, OAuth Client ID Metadata Documents, URL-mode elicitation, icons metadata, tool calling inside sampling, experimental tasks, JSON Schema 2020-12 as the default dialect, and an SDK tiering system.57

The 2026-07-28 revision is the Current specification (release candidate May 21, 2026; final July 28, 2026), the largest MCP revision since launch. Its headline changes: a stateless protocol core (the initialize handshake and Mcp-Session-Id header are removed, so servers no longer track per-connection state, and a server/discover method, which every server must implement, advertises versions and capabilities), an extensions framework with two official extensions, MCP Apps (server-rendered HTML in sandboxed client iframes) and Tasks (tasks/get, tasks/update, tasks/cancel for long-running work), hardened OAuth 2.0 / OIDC authorization, and a twelve-month feature-deprecation lifecycle policy. The same revision deprecates Roots, Sampling and Logging, and replaces server-initiated requests (elicitation included) with the multi-round-trip pattern, where the server returns an input_required result and the client answers in a new request; a vault server that prompts the user should plan for that shape.2232 For personal vault servers, STDIO remains the simplest path, and the stateless core makes single-user STDIO servers even thinner. Streamable HTTP, .well-known discovery and MCP Apps primarily benefit enterprise HTTP deployments. Monitor the MCP roadmap for updates that affect your transport choice.

Capability Design

The MCP server should expose a minimal set of tools. Each definition declares its arguments under inputSchema, a JSON Schema object with type: "object", and the optional annotations.readOnlyHint tells a client that the tool does not modify anything, which is true of all four here.52

search — The primary tool. Runs hybrid retrieval and returns ranked results.

{
  "name": "obsidian_search",
  "description": "Search the Obsidian vault using hybrid BM25 + vector retrieval",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": { "type": "string", "description": "Search query" },
      "limit": { "type": "integer", "default": 5 },
      "max_tokens": { "type": "integer", "default": 2000 }
    },
    "required": ["query"]
  },
  "annotations": { "readOnlyHint": true }
}

read_note — Read the full content of a specific note by path. Useful when the agent wants to see the complete context of a search result.

{
  "name": "obsidian_read_note",
  "description": "Read the full content of a note by file path",
  "inputSchema": {
    "type": "object",
    "properties": {
      "file_path": { "type": "string", "description": "Relative path within vault" }
    },
    "required": ["file_path"]
  },
  "annotations": { "readOnlyHint": true }
}

list_notes — List notes matching a filter (by folder, tag, type, or date range). Useful for exploration when the agent does not have a specific query.

{
  "name": "obsidian_list_notes",
  "description": "List notes matching filters",
  "inputSchema": {
    "type": "object",
    "properties": {
      "folder": { "type": "string", "description": "Folder path within vault" },
      "tag": { "type": "string", "description": "Tag to filter by" },
      "limit": { "type": "integer", "default": 20 }
    }
  },
  "annotations": { "readOnlyHint": true }
}

get_context — A convenience tool that runs a search and formats the results as a context block suitable for injection into a conversation.

{
  "name": "obsidian_get_context",
  "description": "Get formatted context from vault for a topic",
  "inputSchema": {
    "type": "object",
    "properties": {
      "topic": { "type": "string", "description": "Topic to get context for" },
      "max_tokens": { "type": "integer", "default": 2000 }
    },
    "required": ["topic"]
  },
  "annotations": { "readOnlyHint": true }
}

Permission Boundaries

The MCP server should enforce strict boundaries:

  1. Read-only. The server reads the vault and the index database. It does not create, modify, or delete notes. Write operations (capturing new notes) are handled by separate hooks or skills, not the MCP server.

  2. Vault-scoped. The server only reads files within the configured vault path. Path traversal attempts (../../etc/passwd) must be rejected.

  3. Credential-filtered output. Even if the database contains pre-filtered content, apply credential filtering on output as a defense-in-depth measure.

  4. Token-limited responses. Enforce max_tokens on all tool responses to prevent the AI tool from receiving excessively large context blocks.

Error Handling

MCP tools should return structured error messages that help the AI tool recover:

def search(self, query, limit=5, max_tokens=2000):
    if not self.db_path.exists():
        return {
            "error": "Index database not found. Run the indexer first.",
            "suggestion": "python index_vault.py --full"
        }

    results = self.retriever.search(query, limit, max_tokens)

    if not results:
        return {
            "results": [],
            "message": f"No results found for '{query}'. Try broader terms."
        }

    return {
        "results": [
            {
                "file_path": r["file_path"],
                "section": r["section"],
                "text": r["chunk_text"],
                "score": round(r["rrf_score"], 4),
            }
            for r in results
        ],
        "count": len(results),
        "query": query,
    }

Claude Code Integration

Claude Code is the primary consumer of the Obsidian retrieval system. This section covers MCP configuration, hook integration, and the obsidian_bridge.py pattern.

MCP Configuration

Register the custom server with claude mcp add (user scope writes ~/.claude.json; -s project writes a shareable .mcp.json — ~/.claude/settings.json is not an MCP config surface):

claude mcp add obsidian -s user \
  -e VAULT_PATH=/absolute/path/to/vault \
  -e DB_PATH=/absolute/path/to/vectors.db \
  -- python /path/to/obsidian_mcp.py

The equivalent .mcp.json entry, if you prefer to write it by hand:

{
  "mcpServers": {
    "obsidian": {
      "command": "python",
      "args": ["/path/to/obsidian_mcp.py"],
      "env": {
        "VAULT_PATH": "/absolute/path/to/vault",
        "DB_PATH": "/absolute/path/to/vectors.db"
      }
    }
  }
}

After adding the configuration, restart Claude Code. The MCP server will start as a child process. Verify it is running:

> What tools do you have from the obsidian MCP server?

Claude Code should list the available tools (obsidian_search, obsidian_read_note, etc.).

Hook Integration

Hooks extend Claude Code’s behavior at defined lifecycle points. Two hooks are relevant for Obsidian integration:

Hooks are registered in settings (~/.claude/settings.json, under the hooks key, with an event name and matcher) and receive a JSON payload on stdin — there is no hooks directory that Claude Code discovers automatically, and tool details never arrive as $1/$2 arguments.

UserPromptSubmit hook — queries the vault when you submit a prompt and injects relevant context (stdout from this event is added to the conversation):

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "hooks": [{ "type": "command", "command": "/path/to/obsidian-context.sh" }]
      }
    ]
  }
}
#!/bin/bash
# obsidian-context.sh — read the JSON payload from stdin
PROMPT=$(jq -r '.prompt // empty')
[ -z "$PROMPT" ] && exit 0
CONTEXT=$(python /path/to/retriever_cli.py search "$PROMPT" --limit 3 --max-tokens 1500)
if [ -n "$CONTEXT" ]; then
    printf 'Relevant vault context:\n%s\n' "$CONTEXT"   # stdout -> added as context
fi

PostToolUse hook — captures significant tool outputs back to the vault for future retrieval (registered with a matcher so it only fires for the tools you care about):

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [{ "type": "command", "command": "/path/to/capture-insight.sh" }]
      }
    ]
  }
}
#!/bin/bash
# capture-insight.sh — tool name and response arrive as stdin JSON
PAYLOAD=$(cat)
TOOL_NAME=$(printf '%s' "$PAYLOAD" | jq -r '.tool_name')
OUTPUT=$(printf '%s' "$PAYLOAD" | jq -r '.tool_response.stdout // ""')
if [ ${#OUTPUT} -gt 500 ]; then
    python /path/to/capture.py --text "$OUTPUT" --source "claude-code-$TOOL_NAME"
fi

tool_response is an object whose shape depends on the tool. For Bash the text is in .stdout, next to stderr, interrupted and isImage; an Edit or Write response carries file metadata, not output text. That is why this hook is registered on Bash and reads .tool_response.stdout. Stringifying the whole object gives you one line of escaped JSON, not the output.33

The obsidian_bridge.py Pattern

A bridge module provides a Python API that hooks and skills can call:

# obsidian_bridge.py
from retriever import HybridRetriever

_retriever = None

def get_retriever():
    global _retriever
    if _retriever is None:
        _retriever = HybridRetriever(
            db_path="/path/to/vectors.db",
            vault_path="/path/to/vault",
        )
    return _retriever

def search_vault(query, limit=5, max_tokens=2000):
    """Search vault and return formatted context."""
    retriever = get_retriever()
    results = retriever.search(query, limit, max_tokens)

    if not results:
        return ""

    lines = ["## Vault Context\n"]
    for r in results:
        lines.append(f"**{r['file_path']}** — {r['section']}")
        lines.append(f"> {r['chunk_text'][:500]}")
        lines.append("")

    return "\n".join(lines)

The /capture Skill

A Claude Code skill for capturing insights back to the vault:

/capture "OAuth token rotation requires both access and refresh token invalidation"
  --domain security
  --tags oauth,tokens

The skill creates a new note in 00-inbox/ with proper frontmatter and triggers an incremental reindex so the new note is immediately searchable.

Custom Command Patterns

Claude Code skills can wrap vault operations into named commands. Practitioners have built libraries of Obsidian-specific commands that treat the vault as both a read source and a write target.

Signal scanning. A /scan-intel command queries external sources, scores findings against personal research interests, and writes qualifying signals as vault notes with frontmatter:

/scan-intel --topics "agent infrastructure, security" --lookback 7d

The command fetches from configured sources (arXiv, HN, RSS), applies a scoring model (relevance, actionability, depth, authority), and writes passing signals to topic-specific vault folders. The vault becomes the downstream consumer of an automated intelligence pipeline.

Captain’s log. A /captains-log command aggregates daily git activity across all repositories, writes a structured journal entry to the vault, and includes decisions made, realizations, and open threads:

/captains-log

The command pulls commit history from GitHub, groups by repository, and formats as a narrative journal entry. Over time, the daily logs create a searchable record of what shipped and why.

Obsidian capture. A /obsidian-capture command takes an insight from the current Claude Code session and writes it directly to the vault with proper metadata:

/obsidian-capture "SAST gates in agent loops increase security degradation"
  --folder AI-Tools --tags security,agents

The pattern extends to any vault operation: creating MOCs, updating project status notes, linking related signals, or generating weekly digests from accumulated daily logs.

Community examples. Practitioners are publishing their command libraries. One developer shared 22 custom Obsidian + Claude Code commands covering daily planning and close-out, tracing how an idea has changed over time, surfacing connections and contradictions between notes, and promoting ideas from daily notes into permanent ones.58 The commands vary but the architecture is consistent: Claude Code skills as the interface, vault notes as the storage layer, and retrieval infrastructure as the query engine.

Context Window Management

The integration should be mindful of Claude Code’s context window:

  • Limit injected context to 1,500-2,000 tokens per query. More than this competes with the agent’s working memory.
  • Include source attribution. Always include the file path and section heading so the agent can reference the source.
  • Truncate chunk text. Long chunks should be truncated with ... rather than omitted entirely. The first 300-500 characters usually contain the key information.
  • Do not inject on every prompt. Injection runs through UserPromptSubmit (the event whose stdout reaches the model), so budget it there: skip short conversational prompts, inject only when the prompt names code, files, or past decisions, and cap the injected block. Tool-scoped events like PreToolUse and PostToolUse can gate, log, add a short block through hookSpecificOutput.additionalContext, or (PostToolUse only) replace the tool’s output through hookSpecificOutput.updatedToolOutput in their JSON output, but their plain stdout never reaches the model.33

Codex CLI Integration

Codex CLI connects to MCP servers through config.toml. The integration pattern differs from Claude Code in configuration syntax and instruction delivery.

MCP Configuration

Add to ~/.codex/config.toml (user scope), or to a trusted project’s .codex/config.toml to share the server with the repo; Codex loads project-scoped config only after you trust the project. codex mcp add obsidian --env VAULT_PATH=/absolute/path/to/vault --env DB_PATH=/absolute/path/to/vectors.db -- python /path/to/obsidian_mcp.py writes the same block for you. Standing instructions still belong in AGENTS.md:35

[mcp_servers.obsidian]
command = "python"
args = ["/path/to/obsidian_mcp.py"]

[mcp_servers.obsidian.env]
VAULT_PATH = "/absolute/path/to/vault"
DB_PATH = "/absolute/path/to/vectors.db"

AGENTS.md Patterns

Codex CLI reads AGENTS.md for project-level instructions. Include vault search guidance:

## Available Tools

### Obsidian Vault (MCP: obsidian)
Use the `obsidian_search` tool to find relevant context from the knowledge base.
Search the vault when you need:

- Background on a concept or pattern
- Prior decisions or rationale
- Reference material for implementation

Example queries:

- "authentication patterns in FastAPI"
- "how does the review aggregator work"
- "sqlite-vec configuration"

Differences from Claude Code

Feature Claude Code Codex CLI
MCP config claude mcp add → ~/.claude.json / project .mcp.json codex mcp add → ~/.codex/config.toml, or a trusted project’s .codex/config.toml35
Hooks Settings-registered, 33 lifecycle events, stdin JSON33 Supported (stable surface; its own event set)
Skills ~/.claude/skills/ ~/.agents/skills/ (user) and .agents/skills/ in the repo; /etc/codex/skills/ for admins36
Instruction file CLAUDE.md AGENTS.md
Permission surface Modes: Manual (default) / acceptEdits / plan / auto (the starting mode for interactive terminal and VS Code sessions on every plan since v2.1.283; claude -p and the Agent SDK start in Manual unless --permission-mode auto is passed) / dontAsk / bypassPermissions34 Approval policy on-request / never (plus granular rules; untrusted is no longer supported and on-failure is deprecated) × sandbox read-only / workspace-write / danger-full-access (codex exec --full-auto removed in v0.147.0)35

Key difference: both tools now support hooks and skills; what differs is shape. Codex pairs an approval policy with an OS-level sandbox mode instead of Claude Code’s permission modes, and its hook events differ — port the pattern (query the vault before work, capture after), not the config. AGENTS.md remains the right place for standing search-the-vault-first instructions in Codex.


Cursor and Other Tools

Cursor and other AI tools that support MCP can connect to the same Obsidian MCP server. This section covers configuration for common tools.

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "obsidian": {
      "command": "python",
      "args": ["/path/to/obsidian_mcp.py"],
      "env": {
        "VAULT_PATH": "/absolute/path/to/vault",
        "DB_PATH": "/absolute/path/to/vectors.db"
      }
    }
  }
}

Put the standing instruction in AGENTS.md at the project root (Cursor and Codex read it, and Claude Code reads it in a project that has no CLAUDE.md), or in a project rule at .cursor/rules/obsidian.mdc with alwaysApply: true in its frontmatter. The single .cursorrules file in the project root is legacy and will be deprecated:37

---
alwaysApply: true
---
When working on implementation tasks, search the Obsidian vault
for relevant context before writing code. Use the obsidian_search
tool with descriptive queries about the concept you're implementing.

Compatibility Matrix

Tool MCP Support Transport Config Location
Claude Code Full STDIO + HTTP (WebSocket through a JSON entry; SSE deprecated)40 claude mcp add → ~/.claude.json / project .mcp.json
Codex CLI Full STDIO + HTTP ~/.codex/config.toml or a trusted project’s .codex/config.toml35
Cursor Full STDIO + SSE + Streamable HTTP50 .cursor/mcp.json (project) or ~/.cursor/mcp.json (global)
Devin Local in Devin Desktop (Windsurf’s MCP docs now redirect here) Full STDIO + Streamable HTTP + SSE .devin/mcp_config.json (project), .devin/mcp_config.local.json (local override), ~/.config/devin/mcp_config.json (user)38
Continue.dev Full STDIO + HTTP config.yaml (mcpServers block) or a block file in the workspace’s .continue/mcpServers/39
Zed Full (context servers) STDIO + remote (url entries)50 settings.json (context_servers)
Claudian (Obsidian plugin) N/A (embedded) Agent CLIs (Claude Code, Codex, others) Obsidian plugin settings
Agent Client (Obsidian plugin) N/A (embedded) ACP Obsidian plugin settings

Fallback for Non-MCP Tools

For tools that do not support MCP, the retriever can be wrapped as a CLI:

# Search from command line
python retriever_cli.py search "query text" --limit 5 --max-tokens 2000

# Output formatted for copy-paste into any tool
python retriever_cli.py context "query text" --format markdown

The CLI outputs structured text that can be manually pasted into any AI tool’s input. This is less elegant than MCP integration but works universally.


Prompt Caching from Structured Notes

Structured notes in the vault can serve as reusable context blocks that reduce token usage across AI interactions. This section covers cache key design and token budget management.

The Pattern

Instead of searching for context on every interaction, pre-build context blocks from well-structured vault notes and cache them:

# cache_keys.py
CONTEXT_BLOCKS = {
    "auth-patterns": {
        "vault_query": "authentication patterns implementation",
        "max_tokens": 1500,
        "ttl_hours": 24,  # Rebuild daily
    },
    "api-conventions": {
        "vault_query": "API design conventions REST patterns",
        "max_tokens": 1000,
        "ttl_hours": 168,  # Rebuild weekly
    },
    "project-architecture": {
        "vault_query": "current project architecture decisions",
        "max_tokens": 2000,
        "ttl_hours": 12,  # Rebuild twice daily
    },
}

Cache Invalidation

Cache invalidation is based on two signals:

  1. TTL expiry. Each context block has a time-to-live. When the TTL expires, the block is rebuilt by re-querying the vault.
  2. Vault change detection. When the indexer detects changes to files that contributed to a cached context block, the block is invalidated immediately.

Token Budget Management

A session starts with a total context budget. Cached blocks consume part of that budget:

Total context budget:    9,500 tokens
├─ System prompt:        1,500 tokens
├─ Cached blocks:        4,500 tokens (pre-loaded)
├─ Dynamic search:       2,000 tokens (on-demand)
└─ Conversation:         1,500 tokens (remaining)

The cached blocks load at session start. Dynamic search results fill the remaining budget on a per-query basis. This hybrid approach gives the agent a baseline of frequently-needed context while preserving budget for specific queries.

Before/After Token Usage

Without caching: Every relevant query triggers a vault search, returning 1,500-2,000 tokens of context. Over 10 queries in a session, the agent consumes 15,000-20,000 tokens of vault context.

With caching: The three pre-built context blocks above consume up to 4,500 tokens total (1,500 + 1,000 + 2,000). Additional searches add 1,500-2,000 tokens per unique query. Over 10 queries where 6 are covered by cached blocks, the agent consumes 4,500 + (4 * 1,500) = 10,500 tokens, against 15,000 uncached at the same 1,500 per search: 30% less.


Capturing Compressed Summaries of Verbose Outputs

Tool outputs can be verbose: stack traces, file listings, test results. A PostToolUse hook can act on that in two ways. Across sessions, it can write a compressed summary to the vault, so future sessions retrieve the two-line verdict instead of re-running or re-reading the 5,000-token original. Within the session, it can replace what the model sees: the hook’s plain stdout replaces nothing, but its JSON output can carry hookSpecificOutput.updatedToolOutput, and Claude Code swaps the tool’s output for that value before Claude reads it (every tool since Claude Code 2.1.121; MCP tools only before that).33

The replacement value must match the tool’s output shape, which for Bash is an object with stdout, stderr, interrupted and isImage. A value that does not match is ignored and the original output is used. Replacing output hides detail from the model, so keep the failing lines. The other in-session levers are unchanged: /compact, scoped prompts, and asking for less verbose output.33

The Problem

A Bash tool call that runs tests might return:

PASSED tests/test_auth.py::test_login_success
PASSED tests/test_auth.py::test_login_failure
PASSED tests/test_auth.py::test_token_refresh
PASSED tests/test_auth.py::test_session_expiry
... (200 more lines)
FAILED tests/test_api.py::test_rate_limit_exceeded

The full output is 5,000 tokens, but the signal is in 2 lines: 200 passed, 1 failed.

Hook Implementation

Registered on PostToolUse with a Bash matcher (settings-registered, stdin JSON, as in Hook Integration above). The Bash output text is the stdout field of tool_response:

#!/bin/bash
# summarize-and-capture.sh — write a compressed summary to the vault
PAYLOAD=$(cat)
OUTPUT=$(printf '%s' "$PAYLOAD" | jq -r '.tool_response.stdout // ""')

# Only summarize large outputs
[ ${#OUTPUT} -lt 2000 ] && exit 0

if printf '%s' "$OUTPUT" | grep -q "PASSED\|FAILED"; then
    PASSED=$(printf '%s' "$OUTPUT" | grep -c "PASSED")
    FAILED=$(printf '%s' "$OUTPUT" | grep -c "FAILED")
    SUMMARY="Tests: $PASSED passed, $FAILED failed"
    [ "$FAILED" -gt 0 ] && SUMMARY="$SUMMARY
$(printf '%s' "$OUTPUT" | grep 'FAILED')"
    python /path/to/capture.py --text "$SUMMARY" --source "test-run-summary" >/dev/null
fi
exit 0

To shrink the output inside the session as well, have the hook print the replacement as its only stdout, directly after the capture.py line. Merging into the original tool_response keeps the object in the Bash shape and swaps only stdout:

    printf '%s' "$PAYLOAD" | jq --arg out "$SUMMARY" '{
      hookSpecificOutput: {
        hookEventName: "PostToolUse",
        updatedToolOutput: (.tool_response + {stdout: $out})
      }
    }'

Each hook invocation is a fresh process, so no recursion guard is needed — a hook’s own writes don’t re-fire it, and exported variables never survive into the next invocation.

Compression Heuristics

Output Type Detection Compression Strategy
Test results PASSED / FAILED keywords Count pass/fail, show failures only
File listings ls or find in command Truncate to first 20 entries + count
Stack traces Traceback keyword Keep first and last frame + error message
Git status modified: / new file: Summarize counts by status
Build output warning: / error: Strip info lines, keep warnings/errors

Signal Intake and Triage Pipeline

The intake layer determines what enters the vault. Without curation, the vault accumulates noise. This section covers the scoring pipeline that routes signals to domain folders.

Sources

Signals come from multiple channels:

  • RSS feeds: Technical blogs, security advisories, release notes
  • Bookmarks via Web Clipper: The official Obsidian Web Clipper extension (Chrome, Firefox, Safari) is the highest-fidelity intake path for browser-side capture. The April to July 2026 release cycle made it materially more useful for AI workflows:20
    • 1.4.0 (Apr 9): Interactive YouTube transcript UI — pin the video, scrub through the transcript, auto-scroll, and highlight the current position. Plus an “Open in Reader” default that sends a one-click capture straight to Reader mode.
    • 1.5.0–1.5.1 (Apr 15): Highlights viewer — browse and search captured highlights across the vault. Fade-in transition into Reader. Smoother YouTube play/pause. 1.5.1 fixed a webpack compilation regression.
    • 1.6.0–1.6.2 (Apr 21–23): Highlighter UX overhaul with mobile support. Defuddle 0.18 adds source-specific extractors for LinkedIn, Threads, Bluesky, Discourse, and Medium. 1.6.2 fixes a Safari embedded-mode clipboard regression.
    • 1.7.0 (Jun 16) and 1.7.1 (Jul 22, current): {{content}} keeps ==highlight== markers, highlights persist between the live page and Reader, highlight import, Defuddle 0.19.x, and {{model}}, {{modelId}} and {{modelProvider}} template variables for the Interpreter, so a clipped note can record which model summarized it. Configure templates per source domain so YouTube transcripts, GitHub READMEs, and longform articles each land in a sensibly-named note with the right frontmatter for the scoring pipeline below.
  • Newsletters: Key excerpts from email newsletters
  • Manual capture: Notes written during reading, conversations, or research
  • Tool output: Significant AI tool outputs captured via hooks
  • iOS Share Sheet and Quick Capture: Obsidian’s iOS app saves content shared from Safari and other apps into the vault. Mobile 1.13 (public July 30, 2026) brought a new Share Sheet with configurable Locations, and its template has a url variable that inserts the shared link into the note. On iOS 26, Quick Capture (mobile 1.14) creates a note from the Lock Screen, Control Center, a Home Screen widget or Shortcuts without waiting for the vault to load, and mobile 1.15 adds voice notes transcribed as you speak, plus Spotlight and Siri actions.17 Together these make a low-friction mobile intake path: share an article from Safari, or dictate a thought, and it arrives as a vault note ready for scoring.
  • Obsidian CLI: Shell scripts and hooks can create notes with obsidian create path="00-inbox/<name>.md" content="..." (add template=<name> to start from a template) or append to existing notes with obsidian append file=<name> content="...", enabling automated intake pipelines on desktop.57

Scoring Dimensions

Each signal is scored on four dimensions (0.0 to 1.0 each):

Dimension Question Low Score (0.0-0.3) High Score (0.7-1.0)
Relevance Does this relate to my active domains? Tangential, outside scope Directly relevant to active work
Actionability Can I use this information? Pure theory, no application Specific technique or pattern I can apply
Depth How substantive is the content? Headlines, shallow summary Detailed analysis with examples
Authority How credible is the source? Anonymous blog, unverified Primary source, peer-reviewed, recognized expert

Composite Score and Routing

composite = (relevance * 0.35) + (actionability * 0.25) +
            (depth * 0.25) + (authority * 0.15)
Score Range Action
0.55+ Auto-route to domain folder
0.40 - 0.55 Queue for manual review
< 0.40 Drop (do not store)

Domain Routing

Signals scoring above 0.55 route to one of 12 domain folders based on keyword matching and topic classification:

05-signals/
├── ai-tooling/        # Claude, LLMs, AI development tools
├── security/          # Vulnerabilities, auth, cryptography
├── systems/           # Architecture, distributed systems
├── programming/       # Languages, patterns, algorithms
├── web/               # Frontend, backends, APIs
├── data/              # Databases, data engineering
├── devops/            # CI/CD, containers, infrastructure
├── design/            # UI/UX, product design
├── mobile/            # iOS, Android, cross-platform
├── career/            # Industry trends, hiring, growth
├── research/          # Academic papers, whitepapers
└── other/             # Signals that don't fit a domain

Production Stats

Over 14 months of operation:

Metric Value
Total signals processed 7,771
Auto-routed (>0.55) 4,832 (62%)
Queued for review (0.40-0.55) 1,543 (20%)
Dropped (<0.40) 1,396 (18%)
Active domain folders 12
Average signals per day ~18

Knowledge Graph Patterns

Obsidian’s wiki-link graph encodes relationships between notes. This section covers link semantics, graph traversal for context expansion, and anti-patterns that degrade graph quality.

Every wiki-link creates a directed edge in the graph. Obsidian tracks both forward links and backlinks:

  • Forward link: Note A contains [[Note B]] → A links to B
  • Backlink: Note B shows that Note A references it

The graph encodes different types of relationships depending on context:

Link Pattern Semantic Example
Inline link “Is related to” “See [[OAuth Token Rotation]] for details”
Header link “Has subtopic” ”## Related\n- [[Token Rotation]]\n- [[Session Management]]”
Tag-like link “Is categorized as” ”[[type/reference]]”
MOC link “Is part of” A Map of Content note listing related notes

Maps of Content (MOCs)

MOCs are index notes that organize related notes into a navigable structure:

---
title: "Authentication & Security MOC"
type: moc
domain: security
---

## Core Concepts

- [[OAuth 2.0 Overview]]
- [[JWT Token Anatomy]]
- [[Session Management Patterns]]

## Implementation Patterns

- [[OAuth Token Rotation]]
- [[Refresh Token Security]]
- [[PKCE Flow Implementation]]

## Failure Modes

- [[Token Expiry Handling]]
- [[Session Fixation Prevention]]
- [[CSRF Defense Strategies]]

MOCs benefit retrieval in two ways:

  1. Direct match. A search for “authentication overview” matches the MOC itself, providing the agent with a curated list of related notes.
  2. Context expansion. After finding a specific note, the retriever can check if the note appears in any MOCs and include the MOC’s structure in the results, giving the agent a map of the broader topic.

Graph Traversal for Context Expansion

A future enhancement to the retriever: after finding top results, expand the context by following links:

def expand_context(results, depth=1):
    """Follow wiki-links from top results to find related context."""
    expanded = set()
    expanded_results = []
    for result in results:
        # Parse wiki-links from chunk text
        links = extract_wiki_links(result["chunk_text"])
        for link_target in links:
            # Resolve link to file path
            target_path = resolve_wiki_link(link_target)
            if target_path and target_path not in expanded:
                expanded.add(target_path)
                # Include one chunk per linked note (the first here;
                # rank them against the query if you have a scorer)
                target_chunks = get_chunks_for_file(target_path)
                if target_chunks:
                    expanded_results.append(target_chunks[0])
    return results + expanded_results

This is not implemented in the current retriever but represents a natural extension of the graph structure. extract_wiki_links, resolve_wiki_link and get_chunks_for_file are the three helpers you would write: a regex over [[...]], a lookup from link text to file path, and a SELECT on chunks by file_path.

Anti-Patterns

Orphan clusters. Groups of notes that link to each other but have no connections to the rest of the vault. The graph panel in Obsidian makes these visible as disconnected islands. Orphan clusters indicate missing MOCs or missing cross-domain links.

Tag sprawl. Using tags inconsistently or creating too many fine-grained tags. A vault with 2,500 unique tags across 5,000 notes averages two notes per tag, so the tags are not useful for filtering. Consolidate to 20-50 high-level tags that map to your domain folders.

Link-heavy, content-light notes. Notes that consist entirely of wiki-links with no prose. These notes index poorly because the chunker has no text to embed. Add at least a paragraph of context explaining why the linked notes are related.

Bidirectional links for everything. Not every reference needs to be a wiki-link. Mentioning “OAuth” in passing does not require [[OAuth 2.0 Overview]]. Reserve wiki-links for intentional, navigable relationships where clicking the link would provide useful context.


Developer Workflow Recipes

Practical workflows that combine vault retrieval with daily development tasks.

Morning Context Load

Start the day by loading relevant context:

Search my vault for notes about [current project] updated in the last week

The retriever returns recent notes about your active project, giving you a quick refresher on where you left off. More effective than re-reading yesterday’s commit messages.

Research Capture During Coding

While implementing a feature, capture insights without leaving the editor:

/capture "FastAPI dependency injection with async generators requires yield,
not return. The generator is the dependency lifecycle."
  --domain programming
  --tags fastapi,dependency-injection

The captured insight is immediately indexed and available for future retrieval. Over months, these micro-captures build a corpus of implementation-specific knowledge.

Project Kickoff

When starting a new project or feature:

  1. Search the vault: “What do I know about [technology/pattern]?”
  2. Review the top 5 results for prior decisions and gotchas
  3. Check if a MOC exists for the domain; if not, create one
  4. Search for failure modes: “problems with [technology]”

When encountering an error or unexpected behavior:

Search my vault for [error message or symptom]

Prior debugging notes often contain the root cause and fix. This is particularly valuable for recurring issues across projects — the vault remembers what you forget.

Code Review Preparation

Before reviewing a PR:

Search my vault for patterns and conventions about [module being changed]

The vault returns prior decisions, architectural constraints, and coding standards relevant to the code under review. The review is informed by institutional knowledge, not just the diff.


Performance Tuning

This section covers optimization strategies for different vault sizes and usage patterns.

Index Size Management

Vault Size Chunks DB Size Full Reindex Incremental
500 notes ~1,500 3 MB 15 seconds <1 second
2,000 notes ~6,000 12 MB 45 seconds 2 seconds
5,000 notes ~15,000 30 MB 2 minutes 4 seconds
15,000 notes ~50,000 83 MB 4 minutes <10 seconds
50,000 notes ~150,000 250 MB 15 minutes 30 seconds

At 50,000+ notes, consider:

  • Increasing the batch size from 64 to 128 for faster embedding
  • Using WAL mode (set once with PRAGMA journal_mode=WAL; SQLite’s default is DELETE mode) for concurrent access42
  • Running full reindex during off-hours

Query Optimization

WAL mode. SQLite’s Write-Ahead Logging mode enables concurrent reads while the indexer writes:

db.execute("PRAGMA journal_mode=WAL")

This is critical when the MCP server handles queries while the indexer runs an incremental update.

Connection pooling. The MCP server should reuse database connections rather than opening a new connection per query. A single long-lived connection with WAL mode supports concurrent reads.

# MCP server initialization
db = sqlite3.connect(DB_PATH, check_same_thread=False)
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA mmap_size=268435456")  # 256 MB mmap

Memory-mapped I/O. The mmap_size pragma tells SQLite to use memory-mapped I/O for the database file. For an 83 MB database, mapping the entire file into memory eliminates most disk reads.

FTS5 optimization. After a full reindex, run:

INSERT INTO chunks_fts(chunks_fts) VALUES('optimize');

This merges FTS5’s internal b-tree segments, reducing query latency for subsequent searches.

Scaling Benchmarks

Measured on Apple M3 Pro, 36 GB RAM, NVMe SSD:

Operation 500 notes 5K notes 15K notes 50K notes
BM25 query 2ms 5ms 12ms 25ms
Vector query 1ms 3ms 8ms 20ms
RRF fusion <1ms <1ms 3ms 5ms
Full search 3ms 8ms 23ms 50ms

All benchmarks include database access, query execution, and result formatting. Network latency for MCP STDIO communication adds 1-2ms.


Troubleshooting

Index Drift

Symptom: Search returns stale results or misses recently added notes.

Cause: The incremental indexer did not run after adding notes, or a file’s mtime was not updated (e.g., synced from another machine with preserved timestamps).

Fix: Run a full reindex: python index_vault.py --full

Embedding Model Swap

Symptom: After changing the embedding model, vector search returns nonsensical results.

Cause: Old vectors (from the previous model) are being compared against new query vectors. The dimensions or vector space semantics are incompatible.

Fix: The indexer should detect the model hash mismatch and trigger a full reindex automatically. If it does not, manually clear the database and reindex:

# Stop the MCP server (and anything else holding the database) first
rm -f vectors.db vectors.db-wal vectors.db-shm
python index_vault.py --full

Restart the MCP server afterwards. In WAL mode the database has two companion files, vectors.db-wal and vectors.db-shm, and a server that still has the old file open cannot use the rebuilt one: its queries fail until it restarts.42

FTS5 Maintenance

Symptom: FTS5 queries return incorrect or incomplete results after many incremental updates.

Cause: The FTS5 index is out of sync with chunks: rows were removed or changed without the matching FTS5 'delete' (see Handling Deletions). Fragmented segments after many small updates slow queries down but do not change results.

Fix: Rebuild and optimize:

INSERT INTO chunks_fts(chunks_fts) VALUES('rebuild');
INSERT INTO chunks_fts(chunks_fts) VALUES('optimize');

MCP Timeout

Symptom: AI tool reports that the MCP server timed out.

Cause: The first query triggers model loading (lazy initialization), which takes 2-5 seconds. The AI tool’s default MCP timeout may be shorter.

Fix: Pre-warm the model on server startup:

# In MCP server initialization
retriever = HybridRetriever(db_path, vault_path)
retriever.search("warmup", limit=1)  # Trigger model load

In Claude Code, MCP_TIMEOUT=15000 claude raises the server startup timeout to 15 seconds while you keep lazy loading, and a per-server "timeout" key (milliseconds) in the server’s .mcp.json entry bounds each tool call.40

SQLite File Locks

Symptom: SQLITE_BUSY or SQLITE_LOCKED errors.

Cause: Multiple processes writing to the database simultaneously. WAL mode allows concurrent reads but only one writer.

Fix: Ensure only one process (the indexer) writes to the database. The MCP server and hooks should only read. If you need concurrent writes, use WAL mode and set a busy timeout:

db.execute("PRAGMA busy_timeout=5000")  # Wait up to 5 seconds

sqlite-vec Not Loading

Symptom: Vector search is disabled; retriever runs in BM25-only mode.

Cause: The sqlite-vec extension is not installed, incompatible with the SQLite version, or loaded with db.load_extension("vec0") instead of sqlite_vec.load(db); the pip package keeps the binary inside site-packages/sqlite_vec/, where the dynamic loader does not look, so the bare name fails with “no such file”.

Fix:

# Install via pip
pip install sqlite-vec

# Or compile from source
git clone https://github.com/asg017/sqlite-vec
cd sqlite-vec && ./scripts/vendor.sh && make loadable
# builds dist/vec0.dylib (macOS), dist/vec0.so (Linux) or dist/vec0.dll (Windows);
# load it with db.load_extension("/path/to/sqlite-vec/dist/vec0")

Verify the extension loads:

import sqlite3
import sqlite_vec

db = sqlite3.connect(":memory:")
db.enable_load_extension(True)
sqlite_vec.load(db)
db.enable_load_extension(False)
print("sqlite-vec", db.execute("select vec_version()").fetchone()[0])  # e.g. v0.1.9

Once the extension loads, run python index_vault.py --full: files indexed or removed while it was missing left the vector table out of step with chunks (see Handling Deletions).

Large Vault Memory Issues

Symptom: Out-of-memory errors during full reindex of a large vault (50,000+ notes).

Cause: Embedding batch size is too large, or all file contents are loaded into memory simultaneously.

Fix: Reduce the batch size and process files incrementally:

BATCH_SIZE = 32  # Reduce from 64

Also ensure the indexer processes files one at a time (reading, chunking, and embedding each file before moving to the next) rather than loading all files into memory.


Migration Guide

From Apple Notes

  1. On a Mac, install and enable Obsidian’s official Importer plugin (Settings → Community plugins), then open it from the command palette or the ribbon icon
  2. Under File format choose Apple Notes, select Import, and select Open when macOS asks for access to the group.com.apple.notes folder
  3. Review the generated template and the preview (this is where you set the note name and properties), then select Import again
  4. Add any missing frontmatter (type, domain, tags) and move the notes to appropriate domain folders

Notes on macOS exports one note at a time (File > Export To), so the Importer is the practical route for a whole library. It converts to Markdown itself, skips locked notes, and runs on macOS only.51

From Notion

  1. Install and enable Obsidian’s official Importer plugin. The recommended route is Notion (API) under File format: create an access token in Notion’s Connections dashboard, give that connection access to the pages and databases you want, and load them in the Importer. This route needs an internet connection and converts databases and formulas to Bases
  2. For a file import, choose Notion (.zip). Export from Notion with Settings → Workspace → General → Export all workspace content, in HTML format with folders for subpages, and give the ZIP to the Importer. This route does not preserve databases, and Obsidian advises against Notion’s Markdown export because it omits data51
  3. Import the whole workspace in one pass so internal links resolve, into a folder such as 00-inbox/
  4. Add standard frontmatter (type, domain, tags) where it is missing, then move pages to domain folders

From Google Docs

  1. Use Google Takeout to export all documents
  2. Convert .docx files to markdown: pandoc -f docx -t markdown input.docx -o output.md
  3. Batch convert: for f in *.docx; do pandoc -f docx -t markdown "$f" -o "${f%.docx}.md"; done
  4. Move to vault, add frontmatter, organize into folders

From Plain Markdown (No Obsidian)

If you already have a directory of markdown files:

  1. Open the directory as an Obsidian vault (Obsidian → Open Vault → Open folder)
  2. Add .obsidian/ to .gitignore if the directory is version-controlled
  3. Create frontmatter templates and apply to existing files
  4. Start linking notes with [[wiki-links]] as you read and organize
  5. Run the indexer immediately — the retrieval system works on day one

From Another Retrieval System

If you are migrating from a different embedding/search system:

  1. Do not try to migrate vectors. Different models produce incompatible vector spaces. Run a full reindex with the new model.
  2. Migrate the content, not the index. The vault files are the source of truth. The index is a derived artifact.
  3. Verify after migration. Run 10-20 queries you know the answers to and verify the results match your expectations.

Changelog

Date Change Source
2026-11-11 If you use the Local REST API with MCP plugin, run 5.4.0 or later. Three high-severity advisories cover earlier builds of the plugin behind mcp-obsidian and obsidian-mcp-server: host file access through /vault/{path}, writes outside the vault through vault_write, and code execution through the configuration folder. The setup note and server table now give the affected and fixed versions. Also corrected: the OpenAI re-embed estimate (about $0.04, not $0.30), Obsidian’s setting names, the RRF result (one TREC exception), database growth for larger models, and the model-hash, hook and database-reset examples. obsidian-mcp-server is at 3.9.0. 54 1 2 26
2026-11-10 Obsidian 1.15.0 is the public desktop release (November 2, 2026; 1.15.1 in Catalyst); the CLI section uses the documented parameter=value grammar. Code corrected and re-run end to end: BM25 queries go through fts_query (raw text bound to MATCH raised on --incremental and AGENTS.md); KNN pagination asks for distance > the cursor; remove_file issues the FTS5 'delete' rows; the BM25-only fallback returns the same fields as hybrid; hooks read tool_response.stdout; an HTTP MCP entry carries "type": "http"; quantize_to="int8", sqlite_vec.load(db) and distance_metric=cosine. Facts corrected: a PostToolUse hook can replace tool output (updatedToolOutput, Claude Code 2.1.121 and later); sqlite-vec DELETE always worked and 0.1.7 made it reclaim space; MCPVault 0.17.0 has a symlink-write advisory, fixed in 0.17.1 (current: 0.18.3); the plugin table now lists registry names (Hybrid Search, Drift, EngramQuest); Apple Notes and Notion migrate through the Importer; Claude Code, Cursor and Zed all reach remote MCP servers. 12 24 29 31 33 40 42 45 51
2026-08-14 Quick start corrected to npm obsidian-mcp (StevenStavrakis; obsidian-mcp-server is cyanheads’ REST-API server), run as npx -y obsidian-mcp@2 serve in the Claude Code, Codex and Cursor blocks. Claude Code MCP configuration moved from ~/.claude/settings.json (which Claude Code ignores for MCP) to claude mcp add and .mcp.json. Hook examples rewritten as settings-registered hooks that read stdin JSON, with context injection on UserPromptSubmit, the event whose stdout reaches the model; the output-compression section recast as capture-to-vault, since a hook cannot shrink what the model has already seen. Obsidian bumped to 1.13.7; the MCP 2026-07-28 note past-tensed; the Codex comparison rebuilt around approval policy plus sandbox mode; compatibility matrix refreshed; MCPVault 0.15.0; the 1.13 Share Sheet url variable folded into the intake section. 22 24
2026-08-07 Obsidian 1.13 reached the public channel: 1.13.4 promoted July 30, 2026 (per desktop-releases.json: latestVersion 1.13.4, beta.latestVersion 1.13.6). The three body references to “public channel remains 1.12.7” are updated. What the 1.13 line ships, now that it is generally available: a settings overhaul (separate window, search by name/description, keyboard + Vim navigation), a full-screen image viewer with per-file navigation and Live Preview resize controls, searchable Bookmarks, Sync multi-selection, and, most relevant to this guide’s audience, Obsidian URI security: obsidian:// actions now require a confirmation dialog unless allow-listed. This guide’s automation story runs through MCP and the CLI, which are unaffected, but any workflow that drives Obsidian via URIs (Shortcuts, scripts, other launchers) now needs its actions allow-listed once or it will prompt every time. Developer-facing: a new Settings API with migration guide, a breaking --callout-color change (requires valid CSS colors, no longer RGB triplets; themes and snippets must update), Electron 43.1.1, CodeMirror and Mermaid 11.13.0 upgrades. No AI, MCP, or CLI changes. Catalyst early access is at 1.13.6. 24
2026-07-27 Version update and an advisory correction. MCPVault moved 0.12.1 → 0.12.4 on July 23 (0.12.2 and 0.12.4 reached npm; 0.12.3 is a changelog entry that was not published on its own). The 0.12.3 entry adds a wiki_link tool (resolves [[Document Name]], [[Name\|Display]], table-escaped and #fragment forms, returning content plus the resolved path and ambiguous alternatives) and excludes .trash/ from every tool through the default path filter; v0.12.4 extends it to path-qualified [[folder/Note]] links; v0.12.2 fixes patch_note corrupting $-pattern insertions, normalizes vault-prefixed paths, and clears npm audit high-severity findings. The MCP-server section and [^24] now describe all three. Correction: the 2026-07-07 row said v0.12.1 was “carrying two medium-severity path-filter advisories.” It was not. The GitHub Advisory API gives GHSA-9c83-rr99-vfwj a vulnerable range of < 0.11.5 and GHSA-j99q-93c9-h869 < 0.11.4: both patched before the 0.12 line opened, so 0.12.1 was already clear when that row was written. The body and footnote now state the first-patched versions instead of leaving “run a current release” to imply an open exposure. Also: Obsidian 1.13.4 desktop + mobile (July 27) is Catalyst early access; the desktop-releases.json manifest still reports latestVersion 1.12.7 with beta.latestVersion 1.13.4, so the public channel is unmoved and version references here stand. Contents are UX-grade (lightbox filename display, Live Preview image alignment and padding, a stuck-file save race, settings layout) with no AI, MCP, or CLI change. 12 24
2026-07-22 Version update; no workflow changes. Obsidian 1.13.3 desktop + mobile (July 21) is Catalyst early access only; per desktop-releases.json the public channel stays 1.12.7; contents are UX-grade (image-embed lightbox/zoom, Live Preview line-height fix, File Recovery arrow keys, unique URI paneType fix), no AI/MCP/CLI changes. Note: 1.13.3 was published later on July 21, so the prior row’s “no new releases” was accurate when written. Web Clipper 1.7.1 (July 22, GitHub release): Import highlights, {{model}}/{{modelId}}/{{modelProvider}} Interpreter template variables, refreshed provider presets, Defuddle 0.19.2, Interpreter fixes for recent Anthropic models, native Gemini API keys, DeepSeek and Azure OpenAI handling; store rollouts may lag the GitHub date. No official MCP server news; stateless spec still due July 28. 24
2026-07-21 Accuracy correction: public desktop channel is 1.12.7, not 1.13.1. The 2026-06-10 entry (and body references since) treated 1.13.1 as a public-channel release; the 1.13.1 changelog page is badged Early access, and the official auto-update manifest (obsidianmd/obsidian-releases, desktop-releases.json) lists public latestVersion 1.12.7 with 1.13.2 on the beta channel; the entire 1.13.x line is Catalyst-only. Body references and 24 corrected. No new releases between 2026-07-17 and 2026-07-21: core stays 1.13.2 Catalyst, Clipper 1.7.0, no official MCP server news, MCP stateless spec still due July 28. 24
2026-07-17 Version update; no workflow changes. Obsidian 1.13.2 (July 14) is Catalyst early-access only; the public channel remains 1.13.1, so version references here stay current; its one guide-adjacent item is a new url variable in the iOS Share Sheet template (inserts the shared link into the note), which will fold into the capture-path section when 1.13.2 goes public. Web Clipper 1.7.0 (June 16, previously uncovered): Defuddle 0.19.0 upgrade, {{content}} now preserves ==highlight== markers, and highlights persist between the live page and Reader view. No official Obsidian MCP server or AI integration has been announced; the MCP spec’s stateless release remains scheduled for July 28. Verified against obsidian.md/changelog, github.com/obsidianmd/obsidian-clipper releases, and blog.modelcontextprotocol.io.
2026-07-07 Accuracy corrections. MCPVault clarified as its own project (npm @bitbonsai/mcpvault, repo bitbonsai/mcpvault), now v0.12.1, carrying two medium-severity path-filter advisories (GHSA-9c83-rr99-vfwj, GHSA-j99q-93c9-h869); the earlier [^24] link pointed at the wrong repo (MarkusPfundstein/mcp-obsidian). Corrected MarkusPfundstein/mcp-obsidian’s status: it is actively maintained (commits through May 15, 2026, adding search_by_tag/get_frontmatter), not “dormant since June 2025”; it still ships no tagged releases. Verified against GitHub commit history, GitHub Security Advisories, and npm.
2026-07-06 Editorial restructure for findability: retitled “Quick Start: First AI-Connected Vault” to Obsidian MCP Setup (anchor #obsidian-mcp-setup) and added a “What Claude can do once connected” capability summary (search, read, list, formatted context; read-only boundary with writes handled by hooks) consolidated from the MCP Server Architecture section. No new facts; internal links updated.
2026-06-10 Version-currency bump. Obsidian 1.13.1 desktop reached the public channel (June 9, 2026) — a settings-UX + CodeMirror upgrade over 1.13.0, with no major AI/automation change. Current-version body references moved from 1.13.0 to 1.13.1 (public, June 9 2026). 24
2026-06-09 Ecosystem refresh. MCP 2026-07-28 specification entered Release Candidate (announced May 21, 2026) — the largest MCP revision since launch: stateless protocol core (removes the initialize handshake and Mcp-Session-Id), MCP Apps (server-rendered HTML in sandboxed iframes), Tasks graduating from experimental core to an official extension, OAuth 2.0/OIDC hardening, and a 12-month deprecation-lifecycle policy (final spec July 28, 2026); replaced the speculative “tentatively mid-2026” roadmap framing in the MCP Spec Evolution note with the concrete RC. sqlite-vec v0.1.10-alpha (Mar 31 – May 18, 2026) adds approximate-nearest-neighbor index types (rescore, experimental ivf, disk-based DiskANN) beyond brute-force KNN — flagged as on-the-horizon/experimental since the 0.1.10 line is still pre-release. Obsidian 1.13.0 desktop (early access, May 28, 2026) bumped as the current version across body references; it is a UX/security/dev-tooling release with no new AI/automation capabilities. 22 21 23
2026-06-08 Maintenance check. Model2Vec v0.8.2 (May 29, 2026) released: a maintenance release adding a frozen-weights option for training, plus multiword-token fixes, a training refactor, and non-quantized weight-handling fixes; footnote updated. Nothing else newer than the existing baseline: Obsidian latest remains 1.13.0 (May 28, already documented below), sqlite-vec stable remains v0.1.9 (v0.1.10 still alpha), and the MCP specification remains the 2025-11-25 revision. No body change beyond the Model2Vec version note. 8
2026-05-28 Obsidian 1.13.0 desktop + 1.13.0 mobile (Catalyst early-access) released. Desktop: revamped Settings panel that opens in its own window with built-in search and keyboard navigation; Obsidian URIs now present a confirmation dialog before firing actions; new warning before loading HTML resources from network drives; Search added to the Bookmarks view; enhanced editor image handling; File Explorer / Properties / Sync improvements; numerous developer-API and bug fixes. Mobile: new iOS Share Sheet with configurable target locations; tab reordering from the tab switcher; tablet press-and-hold gestures to resize splits and pinned sidebars; Bases gains a menu item to resize columns in table views; iOS and search bug fixes. Implications for AI workflows: the confirmation dialog on Obsidian URIs adds a deliberate gate to URI-driven MCP/agent integrations; the Bases column-resize menu makes Bases more usable as a vault-front index that agents query; the iOS Share Sheet configurable target makes the iPhone capture path (already documented as the primary intake) faster to wire up for Claude/Codex pipelines.
2026-05-06 Refresh source-verified currentness: Smart Connections v4.5.0 moved footer connections into Core; sqlite-vec v0.1.8/v0.1.9 stable releases updated packaging and DELETE behavior; Model2Vec v0.8.x updated tokenizer/persistence internals and benchmark tables; correct the Obsidian CLI chronology from “1.12.7 introduced CLI” to “1.12.0 introduced CLI, 1.12.7 improved installation/runtime packaging.”
2026-04-27 Web Clipper April cycle: 1.4.0 (interactive YouTube transcript UI + Open in Reader default), 1.5.0 (Highlights viewer), 1.6.0 (Highlighter UX overhaul + Defuddle 0.18 source extractors for LinkedIn/Threads/Bluesky/Discourse/Medium), 1.6.1 + 1.6.2 (Reader and Safari fixes). Reframe Web Clipper as the primary browser-side intake path for AI workflows rather than a passing bookmark mention. No Obsidian desktop, Sync, or Bases releases in the window.
2026-04-16 Smart Connections v4.3.0 (graph view, configurable dock, block-embedding recovery, Substrate cross-plugin env). Document April 2026 AI-native plugin wave (Cortex, VaultSearch, LLM Wiki, Drift, EngramQuest, Hybrid Search MCP). Flag MarkusPfundstein/mcp-obsidian as maintenance-mode (last commit June 2025). Dataview dormant; Bases is the successor for new work. Obsidian CLI 1.12.7 continues to be the preferred bridge for AI assistants.
2026-04-01 Add Obsidian CLI section (v1.12 commands for AI workflows). Add agent plugin section (Claudian, Agent Client). Document Bases core plugin for vault organization. Update plugin count to 2,500+. Add iOS Share Extension as intake source. Update compatibility matrix with embedded agent plugins.
2026-03-30 MCPVault v0.11.0: list_all_tags tool. Earlier that month, 0.8.2 added .base/.canvas support and 0.9.0 renamed the package to @bitbonsai/mcpvault. Obsidian Desktop v1.12.7 bundles CLI binary for faster terminal interactions.
2026-03-23 Document sqlite-vec v0.1.7 stable: DELETE support for vec0 tables, KNN distance constraints for pagination. DiskANN approximate nearest neighbor index announced for upcoming release.
2026-03-07 Add potion-multilingual-128M (101 languages, May 2025) to embedding model comparison. sqlite-vec at v0.1.7-alpha.10 (CI/CD fixes, no feature changes). MCP spec and retrieval techniques confirmed current.
2026-03-03 Update MCP spec evolution (Streamable HTTP and tool annotations from the March 2025 revision, .well-known discovery from November 2025). Add Model2Vec fine-tuning and BPE/Unigram tokenizer support. Add community MCP server comparison table. Update Smart Connections to v4.
2026-03-02 Add potion-base-32M and potion-retrieval-32M to model comparison. Add quantization/dimensionality reduction section. Add MCP spec evolution note.
2026-03-01 Initial release

References


  1. Cormack, G.V., Clarke, C.L.A., and Buettcher, S. Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR ‘09, July 2009, pp. 758-759 (authors’ PDF, read November 11, 2026). From the abstract: RRF “consistently yields better results than any individual system, and better results than the standard method Condorcet Fuse. This result is demonstrated by using RRF to combine the results of several TREC experiments”. On the constant: “k = 60 was fixed during a pilot investigation”, and the pilot “indicated that k = 60 was near-optimal, but that the choice was not critical”. On the comparison: “The MAP score for RRF exceeds that of Condorcet Fuse in all cases, and CombMNZ in all but one.” On single systems: “RRF also outperforms the best ranking in each experiment, with the exception of TREC 9, where the best ranking was derived using a human-in-the-loop.” ↩↩↩↩↩

  2. OpenAI API pricing. text-embedding-3-small: $0.02 per million input tokens (read November 11, 2026). Estimated cost of one full re-embed: about $0.04. The index behind this guide averages 166 characters of text per chunk (measured November 11, 2026), so 49,746 chunks hold about 8.3 million characters, or about 2.1 million tokens at 4 characters per token. ↩↩↩

  3. Tulkens, S. and van Dongen, T. Model2Vec: Fast State-of-the-Art Static Embeddings. Zenodo, 2024. doi:10.5281/zenodo.17270888. How it works, from the project’s introduction (read November 10, 2026): “computing one fixed vector per token, plus lightweight post-processing. Sentence embeddings are then produced by simply averaging token vectors”, with Smooth Inverse Frequency weighting applied to the token vectors during distillation. The README says Model2Vec “reduces model size by a factor up to 50 and makes models up to 500 times faster”, and “up to 500 times faster on CPU than the original model”. Install requirements of model2vec 0.10.0 on PyPI (read November 11, 2026): jinja2, joblib, numpy, safetensors, tokenizers, tqdm and huggingface-hub; torch is pulled in only by the train, distill and onnx extras. ↩

  4. potion-base-8M Model Card and Model2Vec results. Current published tables report potion-base-8M at 51.32 Avg (All) / 51.08 Avg (MTEB), compared with all-MiniLM-L6-v2 at 55.80 Avg (All) / 55.93 Avg (MTEB), or roughly 92% retention on the all-task score. ↩↩

  5. Model Context Protocol Specification. The MCP standard for connecting AI tools to data sources. ↩

  6. Model2Vec Potion Models, potion-base-32M, and potion-retrieval-32M. The published results tables report potion-base-32M at 52.83 Avg (All) and potion-retrieval-32M at 35.06 on the retrieval table. Each model’s config.json gives the output dimension: "hidden_dim": 512 for potion-base-32M and potion-retrieval-32M, "hidden_dim": 256 for potion-base-8M and potion-multilingual-128M; model.safetensors is 129.2 MB for each 32M model (read November 10, 2026). ↩↩↩↩

  7. MCP specification changelogs, read November 10, 2026. 2025-03-26: “Added a comprehensive authorization framework based on OAuth 2.1”, “Replaced the previous HTTP+SSE transport with a more flexible Streamable HTTP” and “Added comprehensive tool annotations for better describing tool behavior, like whether it is read-only or destructive”. 2025-06-18: “Add support for structured tool output”, “Classify MCP servers as OAuth Resource Servers”, “Require MCP clients to implement Resource Indicators as described in RFC 8707”, “Add support for elicitation”. 2025-11-25: “Enhance authorization server discovery with support for OpenID Connect Discovery 1.0”, “Add support for OAuth Client ID Metadata Documents”, “Added support for URL mode elicitation”, “Add tool calling support to sampling”, “Add experimental support for tasks”, “Establish JSON Schema 2020-12 as the default dialect”, “Establish SDK tiering system”. ↩

  8. Model2Vec Releases. v0.4.0 (Feb 2025): training/fine-tuning support. v0.5.0 (Apr 2025): backend rewrite, quantization, dimensionality reduction. v0.7.0 (Oct 2025): vocabulary quantization, configurable pooling for distillation, classifier freezing. v0.8.0/v0.8.1 (Mar 2026): tokenizer and persistence refactors, Python 3.9 deprecation, MTEB V2 result updates, and Windows path compatibility. v0.8.2 (May 29, 2026): a maintenance release adding a frozen-weights option for training, plus multiword-token fixes, a training refactor, and non-quantized weight-handling fixes. v0.9.0 (Aug 12, 2026): regression head, Lightning removed from training, inference refactor. v0.10.0 (Oct 11, 2026; the current PyPI release on November 10, 2026): PCA replacement, export of classifier and regressor pipelines to ONNX, a pairwise trainer with InfoNCE pair training, token dropout in the training pipeline, higher-speed encoding. ↩↩↩↩

  9. Smart Connections for Obsidian. Smart Connections v4: local-first AI embeddings, semantic search works offline after initial indexing. ↩

  10. potion-multilingual-128M. Minish Lab, May 2025. 101-language static embedding model, best performing multilingual static embeddings. Same PyTorch-free inference as the other potion models. ↩

  11. sqlite-vec v0.1.7 Release. March 17, 2026, read November 10, 2026. “Proper DELETE support: Deleting vectors will now clear up space (if you DELETE enough to clear out a chunk, ~1024 vectors)” and “You can now use >/>=/</<= constraints on the distance column in vec0 KNN queries, which allows you to “paginate” through KNN queries. This isn’t “real” pagination”. Run against sqlite-vec 0.1.6 and 0.1.9 from PyPI: DELETE FROM chunk_vecs WHERE id = ? removes the row on both, and after deleting 2,500 of 3,000 vectors the table keeps 3 storage chunks on 0.1.6 and drops to 1 on 0.1.9. ↩↩↩

  12. MCPVault (bitbonsai/mcpvault). npm @bitbonsai/mcpvault, latest v0.18.3 on November 11, 2026 (published 2026-10-31 at 23:34 UTC; engines.node >=22.12.0); a distinct project from MarkusPfundstein/mcp-obsidian, not a rename of it. npm publish times and the repository CHANGELOG.md are the primary sources; GitHub releases exist only for some recent versions. The changelog entries for 0.15.0 through 0.18.3 are quoted in 41, and the security advisories are in 56. Earlier entries: 0.8.2 “Support for Obsidian first-party note formats: .base and .canvas”; 0.11.0 (npm 2026-03-22) “New list_all_tags tool: scans all vault notes for frontmatter tags and inline #hashtags, returns deduplicated list sorted by frequency”. The changelog dates 0.12.2, 0.12.3 and 0.12.4 to 2026-07-23; npm has 0.12.2 (09:51 UTC) and 0.12.4 (10:10 UTC) and no 0.12.3. 0.12.2: patch_note inserts newString literally instead of expanding $', $&, $`, $$ (issue #149 / PR #153); vault-prefixed absolute or ~/-style paths normalized to vault-relative (issue #122 / PR #151); npm audit high-severity findings cleared by lockfile-only updates (PR #154). 0.12.3: new wiki_link tool (PR #101) and .trash/ excluded from all tools via the default path filter. 0.12.4: wiki_link resolves path-qualified links like [[folder/Note]] by full vault-relative path rather than basename. ↩↩↩↩

  13. Introduction to Bases and Bases views, read November 10, 2026. Database-like views over vault files using frontmatter properties as fields, saved as .base files. Bases arrived in 1.9.0 desktop (early access, May 21, 2025: “Introducing Bases, a new core plugin that lets you turn any set of notes into a powerful database”) and reached the public channel in 1.9.10 (August 18, 2025). The views page lists Table and Cards (1.9), List (1.10), Kanban (1.14) and Map (1.10), the last described as “Display files as pins on an interactive map. Requires the Maps plugin.” ↩

  14. Obsidian Desktop v1.12.0 Changelog and Obsidian Desktop v1.12.7 Changelog. v1.12.0 introduced the CLI for terminal-based vault automation; v1.12.7 improved installation/runtime packaging with a standalone binary, TUI, and socket-file behavior. See also the CLI documentation. ↩↩

  15. Claudian. Its README (read November 10, 2026) describes “An Obsidian plugin that embeds AI coding agents (Claude Code, Codex, Grok Build, Antigravity CLI, OpenCode, Pi, DeepSeek Harness, and more to come) in your vault”, requires “Obsidian v1.13.0+” and is “Desktop only”; listed features include inline edit, projects, slash commands and skills, @ mentions, cross-session messaging and MCP servers. The Claude Code permission modes it offers are auto, manual, acceptEdits and yolo (mapped to bypassPermissions), defined in src/providers/claude/permissionModes.ts. ↩

  16. Agent Client. Obsidian plugin providing a unified interface for coding agents via Agent Client Protocol (ACP). Its README (read November 10, 2026) lists nine presets (Claude Code, Codex, Gemini CLI, Mistral Vibe, OpenCode, Kiro, Hermes Agent, Pi, Grok Build) “plus any ACP-compatible agent as a custom entry”, and says of permission prompts that “Auto-allow is opt-in”. On working directories: “Point a chat at any directory with New chat in directory” (read November 11, 2026). Supports note mentions, shell execution, and action approval. ↩

  17. Obsidian mobile changelog pages, read November 10, 2026. Mobile 1.13.0 (early access, May 28, 2026): “iOS: New Share Sheet with configurable Locations. Configure Locations to point to different destinations across any of your vaults.” Mobile 1.13.2 (July 14, 2026): “iOS: Added a url variable to the Share Sheet template for inserting the shared link into your note.” The 1.13 line reached the public channel on July 30, 2026. Mobile 1.14 public (October 5, 2026): “On iOS 26, Quick Capture lets you instantly create notes from the Lock Screen, Control Center, Home Screen widgets, or Shortcuts without waiting for your vault to load.” Mobile 1.15 public (November 2, 2026): “Quick Capture can now record voice notes and transcribe them as you speak, including on the Lock Screen. Recordings can be saved to your vault. Requires iOS 26. Added Quick Capture to Spotlight actions and Siri”. ↩

  18. MarkusPfundstein/mcp-obsidian. Actively maintained: the commit history (read November 11, 2026) runs through August 31, 2026, with the MCP SDK pinned below 2.0.0 to stop an import error (PR #157, opened August 1, merged August 20) and Docker packaging merged (August 31, PR #161); the May 2026 work added tools including search_by_tag and get_frontmatter plus expanded test coverage. Still ships no tagged releases, so install from a pinned commit. Local-REST-API-based: an updated option for existing REST-API deployments. Open issues #158 (August 13, 2026) and #166 (September 11, 2026) report patch_content failing against plugin 5.x with “Error 40084: Header-based PATCH targeting is ambiguous between the two patch formats, so it requires an explicit ‘Markdown-Patch-Version’ header”; the pull requests that add the header (#159, #168, #169) were unmerged on November 11, 2026. The plugin’s 5.0.0 release notes: “Old-style requests sent without a version header now fail loudly (400)”. ↩↩

  19. Smart Connections v4.5.0 Release and v4.7.0 Release. v4.5.0 (May 5, 2026): footer connections became a Core feature. v4.3.0 (April 1, 2026) “adds a graph view for connections lists, gives you control over where the Connections view opens, and improves recovery when embedding work was interrupted”, and points to the Substrate Update, which says “Smart Environment is the setup Smart Plugins reuse”, loaded once “so Smart Connections, Smart Context, Smart Chat, Smart Templates, and companion plugins can work from the same environment” (read November 11, 2026). v4.7.0 (August 5, 2026; 4.7.2 of August 6 is the latest release on November 10, 2026): “Smart Connections Core v4.7 requires Smart Environment v3”, which “adds more built-in local embedding models, and lets you switch models without restarting Obsidian or deleting the embeddings you may want to return to”, with “a broader built-in catalog, including more lightweight and multilingual local models”. ↩↩

  20. obsidianmd/obsidian-clipper releases: primary source for the Web Clipper version-feature mapping. April 2026 cycle: 1.4.0 (Apr 9, YouTube transcript UI + Open in Reader default), 1.5.0 (Apr 15, Highlights viewer + Reader fade-in), 1.5.1 (Apr 15, webpack compilation fix), 1.6.0 (Apr 21, Highlighter UX + Defuddle 0.18 with LinkedIn/Threads/Bluesky/Discourse/Medium extractors), 1.6.1 (Apr 22, Reader outline fixes + highlights search), 1.6.2 (Apr 23, Safari embedded-mode clipboard fix). Later: 1.6.3 (May 26, app icon), 1.7.0 (Jun 16: “Update to Defuddle 0.19.0”, “Fix missing highlight ==markers== in {{content}} variable”, “Highlights persist across live page and Reader”), and 1.7.1 (Jul 22, the latest release on November 10, 2026: “Import highlights”, “Add {{model}}, {{modelId}}, and {{modelProvider}} variables for Interpreter”, “Update to Defuddle 0.19.2”). Cross-listed on the Mozilla Add-ons store and Chrome Web Store. ↩

  21. sqlite-vec v0.1.8, sqlite-vec v0.1.9, sqlite-vec v0.1.10-alpha.3, and sqlite-vec v0.1.10-alpha.4. v0.1.8 fixed npm packaging; v0.1.9 fixed a DELETE bug for metadata text columns longer than 12 characters; v0.1.10-alpha.3 adds proper INSERT OR REPLACE INTO support; v0.1.10-alpha.4 (May 18, 2026) fixes ALTER TABLE RENAME failing on vec0 tables using the new ivf/diskann features and a cached-statement cleanup bug in DiskANN. The 0.1.10 line is still prerelease. ↩↩↩↩↩

  22. MCP 2026-07-28 Specification Release Candidate. Announced May 21, 2026; final spec shipped July 28, 2026. Largest MCP revision since launch: stateless protocol core (removes the initialize handshake and Mcp-Session-Id header), MCP Apps (server-rendered HTML in sandboxed client iframes), Tasks graduating from experimental core to an official extension (tasks/get, tasks/update, tasks/cancel), OAuth 2.0 / OIDC authorization hardening, and a 12-month feature-deprecation lifecycle policy. ↩↩↩

  23. Obsidian Desktop v1.13.0 Changelog. Early access, May 28, 2026. UX/security/developer-tooling release: revamped Settings panel that opens in its own window with search and keyboard navigation, confirmation dialogs before Obsidian URIs fire, a new Settings API for plugin developers, and a CLI fix for flatpak installs. No major new AI/automation capabilities beyond the 1.12.x CLI surface. ↩↩

  24. Obsidian Changelog and the official auto-update manifest (obsidianmd/obsidian-releases, desktop-releases.json). The public channel as verified on each date: 1.13.1 shipped June 9, 2026 as a Catalyst early-access release (its changelog page is badged “Early access”), and the manifest listed public latestVersion 1.12.7 on 2026-07-21 and 2026-07-27 (beta 1.13.2, then 1.13.4), while the Atom feed at obsidian.md/changelog.xml badged 1.13.1 through 1.13.4 “(Early access)” and its newest “(Public)” entry remained 1.12.7 of 2026-03-23. 2026-08-07: latestVersion 1.13.4 (public promotion July 30, 2026), beta 1.13.6. 2026-08-14: latestVersion 1.13.7, beta 1.13.7. Re-verified 2026-11-10: latestVersion 1.15.0, beta.latestVersion 1.15.1; the GitHub release list adds 1.13.8 (August 21), 1.14.4 (October 5, the 1.14 public promotion), 1.14.5 (October 22) and 1.15.0 (November 2). Changelog pages: 1.14 public (“support for opening individual Markdown files, a kanban view for Bases, and highlight colors”), 1.14.5 (“updates Electron from 43.7.5 to 43.8.3, which includes several important security fixes. Download Obsidian from our website and reinstall the app”), 1.15 public (“URI/CLI: Added commands to append or prepend to a heading, and control for line breaks around appended/prepended content”), and 1.15.1 early access (“CLI: eval now supports top-level await”; “Allow URI callbacks is now set once on each device for all vaults instead of syncing with the vault”). ↩↩↩↩↩↩↩↩↩↩↩

  25. Obsidian community plugin registry (community-plugins.json in obsidianmd/obsidian-releases, the list the in-app plugin browser reads). The JSON array held 10,232 entries on November 10, 2026, and 2,736 in the version of the file committed on March 3, 2026. ↩↩

  26. npm registry engines fields, read with npm view <package> engines on November 11, 2026: obsidian-mcp 2.0.1 node >=22; @bitbonsai/mcpvault 0.18.3 node >=22.12.0; obsidian-mcp-server 3.9.0 (published November 11, 2026) node >=24.0.0, whose README describes the server as “STDIO or Streamable HTTP”. ↩↩↩

  27. jacksteamdev/obsidian-mcp-tools is marked archived by GitHub (repository API archived: true, last push May 13, 2026). iansinnott/obsidian-claude-code-mcp is not archived, but its last push is June 27, 2025. Both read November 10, 2026. Neither plugin is in the community registry today: mcp-tools was listed in the registry file through its May 1, 2026 version and gone by May 15, and claude-code-mcp is in none of the 2026 versions checked, so it installs by hand from its GitHub releases. Its README describes a “Dual Transport MCP Server: Supports both WebSocket (for Claude Code) and HTTP/SSE (for Claude Desktop)”. ↩↩

  28. obsidian-mcp README as published to npm with 2.0.1 (npm view obsidian-mcp readme, read November 10, 2026): “Run with npx without installing the package globally. Pinning the major version receives compatible 2.x updates without automatically crossing a future major version”; “requires Node.js 22 or newer”; “MCP clients can invoke destructive tools. Back up important vaults, review client permission prompts”. Its tool table lists obsidian_create_note, obsidian_edit_note, obsidian_delete_note and obsidian_move_note. npm’s npx documentation (read November 11, 2026): packages not already present “are installed to a folder in the npm cache”. ↩↩

  29. Obsidian CLI help, read November 11, 2026. On grammar: “A parameter takes a value, written as parameter=value. If the value has spaces, wrap it in quotes”; “A flag is a boolean switch with no value”; “For multiline content use \n for newline.” The page’s own forms for the commands used in this guide are listed in 57. The Daily notes help says the plugin “opens a note based on today’s date, or creates it if it doesn’t exist”. The Obsidian URI help gives the URI side of the 1.15 change: heading (“add the content to the section under this heading instead of the whole file”) and spacing (auto, newline or inline), both of which require append or prepend. In the help repository, the November 2, 2026 commit “URI/CLI: Document heading and spacing parameters” added heading=<name> to the CLI page; the page’s inline flag (“append without newline”) was already there in April 2026. ↩↩↩↩

  30. Model2Vec source on GitHub, read November 10, 2026. model2vec/model.py declares from_pretrained(cls, path, token=None, normalize=None, subfolder=None, quantize_to=None, dimensionality=None, vocabulary_quantization=None, ...) and the properties dim, normalize, max_length, embedding_dtype and vocabulary_quantization. model2vec/train/__init__.py exports StaticModelForClassification, StaticModelForSimilarity, StaticModelForRegression and StaticModelForPairSimilarity, and the README trains with StaticModelForClassification.from_pretrained(path=...) followed by classifier.fit(...). The training README saves a trained model through pipeline = classifier.to_pipeline() and pipeline.save_pretrained(path): “This strips away torch: the head’s weights are plain numpy arrays, so the resulting StaticModelPipeline can be used for inference without installing torch.” Run against model2vec 0.10.0 from PyPI: the fine-tuning snippet in this guide trained, saved and reloaded a classifier, and save_pretrained called on the classifier itself raises AttributeError; quantize_to="int8" loads potion-base-8M as an int8 matrix of shape (29528, 256); and _compute_model_hash() called before any embedding returns 66de115ac7930385 for potion-base-8M (run November 11, 2026). The accepted quantize_to values are the DType enum in model2vec/quantization.py: float16, float32, float64 and int8. ↩↩↩

  31. sqlite-vec documentation, read November 10, 2026. KNN queries: with a column declared float[768] distance_metric=cosine, the example’s comment says the MATCH “will now use cosine distance instead of the default L2 distance”. Python usage: “use the sqlite_vec.load() function to load sqlite-vec SQL functions into a SQLite connection”, called between db.enable_load_extension(True) and db.enable_load_extension(False). The v0.1.7 release notes call ANN support “coming soon” and the v0.1.10-alpha.1 notes introduce the first ANN indexes, so MATCH on a stable release is an exact scan. Run against sqlite-vec 0.1.9 from PyPI: db.load_extension("vec0") fails with dlopen(vec0.dylib ...) (no such file), sqlite_vec.load(db) reports v0.1.9, and the same three unit vectors return L2 distances 0.6983, 1.1485 and 1.4522 from a default table against cosine distances 0.2438, 0.6596 and 1.0545 from a distance_metric=cosine table. Building from source, per Compiling (read November 11, 2026): “git clone https://github.com/asg017/sqlite-vec”, “./scripts/vendor.sh”, then “make loadable”; the v0.1.9 Makefile sets the output to dist/vec0 plus the platform suffix. ↩↩↩↩

  32. MCP 2026-07-28 specification changelog, read November 10, 2026: “Make MCP stateless: remove the initialize/notifications/initialized handshake”; a new server/discover call that every server implements “to advertise their supported protocol versions, capabilities, and identity”; a multi-round-trip request pattern “which replaces the previous approach of sending server-initiated requests, such as roots/list, sampling/createMessage, or elicitation/create”; and, under Deprecated, “Deprecate the Roots, Sampling, and Logging features” with “a minimum twelve-month deprecation window”. The release-candidate post says “This release includes two official extensions”, MCP Apps and Tasks. ↩

  33. Claude Code hooks reference, read November 10, 2026. Its event table lists 33 events, from SessionStart to SessionEnd. On output: “UserPromptSubmit, UserPromptExpansion, SessionStart, and PostModelSwitch: Claude Code adds plain-text stdout as context that Claude can see and act on”; for PreToolUse and PostToolUse, the additionalContext field of hookSpecificOutput is a “String added to Claude’s context alongside the tool result”. On PostToolUse output replacement: updatedToolOutput “Replaces the tool’s output with the provided value before it is sent to Claude. The value must match the tool’s output shape”; “Bash returns an object with stdout, stderr, interrupted, and isImage fields. For built-in tools, a value that doesn’t match the tool’s output schema is ignored and the original output is used”; “Stripping error details that Claude needs can cause it to proceed on a false assumption.” The changelog entry for 2.1.121 (April 28, 2026): “PostToolUse hooks can now replace tool output for all tools via hookSpecificOutput.updatedToolOutput (previously MCP-only)”. ↩↩↩↩↩↩

  34. Claude Code permission modes, read November 10, 2026: “With Claude Code v2.1.283 or later, auto mode is the built-in starting permission mode for interactive terminal and VS Code sessions. On earlier versions, it’s the built-in starting permission mode only on Pro, Max, Ultra, and Team plans”; “In Claude Code v2.1.302 or later, claude -p and Agent SDK runs with no permission mode set start in Manual mode”. The mode table lists default (Manual), acceptEdits, plan, auto, dontAsk and bypassPermissions. ↩

  35. Codex documentation, read November 10, 2026. Configuration reference: “User-level configuration lives in ~/.codex/config.toml. You can also add project-scoped overrides in .codex/config.toml files. Codex loads project-scoped config files only when you trust the project”; approval_policy accepts on-request, never or a granular table, and “untrusted is unsupported, and on-failure is deprecated”. MCP: codex mcp add <server-name> --env VAR1=VALUE1 --env VAR2=VALUE2 -- <stdio server-command>; an HTTP server is configured with a url key. The 0.147.0 release notes (August 7, 2026; read November 11, 2026): “Remove the deprecated codex exec --full-auto flag; use --sandbox workspace-write instead.” ↩↩↩↩

  36. Codex: build skills, read November 10, 2026: “Codex reads skills from repository, user, admin, and system locations”, listing $CWD/.agents/skills, $CWD/../.agents/skills, $REPO_ROOT/.agents/skills, $HOME/.agents/skills and /etc/codex/skills. ↩

  37. Cursor documentation, read November 10, 2026. Rules: “Project rules live in .cursor/rules as .mdc files and are version-controlled”; AGENTS.md is a “Simple alternative to .cursor/rules”; “If alwaysApply is true, the rule will be applied to every chat session”. Rules help: “The .cursorrules file in your project root is legacy and will be deprecated.” Claude Code’s memory documentation says Claude “can also read a repository’s AGENTS.md files in place of CLAUDE.md”. ↩

  38. Devin Local customizations, the page docs.windsurf.com/windsurf/cascade/mcp redirects to (read November 10, 2026). MCP config scopes: project .devin/mcp_config.json, local override .devin/mcp_config.local.json, user ~/.config/devin/mcp_config.json; “Devin Local supports the stdio, Streamable HTTP, and SSE transports”. The page says legacy .windsurf/ rules and skills are still read; it does not say whether the older ~/.codeium/windsurf/mcp_config.json still loads. ↩

  39. Continue: MCP, read November 10, 2026: “MCP Servers can be added to your config using mcpServers”; “Create a folder called .continue/mcpServers at the top level of your workspace”; a server’s type is one of sse, stdio or streamable-http. ↩

  40. Claude Code MCP documentation, read November 10, 2026: “Configure MCP server startup timeout using the MCP_TIMEOUT environment variable (for example, MCP_TIMEOUT=10000 claude sets a 10-second timeout)”; “Set a per-server tool execution timeout by adding a timeout field in milliseconds to that server’s .mcp.json entry”; “Use the -s or --scope flag to specify where the configuration is stored”. On remote entries: “A JSON entry that has a url but no type is a configuration error, because Claude Code reads an entry with no type as a stdio server. Claude Code skips that server and reports MCP server "<name>" has a "url" but no "type"; add "type": "http" (or "sse" / "ws") to this entry”; “The SSE (Server-Sent Events) transport is deprecated. Use HTTP servers instead, where available”; WebSocket servers are configured with a "type": "ws" JSON entry, since --transport does not accept it. claude mcp add --help on Claude Code 2.1.322: “-t, –transport Transport type (stdio, sse, http). Defaults to stdio if not specified.” ↩↩↩↩

  41. MCPVault CHANGELOG.md, read November 10, 2026, with npm publish dates: 0.15.0 (2026-08-09) “New --read-only CLI mode exposes only the 11 non-mutating tools and rejects direct calls to all vault mutations”; 0.16.0 (2026-08-17) “Migrated to the official MCP TypeScript SDK v2” and “The stdio server is now dual-era: legacy and 2026-07-28 clients are served from the same process”; 0.17.0 (2026-10-22) “Added repeatable --exclude <subtree>”; 0.17.1 (2026-10-23), under Security, “Hardened vault-boundary checks for symlink paths, including missing destination folders. Unresolved symlinks now fail closed”; 0.17.2 (2026-10-30) “delete_note with trashMode: \"system\" now treats the confirmed path as a literal filename”; 0.18.0 (2026-10-31) “Raised the minimum supported Node.js version to 22.12.0”; 0.18.2 (2026-10-31) “Disabled gray-matter’s executable JavaScript frontmatter engine”; 0.18.3 (2026-10-31) fixes frontmatter detection in notes with Windows line endings. ↩↩

  42. SQLite FTS5 Extension, read November 11, 2026. On query strings (section 3): a bareword may hold only non-ASCII characters, ASCII letters and digits, the underscore and the substitute character, and “Strings that include any other characters must be quoted”; phrases “separated only by whitespace are handled as if there were an implicit AND operator between each pair”. On bm25(): “Better matches are assigned numerically lower values.” On the 'delete' command (section 6.3): “It is used to delete the index entries associated with a single row from the full-text index.” Run on SQLite 3.53.4 against this guide’s schema: bound as typed, --incremental, AGENTS.md and what's new? raise fts5: syntax error and sqlite-vec configuration raises no such column: vec; after a file removal that skips the 'delete' rows, the removed rowids still match and INSERT INTO chunks_fts(chunks_fts, rank) VALUES('integrity-check', 1) raises “database disk image is malformed”. PRAGMA journal_mode: “The DELETE journaling mode is the default.” Write-Ahead Logging: “There is an additional quasi-persistent “-wal” file and “-shm” shared memory file associated with each database”. ↩↩↩↩↩↩

  43. Templater whitespace control: “By default, commands in Templater are not removing any newlines. Commands are replaced with their values and that’s it”, and “A dash - at the end of a tag ( -%> ) will trim one newline after the command”. Obsidian Properties help: “Type --- at the very beginning of a file.” Both read November 10, 2026. Rendered outside Obsidian with the template parser Templater 2.27.0 ships (@silentvoid13/rusty_engine 0.4.0): with a command tag on the line above the block, the output starts with an empty line and --- lands on line 2; with the tag removed, or closed with -%>, --- is on line 1. ↩

  44. Obsidian Settings help, under “Strict line breaks” (read November 10, 2026): “Markdown specs ignore single Line breaks in reading view. Turn this off to make single line breaks visible.” ↩

  45. The community plugin registry file cited in 25, read November 11, 2026, lists “Hybrid Search” (id hybrid-search, flowing-abyss/obsidian-hybrid-search-plugin): “Fast hybrid search (BM25 + semantic) over your vault via a background CLI”; “Drift” (id drift, ryanbbrown/obsidian-drift): “Detects external file changes and shows side-by-side diffs with per-chunk accept/reject”; and “EngramQuest” (id engram-quest, bahfahh/engram-quest): “Turn notes into gamified Quest Maps, visual Memory Maps (Canvas), and flashcard Review Decks powered by AI Skills. Supports Claude Code, Gemini CLI, and Cursor.” Each description ends “This plugin has not been manually reviewed by Obsidian staff.” The engine behind Hybrid Search is flowing-abyss/obsidian-hybrid-search, npm obsidian-hybrid-search (0.17.5 on November 11, 2026; engines.node ^22.22.0 || >=23.6.0; binaries ohs and obsidian-hybrid-search-mcp). Its README (read November 10, 2026) describes “BM25 + fuzzy title + semantic embeddings, fused with RRF”, says indexing “stores the results in SQLite with FTS5 and sqlite-vec”, configures the MCP server as npx -y -p obsidian-hybrid-search@latest obsidian-hybrid-search-mcp with OBSIDIAN_VAULT_PATH set, and says “serve starts the MCP server over HTTP by default” in “stateless MCP Streamable HTTP”. The registry has no plugin named Cortex, VaultSearch or LLM Wiki. A plugin called VaultSearch was listed from May to July 2026 and has since been removed, along with its repository. ↩↩↩↩

  46. Local REST API with MCP, registry id obsidian-local-rest-api, listed under that name on November 11, 2026. Release 4.0.0 (May 15, 2026): “Adds a built-in Streamable HTTP MCP server at /mcp/ for use with AI assistants.” The latest release on November 11, 2026 is 5.5.2 (October 17, 2026), which is also the version in the repository’s manifest.json; the security advisories are in 54. Its README (read November 10, 2026): “All requests are served over HTTPS with a locally generated certificate and gated behind API key authentication”, at https://127.0.0.1:27124/ by default. The Obsidian CLI is part of the app and is switched on under Settings → General → Command line interface (see 29). ↩

  47. Obsidian Headless help, read November 10, 2026: “Obsidian Headless (open beta) is a headless client for Obsidian services. It lets you sync vaults from the command line without the desktop app”; among the reasons to use it, “Give agentic tools access to a vault without access to your full computer”; it “requires Node.js 22 or later” and installs with npm install -g obsidian-headless (0.0.14 on npm). On the difference: “Obsidian CLI controls the Obsidian desktop app from your terminal. Obsidian Headless is a standalone client that runs independently, no desktop app required.” ↩

  48. Notion pricing, read November 10, 2026: the plans begin “Free $0 per member / month” and “Plus $10 per member / month”, and the feature list includes “Use Notion offline on the desktop and mobile app” and “Choose pages to download for offline use”. ↩↩

  49. Weight-file sizes from the Hugging Face model API (/api/models/<repo>/tree/main, read November 10, 2026): model.safetensors is 90.9 MB for sentence-transformers/all-MiniLM-L6-v2, 546.9 MB for nomic-ai/nomic-embed-text-v1.5 and 512.4 MB for potion-multilingual-128M. Vendor scores, read November 11, 2026: the nomic model card’s comparison table gives nomic-embed-text-v1.5 an MTEB score of 62.28, and OpenAI’s embeddings guide lists text-embedding-3-small at 62.3% under “Performance on MTEB eval”. ↩

  50. Cursor MCP documentation, read November 10, 2026: “Cursor supports three transport methods” (stdio, SSE and Streamable HTTP); “Create .cursor/mcp.json in your project for project-specific tools” and “Create ~/.cursor/mcp.json in your home directory for tools available everywhere”; its remote-server example is an entry with url and optional headers. Zed MCP documentation, read November 10, 2026: “You can connect both local and remote MCP servers from Settings → AI → MCP Servers”; a remote server is a context_servers entry with url and optional headers. The Zed page does not name the remote transport. ↩↩↩

  51. Obsidian help, read November 10, 2026. Import from Apple Notes: “Under File format choose Apple Notes”, “Currently, Importer only supports migrating from Apple Notes on macOS”, and “Any locked notes will be skipped”. Import from Notion: “Notion account (recommended) connects directly to your workspace and lets you preserve Databases and formulas which are converted to Bases. Requires a Notion integration token and an internet connection”; file import “Does not preserve Databases”; “We recommend that you do not use Notion’s Markdown export as it omits important data”; the export steps run from Settings to “Under Workspace select General” and “Find and select Export all workspace content”. Apple’s Notes User Guide: “Click the note you want to export, choose File > Export To, then choose an option.” ↩↩↩

  52. MCP specification 2026-07-28, Tools, read November 10, 2026: a tool definition includes “inputSchema: JSON Schema defining expected parameters” and “annotations: Optional properties describing tool behavior”. The specification’s schema documents readOnlyHint as “If true, the tool does not modify its environment.” ↩

  53. GitHub Docs, About authentication to GitHub, read November 11, 2026: “GitHub issues tokens that begin with a prefix to indicate the token’s type”, with a table listing ghp_, github_pat_, gho_, ghu_, ghs_ and ghr_. GitHub changelog, May 15, 2026, on the new installation tokens: “Our recommended regex to match both new and current format tokens is ghs_[A-Za-z0-9\.\-_]{36,}.” ↩

  54. Security advisories for Local REST API with MCP, read through the GitHub API on November 11, 2026. All three are rated high; none has a CVE identifier. GHSA-62gx-5q78-wrvx (June 4, 2026): “An authenticated client can read, write, or delete arbitrary files on the host with the Obsidian process’s privileges”; affected <= 4.1.2, patched 4.1.3. GHSA-j9qv-qgpv-345g (September 21, 2026): the MCP server “allows an authenticated MCP client to write arbitrary files outside the vault root via the vault_write tool”; affected = 5.1.0, patched field empty. GHSA-66m9-r757-qvq7 (October 6, 2026): “An authenticated caller can write a plugin into .obsidian/plugins/ and enable it”, which ends in “arbitrary code execution at full user privilege”; affected 5.3.1, patched field empty. The fixes the advisories leave unnamed are in the release notes. 5.2.0 (September 21, 2026): “a path traversal in the MCP tools is fixed (GHSA-j9qv-qgpv-345g)”. 5.4.0 (October 6, 2026): “The API now leaves the configuration directory alone by default, across REST endpoints (403, error code 40321), MCP tools, and signed URLs”, unless “Allow access to the configuration directory” is turned on (GHSA-66m9-r757-qvq7). The repository’s versions.json maps 5.1.0 and every later release to a minimum Obsidian version of 1.13.1 (5.0.3: 1.8.7). ↩↩↩↩↩

  55. Apple, macOS 12.3 release notes (read November 11, 2026): “Python 2.7 was removed from macOS in this update. Developers should use Python 3 or an alternative language instead.” Checked on macOS 27.0 with a default PATH: command -v python finds nothing and command -v python3 prints /usr/bin/python3. A virtual environment made with python3 -m venv has its own bin/python. ↩

  56. MCPVault’s published security advisories, read through the GitHub API on November 11, 2026; all three are rated medium. GHSA-9c83-rr99-vfwj (restricted directories denied only at vault root, not nested): vulnerable range < 0.11.5, patched in 0.11.5. GHSA-j99q-93c9-h869 (deny-list bypass via case and trailing dot/space equivalence): vulnerable range < 0.11.4, patched in 0.11.4. Both fixes predate 0.12.0, so every release from 0.12.0 on has them. GHSA-97wj-cq79-2crq (published 2026-10-23, “Vault boundary bypass when writing through unresolved symlink paths”): vulnerable range = 0.17.0, patched in 0.17.1. From its text: “MCPVault 0.17.0 can write outside the configured vault when a vault contains a dangling file symlink or an outside-directory symlink followed by multiple missing destination directories”, and “Earlier versions have not yet been assessed.” npm published 0.17.0 on 2026-10-22 and 0.17.1 on 2026-10-23. ↩↩↩

  57. Command forms on the Obsidian CLI help page, read November 11, 2026: search with query=<text> (required), path=<folder>, limit=<n> and format=text|json (“Returns matching file paths”); search:context (“Returns grep-style path:line: text output”); daily (“Open daily note”); daily:append with content=<text> (required), heading=<name>, inline and open; daily:prepend; daily:read; create with name=<name>, path=<path>, content=<text>, template=<name>, overwrite and open; append and prepend with file=<name>, path=<path>, content=<text> (required), heading=<name> and inline; templates; template:insert name=<template>, with the note “Use create path=<path> template=<name> to create a file with a template”; property:set with name=<name> and value=<value> (both required), type=, file= and path=; property:read name=<name>; properties (“List properties in the vault”, with the flag counts to “include occurrence counts”); plugins; plugins:enabled; plugin:enable id=<id>; plugin:disable id=<id>; tasks with todo, done, daily, file= and status=; task with ref=<path:line>, done, todo and toggle; eval code=<javascript>. ↩↩↩↩

  58. internetVin, post of February 25, 2026: “Here’s 22 of the commands I am using with Obsidian and Claude Code with descriptions.” x.com/internetvin/status/2026461256677245131. The list is an image attached to the post (read November 10, 2026); it includes /today, /close-day, /trace, /connect, /contradict, /drift, /graduate and /weekly-learnings. ↩