[Feature Request / Plugin] Architecture of a Multimodal "Context Engine": The Company's Memory
Claude · 2026-05-02
Hello everyone,
While reflecting on what could bring massive added value to Paperclip and take it to another dimension, I thought about creating a multimodal Context Engine system for the company so that it has a true centralized memory.
The limits of current memory systems
Today, a simple file-based memory system is not enough. Take Claude Code currently: it saves its memory in its various local files, but it cannot share them with other entities. When we want to change this memory, we cannot really manage, erase, delete, or modify it easily.
The biggest pain point remains the lack of persistence across ecosystems. When switching models, memories are wiped. Each system has its own isolated memory silo: Gemini has its own, Gemini CLI has its own, Codex has its own, and Claude Code CLI has its own. For large companies and big projects, this inevitably creates a critical memory loss problem and unmanageable fragmentation.
This limitation becomes a real wall when we start connecting complex external tools, like Meta Ads MCPs or marketing MCPs (which allow generating videos, for example ). The agents cannot cross-reference their learnings.
The Solution: A Multimodal Context Engine
With this new system, we could connect all our metrics via APIs or MCPs (social media metrics, Google Ads, Google Analytics, etc.). All of this could be linked and mapped using vectors and vector databases.
Ideally, we would use multimodal ingestion models capable of indexing images, audio, videos, text, and PDFs everywhere. We could use Gemini Embedding 2 or Voyage 3 Multimodal for this task. If we want to focus solely on text initially, models like Qwen Embedding would do the job.
Implementation: An Optional and Independent Plugin
Answers
Aron Prins · 2026-05-17
Love that you're framing this as a memory-layer problem rather than a vector-DB problem — that's the right angle. Two things worth knowing before you build, so you don't redo work that already exists:
Hermes already does part of this. The hermeslocal adapter ships with a per-agent persistent memory store plus 30+ built-in tools (web retrieval, structured data, calculation, memory management). It's not multimodal and it's per-agent rather than per-company, but the "agent accumulates structured facts across heartbeats" piece is already solved. There's a guide under Guides → Adapters → "Hermes with Persistent Memory" that lays it out. Worth reading before you decide what your plugin actually has to add. The plugin host is still alpha. Per the Plugins doc: "The runtime and APIs are still shifting, so expect breaking changes for now and pin versions where you can." The host surface has been expanding through #5205, #5597, and #5716, and the LLM Wiki plugin is the closest reference example. So it's a fine moment to prototype, just don't expect a frozen contract yet.
On the design itself: I'd push you toward text-only, single embedding model, company-scoped store as v1. Voyage 3 or Qwen Embedding get you most of the value. Multimodal is a great v2 once retrieval ergonomics and access-control are nailed down. The interesting design question isn't ingestion — it's which agent gets to read what memory, and how does the CEO arbitrate that? That's where Paperclip has unique leverage over a thin Cognee wrapper.
If you scaffold a plugin that exposes a memory.query(scope, q) tool to agents and a small admin UI for inspecting/pruning entries, happy to look at it.
Claude · 2026-05-02
https://www.cognee.ai/
[https://github.com/topoteretes/cognee/blob/](https://github.com/topoteretes/cognee/blob/main/LICENSE)