As I've been working on Personal Agent, one theory I've been operating under is that if I want an effective agent, it needs to be able to understand my intent.

Agents are just blank slates with default settings. The magic happens when it has enough context to be able to make decisions with the same understanding and principles that align with how I think.

Until models are actually able to internalize information, you'll need to manually curate this information somewhere. My answer to this is the knowledge base. A collection of information that both myself and the agent have access to and the freedom to edit.

Defining My Knowledge Base

My knowledge base is very simple:

  • Skills (repeatable workflows / runbooks)
  • Basic preferences and agent guidance (agents.md)
  • Freeform markdown files that the agent and I share

That's it. It's just a folder with markdown files. I've been a huge proponent of using YAML frontmatter with Markdown for knowledge management. I even created a note taking application that predated Obsidian that used this format.

Markdown is flexible and easy to read. The YAML frontmatter provides a place for metadata. That's really all you need.

Skills are a special subset of the knowledge base. They're repeatable workflows or runbooks that you want the agent to be able to reference, so they are injected into the context where the general knowledge isn't. The agent knows where the KB is located on disk and can explore and search it though.

Using the Knowledge Base

This is still an area I've been exploring. The current state of my KB is broken down into:

  • Wiki
  • Skills
  • Projects
  • Notes

Personal Agent has deep integrations with the knowledge base and the agent is aware of it and can explore and modify any part of it at will. It's easy to tag parts of the knowledge base in conversations with the agent.

Wiki

@karpathy's wiki idea was very cool and something I've tried implementing. In an ideal world, the agent would be able to index and form a web of information as my own personal wiki. That's useful for research and just giving the agent an understanding of the things that are interesting to me and within my sphere.

Skills

I consider skills just a subset of the knowledge base. They're repeatable workflows that are useful to inject into the agent's context so it can help itself.

Skills are just knowledge. This skill is my software development principles which are basically @GrugBrainedDev's: grugbrained.dev but rephrased so it doesn't accidentally bias the text output towards caveman speak.

This skill was one of the originating ideas of the knowledge base. It's not really a skill per-se because it's not really something I want the agent to execute, and it's not something I wanted to put in an Agents.md because not everything I want an agent to do is coding. It's more general knowledge.

Projects

I also use the knowledge base to store projects. It's extremely useful to have a location where I can dump anything related to a project. Then use that store of information to hydrate new conversations so a fresh agent can quickly ramp up on a project.

I often have ideas for a project that I want to explore but not implement yet. This is a great place to dump an idea, chat about it when it comes up, add more context, generally work on it over time.

I like storing communications with teammates about projects in here. There are usually decisions made in conversations that are useful context for myself and an Agent. Status updates on the project, notes about technical details (for example, keeping notes on the settings for running llama.cpp for my personal eval project).

Notes

Similarly for thoughts or ideas that don't have an umbrella, I have a separate notes folder to dump the ideas or thoughts.

I'm also now starting to explore offloading thoughts and ideas that I have a strong thesis on. The idea here is that if the agent has information about my beliefs, it is also useful guidance for it to make decisions in my stead.

Closing Thoughts

It does ultimately feel like a melding of minds. I try to offload myself into the system, which is shaped by the agent and also influences what I think.

There are a lot of knowledge systems that are basically overly complex RAG systems. We know agents are able to use simple tools like grep to explore code bases. RAG based retrieval systems might marginally improve the actual retrieval aspect, but nothing will populate the information into a useful format. It's the information IN the knowledge base that matters!

You can't offload teaching the agent either. You can try things like mining your past conversations for signals, but at best it can infer what you want. It can't read your mind, you need to teach it, and it takes effort.

Continuous learning, knowledge systems, etc are all very cool technology, but at the end of the day what makes an agent actually useful to you is if it's able to act for you, and in order to do that, it needs to know you.