I've been developing an agent harness I'm calling Personal Agent (naming is still hard, okay?). Personal Agent is built on top of @badlogicgames's excellent Pi coding agent. It's amazing what you can do with it and I've done quite a bit. PA is built on Electron and is 100% fully vibe coded. I've not looked at a single line of code.

The important thing is that I wanted a tool that's similar to the desktop harnesses coming from the big labs, but without big lab lock-in. With Pi, I can use any model and any harness. You need to own your own harness.

I'm not open sourcing it yet, because frankly this is for me, not for general consumption. However, I'd like to show you some interesting concepts I've built into Personal Agent that you can copy into your own tools.

Desktop UI

First and foremost, it's a desktop application because terminal UI is way too limiting. It's great for quick prototypes, but if you're trying to do anything more advanced you quickly need to start hacking in stuff like tmux which makes the whole thing feel like a hacked together kludge.

So, the first feature is a desktop UI that supports multiple threads. Heavily copied from @OpenAICodexCli. This allows me to easily swap between running conversations, see when conversations need my attention, and is the foundation for the rest of the features.

iOS Companion App

As an aside, I also have an iOS companion application. This companion app allows me to connect to a running instance of PA via my Tailscale network. This is really important as it allows me to walk away from my desk and monitor agent runs on the go.

Agent Lifecycle Management

I have a few features to improve long-running agent tasks and to put more control into the agent's hands. The goal is to give the agent tools so it can continue its own agentic loop.

Scheduled Tasks

The automation feature is powered by a scheduling engine that the agent also has access to. The agent can schedule tasks:

  • On a recurring basis
  • At a specific time in the future
  • Queued as a followup
  • Or as a deferred resume (retrigger the current loop in X minutes — syntactic sugar over option 2)

Deferred Resume

This has been a great feature so the agent doesn't have to abuse bash sleeps. It can just trigger a deferred resume in N minutes to wake its own agent loop up.

Scheduled tasks are exposed under the automations panel. It showcases active jobs and the threads they belong to.

Auto Mode

When auto mode is enabled, it injects a continuation prompt when the agent tries to exit, telling the agent to continue working. The review prompt says something like: continue if there is meaningful remaining work, stop only when the task is complete, blocked, or needs user input. Err toward continuing when useful work remains.

Then the continuation prompt fires: continue working on the current user request from where you left off.

Embedded Screenshot + Excalidraw

I wanted the ability to draw diagrams for the agent so I embedded Excalidraw. When you click the icon, it pops an Excalidraw modal, then attaches the image to your prompt. Very useful! You can also paste screenshots into Excalidraw and annotate them.

There is also a screenshot button that triggers Mac's native screenshot functionality, attaching it to the prompt for me.

Dictation

This feature currently uses OpenAI's dictation endpoint with my ChatGPT subscription. If I ever swap to something else I'd probably swap this out for a local Whisper model, but for now it's included with my ChatGPT sub.

Built-in Knowledge Base

I really wanted a way to combine my knowledge and the agent's knowledge. To do this, our shared understanding needs to be part of the tool. @karpathy showed us his cool Obsidian-based knowledge base, but I wanted it IN the harness. The knowledge base is simply a separate git repo, with a touchless sync engine.

I define "Knowledge" here as a combination of Skills + Agent.md + free-form documents/notes. I'm testing out a Karpathy-style wiki in here as well, but the real workhorse has been Projects + Notes.

I keep a folder of folders of projects, which both myself and the agent update and keep context. I also keep a folder of notes which are just markdown files where I can jot down ideas. It's great to just offload thoughts, ideas, todos.

The agent has full access to this folder — it has a pointer in its system prompt and can explore it. I can also use the @ symbol in my inputs to tag specific files in the knowledge base. All these combined allow me to more seamlessly integrate richer context into the application.

What's notably NOT in this is a RAG-based knowledge base. I experimented with RAG systems a lot both at work and with agents and frankly it's not worth the effort. It's simply easier and cleaner to just maintain a folder of markdown files so both myself and the agent can interact with it.

That's It for Now

There are so many more features in PA, but these are the main ones. You should copy these ideas if they resonate. You really should be building your own harness, owning your own agent.