AI experiment

I invited my AI agent into Romergo, and I don’t know what this is or what to call it.

Most AI integrations in applications are designed in a predictable way. The product adds a spark button, sends a request to its own model, and displays the response in the sidebar.

There is another common option. The application provides an MCP server, and the user opens ChatGPT, Claude, or another agent and asks it to work with the application data. This gives the agent good tools, but the conversation happens somewhere else. You need to leave the editor, explain what is currently open, switch back and check the result.

I got tired of switching between the chat and the editor, and an idea occurred to me: what if I could avoid that entirely?

I added an AI panel directly to the editor, but did not connect it to a model hosted by Romergo. Instead, the user invites their personal AI agent into the open session.

The agent remains where it already lives: in ChatGPT, Claude, Codex, or another MCP-compatible client. It keeps its model, subscription, memory, settings, and available tools. Romergo provides only a workspace, current context, and specialized tools for editing a project.

After connecting, the user can write to the agent directly from Builder:

Answers, questions and intermediate statuses are returned to the same panel. The panel shows not only the text, but also the entire task cycle: a command has been received, assigned to an agent, tools are being executed, confirmation is required, the job is completed, or an error has occurred. You no longer need to leave the editor.

It started as an experiment

I had no intention of inventing a new protocol or coming up with a new product category. I wanted to test one simple idea: could a personal user agent not just control Romergo externally via MCP, but temporarily join a live session inside the editor?

The first version was an experiment. To my surprise, it worked well enough for me to start using it myself.

The shortest diagram can be drawn like this:

Personal AI agent  <->  MCP  <->  Romergo Builder

But the sign in both directions is important.

A typical MCP call starts at the agent: the agent decides to contact the application and calls its tool. In my experiment, Romergo can also initiate work. The user writes a command in Builder, Romergo places it in a secure channel, and the already connected agent receives the command via MCP.

The agent then uses the usual Romergo tools to read or modify the project and returns the result to Builder through the same channel.

This creates a closed loop:

1. User -> Builder panel: "Replace the music in this chapter"
2. Builder -> authenticated channel: command + current page + selection
3. Personal agent -> MCP: wait for the next Builder command
4. Agent -> Romergo MCP tools: inspect and edit the project
5. Agent -> MCP channel: status, question, result, or error
6. Builder panel -> User: live response from the personal agent

Romergo does not run the model at any of these steps.

At the same time, I did not change the MCP itself and did not add a real server push to it. From a protocol perspective, all calls are still initiated by the agent. The reverse direction is implemented as a secure mailbox: Builder writes the command to the channel, and the agent waits for it through the usual long-poll MCP tool. Then the agent sends the event back using the same usual tool call.

The bidirectional behavior appears at the application-session level, not through a new MCP transport. This distinction matters: the current implementation is best understood as a small session contract built on top of standard MCP calls.

How an agent enters a session

Builder creates a temporary channel code. The user copies a short starting prompt into their AI chat with something like this:

Join my Romergo Builder channel. Run each new command once, send the result or question back to Builder, and continue to wait until I explicitly ask you to stop.

The agent then calls the MCP tool romergo_join_builder_channel and passes the code. This is a handshake: Romergo checks the OAuth session, the owner of the channel and its expiration date, and then returns the conversation state and the last command cursor.

Next, the agent calls romergo_wait_for_builder_command. This is a long-poll that waits for the next command from the editor. If nothing arrives, the agent saves the cursor and starts waiting again. If the command exists, it contains:

All attached context is marked as untrusted application content. This is data for work, and not a continuation of the agent's control instruction.

The agent performs the task using existing Romergo MCP tools. For example, it can read the project and chapter, find assets, change the scene, and then verify the saved result.

While it is working, the agent can call romergo_report_builder_event and send the panel:

The response is bound to commandId, and the next command is read from the last processed sequence. When first issued, the API atomically assigns the command to the client_id OAuth client. Another connected agent will not be able to pick up the same task at the same time. Cursor helps continue after a timeout or reconnection, and one-time confirmations of specific tool calls are protected by an argument fingerprint and cannot be reused for another action.

The rights belong to the room, not the prompt

Before connecting, the user sees the channel settings and selects what the agent is allowed to do: read the project, edit content, work with media and audio, publish or delete data. These settings can be opened at any time through the gear in the panel header.

By default, reading, editing, and working with media are available, and a direct request from the user is already considered permission to apply these actions. Publishing and deletion are disabled and enabled separately. If desired, the user can enable a stricter mode and confirm every change again, or require confirmation only for publishing and deletion.

This is not just text in the start prompt. The common server wrapper checks scope before calling the modifying MCP tool. If additional confirmation is enabled, Builder shows the exact action with "allow once" and "reject" buttons, while the original call continues to wait for a decision. The permission is bound to the command, tool and argument fingerprint, and once executed it is no longer valid.

The start, completion, error, and blocking of each modifying tool are automatically returned to the panel. Each final answer must contain a structured summary with a status and a specific description of the result. After changes, the agent also lists the changed entities, the checks performed, and links to the result; Builder displays this summary directly below the response text. The channel can be terminated explicitly from the settings; the old code stops working immediately.

What is in Romergo, and what remains with the user

I like to think of this architecture as a separation of concerns.

Romergo provides:

The user agent brings:

Intelligence does not belong to the application. The app creates a place where this intelligence can operate safely.

Why not make a regular built-in copilot

Built-in AI would be easier to explain and easier to enable. The user presses a button, the application calls the selected model, everything works.

But then Romergo would also have to become an operator of AI infrastructure:

When using a personal agent, all this already exists on the user side. Romergo is not trying to build another ChatGPT. It gives ChatGPT, Claude or another agent a specialized workspace and clear tools.

This is especially interesting for a creative editor. The same agent can know how the user writes dialogue, what tone they prefer, which references have already been discussed, and which external tools they use. Romergo does not need to copy this entire system just to help replace one background or rearrange a scene.

The strangest part: the agent is both outside and inside

The agent does not physically move to Romergo. Its model and agent loop continue to run in the original AI client.

But from the user's point of view, the agent is present inside the editor:

Therefore, the expressions “AI inside the application” and “AI outside via MCP” are both not entirely accurate here. It is rather a temporary presence of an external agent in the application session.

I'm not the first to move in this direction

After the experiment worked, I began to look for similar approaches.

Agent Client Protocol allows editors like Zed and JetBrains to connect external coding agents. Tidewave places Claude Code, Codex and other ACP agents next to the running web application and passes them the browser and runtime context. Obsidian Agent Client shows Claude Code, Codex and Gemini right next to notes and automatically attaches the active document and selection. marimo gives the external agent a live notebook as a shared workspace.

There are also more general experiments. AG-UI standardizes two-way communication between the user interface and the agent backend. WebMCP proposes that web pages register tools available to a connected browser or desktop agent. Agent Application Protocol describes a model in which the application owns the UI and domain tools, and the external agent owns the reasoning, history, and general-purpose tools.

That is, the basic idea already exists in several forms. But most implementations focus on IDEs, local coding agents, or agents that the company itself deploys.

The Romergo experiment is interesting to me because of a slightly different question: what would happen if a personal user agent could enter a regular creative application?

Not a new model. Not an API key to the model. Not a built-in copilot application. An agent that a person already uses every day.

Convenience is only half the problem

Once an external agent can listen for application commands and change a project, security is no longer an optional feature.

Questions arise that cannot be answered with one OAuth screen:

The current version limits the channel by user and time, uses OAuth, server scopes, one-time confirmations, atomic command claims, and an explicit lifecycle. This is still an experiment, but important boundaries are now part of the execution and not just guidelines for the agent.

What to call it

I don't have a final title yet.

Bring Your Own AI usually means your own API key or model selection. Bring Your Own Agent is closer because the user brings not only the model, but also memory, tools and an agent loop. However, BYOA does not describe an important part of the experiment: the application can also contact the agent and maintain a live session with it.

Perhaps this is:

I don't want to come up with a new standard just for the name. First, it's more important for me to understand whether such a model is useful outside of Romergo and what boundaries are needed for it to be trusted.

Applications may not need native AI

Today, almost every product tries to build in its own assistant. As a result, the user has many separate AIs: one in the editor, another in mail, a third in the task system. Each has its own short memory, its own limitations and its own price.

There is another possible model.

The app provides a workspace, live context, and specialized tools. The user brings an agent they already trust. While working, the agent joins the application, helps complete the task, and leaves with the user.

I don’t know yet whether this will become an independent architectural category. But now I know that such a cycle can be assembled and that it can actually be used.

But I do know that experimenting is a lot of fun.

Back to dev blog