Plugins That Stay Local: Extending a Desktop AI Agent to Your Own Email, Sheets, and CRM

Disclosure: This post is written by the aiFetchly team.

Most AI tools for small businesses are SaaS: your emails, documents, and customer data travel to someone else's servers. A desktop AI agent flips that around — it runs where your data already lives. But that creates a different problem: every business runs a different stack. One company lives in Gmail and Google Sheets, another in Outlook and Excel, a third in a homegrown CRM nobody else uses.

If the agent only ships with a fixed set of integrations, it fits nobody perfectly. That is why extensibility matters more than the number of built-in connectors.

The plugin model

A plugin, in this design, is a small, single-purpose connector between the agent and one of your systems. Three principles keep it manageable:

1. One plugin, one job

An email plugin reads and sends mail. A spreadsheet plugin reads rows. A CRM plugin looks up contacts. When a plugin does one thing, it is easy to test, easy to replace, and easy to reason about when something goes wrong.

2. Plugins run locally, data stays on the machine

Because the agent is a desktop app, a plugin can talk to your local files and your accounts from your own computer. Business documents never need to be uploaded to a third-party service just so the agent can read them.

3. Permissions are explicit and narrow

Each plugin declares what it can read and what it can change. The agent should ask before acting — drafting a reply is fine, sending it is a separate decision that a human confirms. A plugin that can read your inbox should not also be able to delete it.

What a plugin looks like

A minimal plugin is mostly a manifest: a name, the system it connects to, the actions it exposes, and the permissions it needs.

{
  "name": "email-connector",
  "system": "email",
  "actions": ["list_unread", "read_message", "draft_reply"],
  "permissions": ["mail.read", "mail.draft"]
}

A typical flow then looks like this:

1. The email plugin lists unread customer messages.
2. A local knowledge plugin searches your own documents for the relevant price list or policy.
3. The agent drafts a reply grounded in those documents.
4. You review the draft and decide whether to send it.

Nothing in that chain requires the data to leave your computer.

What to watch out for

• Keep plugins small. A plugin that tries to do everything becomes a second application, with a second set of bugs.
• Version them. When your CRM changes its API, you want to update one plugin, not rebuild the agent.
• Test against real data. A connector that works on a clean demo account can still fail on a messy real inbox — duplicate contacts, missing fields, non-English subject lines.
• Don't give a plugin broad permissions by default. Start narrow and expand only when a real workflow needs it.

Closing

The point of a plugin system is not to collect the most integrations. It is to let a small team connect the agent to the systems they already use, without handing their business data to another cloud service — and to keep the human in charge of anything that leaves the building.

We build aiFetchly, an open-source, local-first desktop AI agent for business automation. Its plugin system is how it connects to your email, spreadsheets, and CRM while keeping your documents on your own machine. A free Community edition is available.