I Built an AI App. The Hardest Problems Had Nothing to Do With AI.
I started building Knovum because I wanted a better way to work with my personal knowledge. I wanted one workspace where I could collect notes, PDFs, web pages, images, audio, videos, and code — then search across them
I started building Knovum because I wanted a better way to work with my personal knowledge.
I wanted one workspace where I could collect notes, PDFs, web pages, images, audio, videos, and code — then search across them and ask questions without constantly switching between different apps.
That idea eventually turned into a much larger engineering project.
Knovum is an open-source, local-first AI knowledge workspace built with Tauri 2, Rust, Vue 3, SQLite, and LanceDB.
GitHub: https://github.com/madebyzan/knovum
While building it, I ran into problems that had little to do with writing prompts or calling an LLM API. Desktop application environments, native permissions, long-running media processing, task recovery, and source citations all required more engineering than I initially expected.
Here are some things I learned along the way.
1. What is Knovum?
Knovum is designed to bring different kinds of knowledge into one workspace.
Its main interface is an infinite canvas where users can organize and work with different types of content, including:
- Notes and web pages
- PDFs and Office documents
- Images
- Audio and video
- Code and other text-based content
The goal is not just to store files. I also wanted the content to be searchable and useful in AI-assisted workflows.
Some of the capabilities include:
- Keyword and semantic search
- RAG-based question answering
- Source references for retrieved information
- OCR and audio transcription
- Audio and video processing
- Integration with external agents through MCP and ACP
Knovum follows a local-first approach. The application and its knowledge data are designed to remain on the user's device. Some processing can run locally, but when a user chooses a cloud model, the relevant task context is sent to that model provider.
I think that distinction matters. Local-first does not automatically mean every AI operation runs offline.
2. The technology stack
The project uses a combination of desktop, frontend, and backend technologies.
| Layer | Technology |
|---|---|
| Desktop framework | Tauri 2 |
| Backend | Rust |
| Frontend | Vue 3 |
| Structured storage | SQLite |
| Vector database | LanceDB |
| Canvas format | JSON Canvas |
| Embeddings | Qwen3 Embeddings |
I chose Tauri because I wanted a desktop application without shipping a full browser runtime as part of the app.
Rust handles native-side functionality and system integration, while Vue provides the application interface.
SQLite is used for structured data, and LanceDB supports vector-based retrieval.
The stack looks relatively straightforward on paper. The interesting problems appeared when these parts had to work together in a real desktop environment.
3. A desktop app does not run like your terminal
One of the first unexpected problems appeared on macOS.
A development command worked perfectly in my terminal, but launching the application through Finder or the Dock caused it to fail.
The problem was the environment.
An interactive terminal session and a GUI-launched desktop application do not necessarily inherit the same environment variables or executable paths.
In my case, the application could not reliably locate Node.js and npx.
I initially treated this like a dependency problem, but the underlying issue was how the application environment was initialized.
The fix involved resolving the interactive login shell environment and locating the absolute path to npx, rather than assuming the GUI process would inherit the same PATH as my terminal.
I also improved the error handling so that a missing dependency would produce a more useful error instead of an obscure failure.
Related PR: https://github.com/madebyzan/knovum/pull/153
My takeaway: Test desktop applications through the actual launch paths users will use. Running successfully from a development terminal is not enough.
4. Tauri permissions and plugin registration need to stay consistent
Another issue involved object storage functionality.
Some storage-related tests could not work correctly because the required Tauri capabilities and plugin setup were incomplete.
There were multiple pieces involved:
- Capability configuration
- Permission declarations
- Plugin registration
- Build configuration
The storage plugin was also missing from the relevant build.rs setup.
The confusing part was that these settings lived in different places. A feature could appear to be implemented in the application code while still being unavailable because the native-side configuration was incomplete.
I fixed the capability configuration, registered the missing plugin, updated permissions, and added a consistency test to make this kind of configuration problem easier to catch.
Related PR: https://github.com/madebyzan/knovum/pull/163
My takeaway: In a desktop application, configuration is part of the implementation. It deserves the same attention as application code.
5. Long-running audio processing is a memory problem, not just a timeout problem
Audio processing was another area that required a rethink.
The original audio separation approach used Spleeter. The processing path generated a large amount of intermediate STFT data, which created significant memory pressure.
Simply increasing the timeout would not have solved the underlying issue.
I moved the separation workflow to HTDemucs and changed the processing strategy:
- Split longer audio into chunks of roughly one minute.
- Use approximately four seconds of overlap between adjacent chunks.
- Apply crossfading when combining the processed segments.
- Raise the maximum supported processing duration to six hours.
- Explicitly reject inputs that exceed the configured limit.
The chunk overlap and crossfade help reduce discontinuities at segment boundaries.
This was a good reminder that long-running media workflows need to be designed around memory usage, processing boundaries, and failure handling — not just execution time.
Related PR: https://github.com/madebyzan/knovum/pull/162
6. AI tasks should not silently change models during recovery
There is another problem that is easy to overlook when building an AI application.
Imagine a user submits a task using Model A. The task fails or needs to be retried later.
Meanwhile, the user changes the application's default model to Model B.
Should the retry use Model A or Model B?
I think the answer should be Model A.
The model choice is part of the original task configuration. A retry should not silently change the behavior of a task just because the global default changed in the meantime.
I updated the task flow so that the model selection is frozen when the task is submitted.
That makes retries and recovery more predictable.
Commit: https://github.com/madebyzan/knovum/commit/68cc5eb5d2aef418a9d58320c5a3021264b2104f
My takeaway: Persist the configuration that actually matters to a task. Do not rely on mutable global defaults when recovering previous work.
7. RAG citations need to point to something users can verify
Building RAG functionality also made me think more carefully about citations.
A vector search result is not necessarily a useful citation.
For example, when retrieving information from a PDF, users should ideally be able to locate the relevant page.
For audio, a timestamp is much more useful than a generic reference to the audio file.
For notes, a title and paragraph-level reference provide a better way to verify the answer.
There is also a less obvious issue: source content can change.
Suppose a document is updated after an AI-generated answer has been stored. The old citation may no longer point to the same content.
I do not think an application should continue presenting that reference as though it were still valid.
A more trustworthy approach is to recognize when a source has changed and mark the old reference as invalid or detached.
For me, this is an important part of making RAG useful beyond a demo.
The answer matters, but so does the user's ability to check where it came from.
8. The API bill became a significant part of development
Building Knovum also involved substantial AI development costs.
The API usage alone reached approximately 6,000–7,000 USD.
In addition, I paid for several development subscriptions and services:
- Codex membership: over RMB 1,000
- Qoder membership: approximately RMB 700
- Gemini subscription: approximately RMB 60
- Qoder API usage: approximately RMB 100
There were other tools involved as well, but I am not including those costs here.
These figures are development expenses, not production operating costs or business revenue.
I also cannot give a reliable total token count. Usage was spread across different providers, accounts, subscriptions, and API billing systems, so combining everything into one accurate token number would be misleading.
The more interesting question for me is not how many tokens I used. It is what that spending helped me build, what I learned from it, and how much iteration was required to get the application working.
AI coding tools can accelerate implementation, but they do not remove the need to understand the system being built.
They can help write code. They cannot automatically decide whether a retry should preserve its original model, whether a media-processing pipeline will exhaust memory, or whether a citation still refers to the correct source.
Those decisions still require engineering judgment.
9. What I learned
Looking back, a few lessons stand out.
First, desktop applications have their own class of problems.
Environment variables, native permissions, plugin registration, packaging, and GUI launch behavior are not minor details. They are part of the product.
Second, long-running workflows need explicit resource boundaries.
For media processing, chunking, overlap handling, memory usage, and input limits are more important than simply raising timeouts.
Third, AI tasks need predictable state.
Model selection, task configuration, retries, and recovery should behave consistently. A global setting should not unexpectedly rewrite the meaning of an existing task.
Fourth, RAG needs verifiable sources.
A generated answer is much more useful when users can navigate to the supporting page, timestamp, or paragraph — and when the application can recognize outdated references.
Finally, AI tools do not replace engineering fundamentals.
They can make development faster, but reliability still comes from understanding the full system and testing how its components behave together.
10. The project is still evolving
Knovum is still under development. There are plenty of things I want to improve, and I do not consider the current implementation finished.
The source code is available on GitHub:
https://github.com/madebyzan/knovum
It is released under the MIT License.
I would especially appreciate feedback from people who have worked on:
- Tauri applications with complex native integrations
- Local-first data architectures
- RAG systems with source-level citations
- Long-running audio and video processing pipelines
- AI task orchestration and recovery
I am interested in practical feedback, especially around architecture and reliability.
One question for other developers: When building an AI application, what has turned out to be harder than you expected — the AI integration itself, or everything around it?
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.