I Built AIVideoCaption Because Hindi and Hinglish Kept Breaking English-First Caption Tools

A sentence like this looks completely normal to an Indian creator:

Aaj reel export karni hai, but captions sync nahi ho rahe.

It contains Hindi, English, and Roman script in a single sentence.

However, many captioning tools do not treat it as a normal sentence. They may skip the Hindi words, convert everything into the wrong script, misinterpret the English terms, or produce a reasonable transcript with captions that appear noticeably late.

I kept encountering variations of this problem while working with short-form Indian videos.

That is why I built AIVideoCaption, a browser-based caption editor designed around Hindi, Hinglish, and other Indian-language videos.

The problem was not just transcription

At first, I thought the main challenge would be speech recognition.

If the system could understand the words correctly, generating captions should be straightforward.

That assumption was wrong.

A useful captioning system has to solve two different problems:

1. Determine what was said.
2. Determine exactly when each word was spoken.

A transcript can be almost perfect while the captions still feel unusable.

If one word stays on screen too long, every word after it can start drifting. Fast speech makes this more visible. Chunking a long audio file can also cut through words or produce duplicates at the boundary between two chunks.

For short-form video, timing is part of the product—not a small finishing detail.

Separating transcription from alignment

The pipeline now treats transcription and timing as separate stages.

I use Sarvam AI for speech-to-text, which provides the initial transcript for Indian-language audio.

For more precise word timings, I run a local alignment stage using an ONNX version of an AI4Bharat IndicConformer model.

The simplified pipeline looks like this:

Video upload
    ↓
Audio extraction
    ↓
Speech-to-text transcription
    ↓
Text and audio normalization
    ↓
Local forced alignment
    ↓
Word-level timestamps
    ↓
Caption editor
    ↓
MP4 or SRT export

For alignment, supported Indic scripts are converted into a shared phonetic representation. This lets the alignment model listen for how the words sound instead of relying only on the script in which they are written.

That distinction is particularly useful for code-switched speech, where Hindi and English may appear in the same sentence.

The chunk-boundary problem

Long audio cannot always be processed as one continuous request, so it needs to be divided into chunks.

The obvious implementation is:

• split the audio;
• transcribe every chunk;
• combine the results.

In practice, that can introduce missing or duplicated words around each boundary.

I changed the pipeline to use padded chunks with overlapping context. After transcription, it checks the overlap and removes likely duplicates before calculating the final timestamps.

I also added checks for suspicious timing patterns—for example, one word absorbing a large section of audio while several following words are squeezed into a tiny interval.

These repairs do not make imperfect audio magically perfect, but they produce a much more usable starting point for the editor.

Preview and export must match

Another unexpectedly difficult part was keeping the browser preview consistent with the exported video.

A creator may carefully adjust:

• font size;
• line breaks;
• word highlighting;
• caption position;
• outline and shadow;
• words per caption.

If the downloaded video looks different from the preview, the editor immediately loses the creator’s trust.

To prevent that, the browser and server use the same preset definitions and layout rules. I also created 166 frozen rendering cases that check different fonts, scripts, sizes, positions, and caption styles.

When the renderer changes, these tests help catch small visual differences before they reach the exported video.

The current stack

The application currently uses:

• FastAPI for the API;
• PostgreSQL and Alembic for data and migrations;
• a PostgreSQL-backed job queue;
• Sarvam AI for transcription;
• AI4Bharat IndicConformer with ONNX Runtime for local alignment;
• FFmpeg for audio and video processing;
• Pillow and FontTools for caption rendering;
• plain JavaScript, HTML, and CSS for the studio interface.

I intentionally kept the infrastructure fairly small. At this stage, predictable rendering and reliable jobs matter more to me than adding more services.

What AIVideoCaption can do today

The current version supports:

• Hindi and Hinglish;
• English;
• Gujarati;
• Marathi;
• Bengali;
• Tamil;
• Telugu;
• Kannada;
• Malayalam;
• Punjabi;
• Odia.

Creators can choose Roman or native script where supported, correct individual words, split or merge caption lines, reposition captions, and select from 17 caption styles.

The result can be exported as a captioned MP4 or an SRT file.

What it does not solve yet

I do not want to present it as perfect.

Clear, single-speaker audio still produces the best result. Background music, overlapping speakers, uncommon names, strong regional accents, and very fast speech can require manual correction.

AIVideoCaption also transcribes what is spoken; it is not intended to be a general-purpose translation tool.

Some regional-language workflows are still in beta, and I am continuing to improve alignment and font handling based on real videos.

What I learned while building it

The biggest lesson was that “supports Hindi” cannot be treated as a checkbox.

Real Indian speech involves:

• code-switching;
• multiple writing systems;
• Romanized Indian languages;
• English brand names inside regional sentences;
• different caption-reading habits.

The second lesson was that transcription accuracy and caption quality are not the same metric. A correct sentence with poor timing still feels broken.

The third lesson was that the preview is a promise. If the export does not match it, even a technically accurate transcription is not enough.

I would value feedback from Indian-language creators

There is a free plan with three transcription minutes, no card required, and MP4 and SRT export:

👉 Try AIVideoCaption

If you test it, I would especially like to know:

1. Which language or code-switching combination did you try?
2. Did the captions remain synchronized during fast speech?
3. Was choosing between Roman and native script clear?
4. Did the exported video match the editor preview?

I built AIVideoCaption, so this post is naturally biased. My goal in sharing it here is to get practical feedback from developers and creators working with the kinds of videos that English-first tools often overlook.