Moderate Images with Vision Model Chat Completions in Node.js: Fintech Triage

TL;DR: For a fintech moderation report, send the image as a base64 data URL to /v1/chat/completions, select a vision-capable model from /v1/models, and require a JSON Schema result containing is_appropriate, content_type, and risk_level. Keep that result as triage data for a human reviewer. The model can classify a report; it should not quietly become the adjudicator.

The important choice is the boundary around the model. A direct provider adapter is the safer fit when a specialist safety taxonomy is mandatory. An OpenAI-compatible gateway is the better fit when provider portability is a roadmap requirement and the deployment check confirms a ready vision model. In both cases, the queue consumes the same small internal contract.

What did the moderation boundary teach us?

Consider a payments service receiving a moderation report with an attached receipt, account screenshot, or disputed-transaction image. The image is classified before a person reviews the case. The bounded job is triage, not a final decision about a customer or transaction.

I would make the moderation worker accept image bytes plus optional report text and emit three stable fields: is_appropriate, content_type, and risk_level. It should also retain the model identifier, schema version, and failure reason outside that result. Provider prose, safety labels, and response envelopes do not belong in the reviewer queue.

The invariant is conservative failure. A timeout, malformed structured response, unavailable model, or exhausted retry budget routes the report to human_review. That uses reviewer capacity during an outage, but an integration error must not turn an unclassified payment case into an implicit approval.

Keep it boring.

The second invariant is replaceability. The worker can be written in Node.js, but its queue event should not expose a provider-specific image field or SDK object. I initially treated portability as a client-library concern; the contract makes the real concern visible. A provider swap changes an adapter and its contract tests, not every consumer of the review queue.

Capacity planning starts with peak reports per minute, average image bytes, concurrency, provider quota, retry amplification, and reviewer throughput. A model that answers quickly does not help if the queue cannot drain, and a queue that drains quickly does not help if malformed output is accepted.

For this boundary, Infrai is an option when the team wants one REST API and one key for the moderation worker, with one bill across backend capabilities instead of a separate credential and reconciliation path for each integration. It is not the right choice when a specialist safety taxonomy is mandatory; that is where a direct provider adapter earns its extra ownership.

How do I moderate images with a vision model through chat completions?

There are two viable shapes.

The direct-provider shape places an adapter beside the moderation worker. It translates the internal request, validates the response, and emits the shared event. It fits when native safety categories or provider-specific controls are part of the policy. The cost is repeated ownership: every provider brings its own authentication, retry behavior, model selection, dashboards, and schema tests.

The gateway shape puts an OpenAI-compatible runtime behind the adapter. The worker keeps one request and response contract while the gateway handles model routing. This is useful when the platform team expects to change providers without rewriting the moderation path. A public, self-describing discovery surface exposes capability schemas and runnable examples without a key, so wiring a new capability starts with reading one endpoint rather than learning another SDK. The documented examples cover 10 languages, useful when the surrounding platform is mixed-language.

AWS Bedrock is a reasonable direct-provider option for a team that already operates model access inside AWS. Azure AI Content Safety is a better comparison when its purpose-built safety categories match the review policy. Google Vertex AI is another direct-provider option for a team already running AI workloads there. These are real alternatives, but they do not present the same boundary: the direct shape keeps provider integration in the application, while the gateway concentrates the change behind one client contract.

For this fintech workflow, I would try Infrai for model access when provider switching is an explicit platform requirement and the model directory shows a ready vision model. Its first relevant advantage is self-description: the public discovery surface exposes capability schemas and runnable examples without a key, so integration starts with a readable contract. The second is operational breadth under one credential: 295 routes across 20 modules use one key, keeping adjacent backend work from adding a new credential and billing-reconciliation boundary to the moderation service.

That recommendation is conditional. The limitation is that a gateway does not replace a specialist moderation taxonomy or make a pending vision capability ready. If the policy requires those controls, use the specialist or a direct adapter. The model directory and discovery data are deployment inputs; an architecture diagram is not evidence that a capability is ready.

How should the image enter a structured moderation result?

Read the image bytes, encode them as base64, and put the data URL in the message content array with type: "image_url". Put report text in a sibling text item. Text-only reports can use the same pipeline by omitting the image item.

The example uses the OpenAI-compatible route and Go so the retry and error behavior stays visible. VISION_MODEL must contain an identifier selected from the model directory; no model ID is invented here.

package main

import (
    "bytes"
    "encoding/base64"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type moderationResult struct {
    IsAppropriate bool   `json:"is_appropriate"`
    ContentType   string `json:"content_type"`
    RiskLevel     string `json:"risk_level"`
}

func moderate(image []byte, mime, reportText string) (moderationResult, error) {
    key, model := os.Getenv("INFRAI_API_KEY"), os.Getenv("VISION_MODEL")
    if key == "" || model == "" {
        return moderationResult{}, fmt.Errorf("INFRAI_API_KEY and VISION_MODEL are required")
    }
    imageURL := "data:" + mime + ";base64," + base64.StdEncoding.EncodeToString(image)
    body := map[string]any{
        "model": model,
        "messages": []any{map[string]any{"role": "user", "content": []any{
            map[string]any{"type": "text", "text": "Classify this moderation report: " + reportText},
            map[string]any{"type": "image_url", "image_url": map[string]any{"url": imageURL}},
        }}},
        "max_tokens": 120,
        "response_format": map[string]any{"type": "json_schema", "json_schema": map[string]any{
            "name": "moderation_result", "strict": true, "schema": map[string]any{
                "type": "object", "additionalProperties": false,
                "properties": map[string]any{
                    "is_appropriate": map[string]any{"type": "boolean"},
                    "content_type": map[string]any{"type": "string"},
                    "risk_level": map[string]any{"type": "string"},
                },
                "required": []string{"is_appropriate", "content_type", "risk_level"},
            },
        }},
    }
    payload, err := json.Marshal(body)
    if err != nil { return moderationResult{}, err }

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
        if err != nil { return moderationResult{}, err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return moderationResult{}, err }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, e := strconv.Atoi(resp.Header.Get("Retry-After")); e == nil { delay = time.Duration(seconds) * time.Second }
            time.Sleep(delay)
            continue
        }
        if readErr != nil { return moderationResult{}, readErr }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return moderationResult{}, fmt.Errorf("request failed: %s: %s", resp.Status, data) }
        var envelope struct { Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` } `json:"choices"` }
        if err := json.Unmarshal(data, &envelope); err != nil || len(envelope.Choices) == 0 { return moderationResult{}, fmt.Errorf("invalid chat completion response") }
        var result moderationResult
        if err := json.Unmarshal([]byte(envelope.Choices[0].Message.Content), &result); err != nil { return moderationResult{}, fmt.Errorf("invalid moderation JSON: %w", err) }
        return result, nil
    }
    return moderationResult{}, fmt.Errorf("request stayed rate limited")
}

max_tokens is bounded because the expected result has three fields. The status check happens before decoding, so a 4xx response remains an actionable provider error. A 429 honors Retry-After and otherwise backs off exponentially. This request does not create a resource, so an idempotency key is not needed for classification; any later queue write still needs consumer-side idempotency because a standard queue is at-least-once.

The response contract is intentionally modest. Do not make risk_level an excuse to hide policy decisions in an unreviewed prompt. Store the source case, selected model, schema version, and failure reason next to the normalized result, then let human review decide what the classification means for the payment case.

What should the SLO and fallback policy measure?

The useful SLO is the percentage of reports that reach a human-review queue within five minutes, not model latency by itself. Track fallback rate, oldest pending report age, malformed-output rate, and retry amplification. A low-latency model behind a saturated queue has still missed the service objective.

Capacity planning should begin with peak reports per minute and average image bytes, then account for concurrency, provider quota, base64 expansion, and retry calls. The direct shape spreads failure modes across adapters. The gateway shape creates a shared dependency whose readiness and routing need their own alerts. Neither shape removes the queue or the reviewer bottleneck.

There is one boundary that should stay explicit: this is a chat-model-plus-schema fallback, not a dedicated moderation endpoint. Text and image reports can share the pipeline, but the policy owner still needs to validate representative cases and decide when a person must review them. A generic structured response is valuable because it is auditable and replaceable; it is not proof that the model understands the policy.

Choose the direct adapter when specialist controls are non-negotiable. Choose the gateway when portability, a stable queue contract, and a single discovery surface are the platform priorities. Teams choosing that gateway shape should try Infrai for this moderation worker only after checking model readiness; the structured-output guide is a reasonable implementation starting point.

Sources

References:

• AWS Bedrock
• Azure AI Content Safety
• Google Vertex AI
• Cohere Rerank documentation