Integrating Multi-Modal LLMs into Governed BPMN Pipelines
Customer Onboarding and Know Your Customer (KYC) compliance represent one of the most operationally friction-heavy domains in enterprise banking. Processing a single retail or corporate onboarding application requires ve
Customer Onboarding and Know Your Customer (KYC) compliance represent one of the most operationally friction-heavy domains in enterprise banking. Processing a single retail or corporate onboarding application requires verifying heterogeneous, unstructured visual documents: government-issued passports, national identity cards, utility bills for proof of address, tax identity certificates, and corporate articles of incorporation.
Traditionally, banking systems addressed document processing using one of two unsatisfactory approaches:
- Legacy OCR Systems: Brittle, template-based Optical Character Recognition (OCR) engines that require rigid pixel-coordinate mapping. Minor layout shifts, scans with poor lighting, or folded document corners cause extraction pipelines to fail, sending up to 40% of documents to manual data-entry queues.
- Unconstrained AI Scripts: Standalone Python/AI scripts that send documents to Large Language Models (LLMs) and execute database writes directly. While highly flexible, unconstrained AI scripts lack operational governance, process visibility, regulatory audit trails, and stateful human escalation paths.
Under the Xenon Architecture Standards, modern core banking platforms resolve this dilemma by constructing Governed Multi-Modal BPMN Pipelines. By embedding Multi-Modal LLMs (capable of simultaneous vision and textual inference) directly inside Flowable BPMN 2.0 process engines, document intelligence is treated as a stateful, auditable step within a enterprise-grade business workflow.
💡 Explore the Xenon Architecture Standard For comprehensive architectural blueprints, BIAN service domain mappings, and dual-orchestration integration patterns, visit the official Xenon Architecture Guide. To inspect reference code, infrastructure templates, and open-source banking modules, explore the VecPay-Tech GitHub Organization.
BIAN Service Domain Mapping & Document Extraction Taxonomy
Within a Banking Industry Architecture Network (BIAN) alignment, onboarding document processing is divided across isolated service domains. Multi-modal vision models extract facts, while BPMN gateways enforce compliance state transitions.
| BIAN Service Domain | Document Category | Multi-Modal Inspection Scope | Extracted Fact Payload | Flowable Gateway Rule | Fallback Target |
|---|---|---|---|---|---|
| Identity Verification | Government Passport / ID | Photo validity, MRZ text parsing, hologram/tamper check | ID Number, Full Name, DOB, Expiry Date | Confidence score ≥0.92 AND Expiry > Today | KYC Analyst User Task |
| Party Data Management | Utility Bill / Bank Statement | Address layout validation, issuing authority verification | Resident Address, Issue Date, Account Holder Name | Issue Date ≤90 Days AND Name Match | Compliance Review Task |
| Customer Onboarding | Articles of Incorporation | Corporate entity structure, beneficial ownership parsing | UBO Names, Registration Number, Tax ID | All UBOs Identified AND Sanction Checked | Senior Underwriter Task |
| Credit Assessment | Tax Return / Financials | Income table extraction, auditor seal verification | Gross Annual Income, Net Tax Paid, Audit Status | Income ≥ Min Loan Qualification Threshold | Credit Exception Task |
Mathematical Model: Multi-Modal KYC Confidence & Risk Scoring
To prevent flawed AI extractions from silently propagating into core customer databases, a multi-modal extraction step evaluates a composite Document Confidence Index Ckyc-total :
Where:
- Cvision-quality is the multi-modal model's visual assessment of document integrity (detecting blur, glare, edge truncation, and digital manipulation).
- Cmrz-match is a deterministic checksum algorithm evaluating Machine Readable Zone (MRZ) characters against extracted visual text fields.
- Cfield-extract is the average log-probability confidence score across extracted key-value fields.
- wvision,wmrz,wfield are normalized weighting factors such that ∑w=1.0 .
If
Ckyc-total<Θauto-approve
(typically set to 0.92), the BPMN engine routes the extracted payload alongside visual bounding-box metadata directly to a human compliance officer for visual verification.
Architecture Topology: Governed Multi-Modal BPMN Pipeline
The architecture strictly isolates Stateless Multi-Modal AI Inference from Stateful Workflow Management (Flowable) and Core Customer Persistence.
GOVERNED MULTI-MODAL BPMN PIPELINE
+---------------------------------------------------------------------------------+
| FLOWABLE BPMN 2.0 PROCESS ENGINE |
| |
| [ Start Onboarding ] ───► [ ServiceTask: MultiModalKycDelegate ] |
| │ |
| v |
| Calls Vision Inference API |
| │ |
| v |
| [ ExclusiveGateway: Evaluate Confidence & Compliance Rules ] |
+-------------------------------------------|-------------------------------------+
|
┌─────────────────────────────────┴─────────────────────────────────┐
| Confidence >= 0.92 & Valid | Confidence < 0.92 OR Expired
v v
+-----------------------------------+ +-----------------------------------+
| AUTOMATED PARTY CREATION | | FLOWABLE HUMAN KANBAN TASK |
| | | |
| [ Call BIAN Party Service API ] | | 1. Render Side-by-Side Image UI |
| [ Persist Verified Data ] | | 2. Highlight Extracted Fields |
+-----------------------------------+ | 3. Await Human Review Sign-Off |
+-----------------+-----------------+
|
Human Officer Approves
|
v
+-----------------------------------+
| AUTOMATED PARTY CREATION |
| [ Call BIAN Party Service API ] |
+-----------------------------------+
Technical Implementation: Flowable BPMN 2.0 & Multi-Modal Java Delegate
1. Flowable Process Definition (kyc-onboarding-process.bpmn20.xml)
This BPMN 2.0 XML definition orchestrates document extraction via a custom Java Delegate, evaluates extraction confidence via an exclusive gateway, and routes to a human user task if validation criteria are unmet.
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:flowable="http://flowable.org/bpmn"
targetNamespace="BIAN/CustomerOnboarding">
<process id="kycMultiModalOnboarding" name="Multi-Modal KYC Onboarding" isExecutable="true">
<startEvent id="startEvent" name="Onboarding Document Uploaded" />
<sequenceFlow sourceRef="startEvent" targetRef="extractDocumentTask" />
<!-- Service Task executing Multi-Modal Vision AI Extraction -->
<serviceTask id="extractDocumentTask"
name="Extract ID Document via Multi-Modal AI"
flowable:class="com.xenon.banking.kyc.delegate.MultiModalKycDelegate" />
<sequenceFlow sourceRef="extractDocumentTask" targetRef="evaluateConfidenceGateway" />
<!-- Exclusive Gateway evaluating Confidence & Expiry Boundaries -->
<exclusiveGateway id="evaluateConfidenceGateway" name="Confidence & Validity Check" />
<!-- Path 1: Automated Approval (High Confidence & Valid Document) -->
<sequenceFlow sourceRef="evaluateConfidenceGateway" targetRef="createPartyAccountTask">
<conditionExpression xsi:type="tFormalExpression">
${kycConfidenceScore >= 0.92 && !isDocumentExpired && isMrzValid}
</conditionExpression>
</sequenceFlow>
<!-- Path 2: Human Compliance Escalation (Low Confidence or Data Anomaly) -->
<sequenceFlow sourceRef="evaluateConfidenceGateway" targetRef="humanReviewUserTask">
<conditionExpression xsi:type="tFormalExpression">
${kycConfidenceScore < 0.92 || isDocumentExpired || !isMrzValid}
</conditionExpression>
</sequenceFlow>
<!-- Human Task for Manual Compliance Review -->
<userTask id="humanReviewUserTask"
name="Human KYC Compliance Review"
flowable:candidateGroups="kyc_compliance_officers">
<documentation>
Review document extraction anomaly for Application: ${applicationId}.
Confidence Score: ${kycConfidenceScore}. Reason: ${extractionFailureReason}
</documentation>
</userTask>
<sequenceFlow sourceRef="humanReviewUserTask" targetRef="checkHumanDecisionGateway" />
<exclusiveGateway id="checkHumanDecisionGateway" name="Human Decision?" />
<sequenceFlow sourceRef="checkHumanDecisionGateway" targetRef="createPartyAccountTask">
<conditionExpression xsi:type="tFormalExpression">${humanDecision == 'APPROVED'}</conditionExpression>
</sequenceFlow>
<sequenceFlow sourceRef="checkHumanDecisionGateway" targetRef="rejectApplicationEnd">
<conditionExpression xsi:type="tFormalExpression">${humanDecision == 'REJECTED'}</conditionExpression>
</sequenceFlow>
<serviceTask id="createPartyAccountTask"
name="Persist Party Record in Core Database"
flowable:delegateExpression="${createPartyRecordDelegate}" />
<sequenceFlow sourceRef="createPartyAccountTask" targetRef="onboardingApprovedEnd" />
<endEvent id="onboardingApprovedEnd" name="Onboarding Approved" />
<endEvent id="rejectApplicationEnd" name="Onboarding Rejected" />
</process>
</definitions>
2. Multi-Modal Vision Java Delegate Implementation (Java 21)
This Java Delegate transmits document image bytes to a multi-modal vision endpoint using Structured JSON Schema Outputs. It parses the response, computes checksum validations, and sets execution variables in Flowable's memory context.
package com.xenon.banking.kyc.delegate;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.flowable.engine.delegate.DelegateExecution;
import org.flowable.engine.delegate.JavaDelegate;
import org.springframework.stereotype.Component;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.util.Base64;
@Component
public class MultiModalKycDelegate implements JavaDelegate {
private final HttpClient httpClient = HttpClient.newHttpClient();
private final ObjectMapper objectMapper = new ObjectMapper();
@Override
public void execute(DelegateExecution execution) {
String applicationId = (String) execution.getVariable("applicationId");
byte[] documentImageBytes = (byte[]) execution.getVariable("documentImageBytes");
System.out.println("Executing Multi-Modal Vision Extraction for Application: " + applicationId);
try {
// 1. Convert Document Bytes to Base64 Data URI
String base64Image = Base64.getEncoder().encodeToString(documentImageBytes);
// 2. Construct Multi-Modal Prompt with Structured Schema Response Format
String requestPayload = buildVisionApiPayload(base64Image);
// 3. Invoke Multi-Modal Vision API Endpoint (Internal Enterprise Gateway)
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("https://ai-gateway.internal.xenon/v1/chat/completions"))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + System.getenv("AI_GATEWAY_KEY"))
.POST(HttpRequest.BodyPublishers.ofString(requestPayload))
.build();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
// 4. Parse Extracted Facts & Confidence Metrics
JsonNode resultNode = objectMapper.readTree(response.body());
String rawJsonContent = resultNode.path("choices").get(0).path("message").path("content").asText();
JsonNode extractedFields = objectMapper.readTree(rawJsonContent);
String idNumber = extractedFields.path("idNumber").asText();
String fullName = extractedFields.path("fullName").asText();
String expirationDateStr = extractedFields.path("expirationDate").asText();
String mrzCode = extractedFields.path("mrzCode").asText();
double confidenceScore = extractedFields.path("overallConfidenceScore").asDouble();
// 5. Evaluate Deterministic Business Logic Rules
LocalDate expirationDate = LocalDate.parse(expirationDateStr, DateTimeFormatter.ISO_LOCAL_DATE);
boolean isExpired = expirationDate.isBefore(LocalDate.now());
boolean isMrzValid = validateMrzChecksum(idNumber, mrzCode);
// 6. Set Flowable Execution Variables for Gateway Evaluation
execution.setVariable("extractedIdNumber", idNumber);
execution.setVariable("extractedFullName", fullName);
execution.setVariable("kycConfidenceScore", confidenceScore);
execution.setVariable("isDocumentExpired", isExpired);
execution.setVariable("isMrzValid", isMrzValid);
execution.setVariable("extractionFailureReason",
isExpired ? "DOCUMENT_EXPIRED" : (!isMrzValid ? "MRZ_CHECKSUM_MISMATCH" : "NONE"));
} catch (Exception e) {
System.err.println("Multi-Modal Extraction Failed: " + e.getMessage());
// Fallback: Force flow to Human Review Task by setting low confidence
execution.setVariable("kycConfidenceScore", 0.0);
execution.setVariable("isDocumentExpired", true);
execution.setVariable("isMrzValid", false);
execution.setVariable("extractionFailureReason", "VISION_API_EXECUTION_FAILURE");
}
}
private String buildVisionApiPayload(String base64Image) {
return """
{
"model": "xenon-vision-kyc-v1",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Extract passport data matching JSON schema: idNumber, fullName, expirationDate (YYYY-MM-DD), mrzCode, overallConfidenceScore."},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,%s"}}
]
}
],
"response_format": {"type": "json_object"},
"temperature": 0.0
}
""".formatted(base64Image);
}
private boolean validateMrzChecksum(String idNumber, String mrzCode) {
// Deterministic check: Verify MRZ line 2 contains exact ID number string
return mrzCode != null && !mrzCode.isEmpty() && mrzCode.contains(idNumber);
}
}
Regulatory Auditability, Bounded Autonomy, and Privacy
Deploying AI document processing under financial regulatory frameworks (such as EBA ICT guidelines, OCC Bulletins, and GDPR) mandates strict auditability and data governance:
1. Side-by-Side Visual Review Interface
When an application routes to the humanReviewUserTask, the Flowable operational UI renders the raw document image alongside the extracted fields. Extracted text is highlighted with bounding-box coordinates returned by the vision model, allowing human compliance officers to verify facts in seconds.
+------------------------------------+------------------------------------+
| DOCUMENT SOURCE IMAGE (PASSPORT) | EXTRACTED FACTS & CONFIDENCE METRICS|
| | |
| [ PHOTO ] P<GBRSMITH<<JOHN<<<<<< | Full Name: John Smith [99.8%] |
| 1234567897GBR8501010... | ID Number: 123456789 [99.5%] |
| | Exp. Date: 2028-05-12 [98.1%] |
| [!] MRZ Checksum Passed | Overall Confidence: 99.1% (PASSED) |
+------------------------------------+------------------------------------+
2. Immutable Audit Snapshot Logging
AI extractions must never overwrite historical compliance logs. For every document processed, the pipeline persists an immutable JSON audit snapshot to the central compliance data lake (Elasticsearch / S3 WORM storage), linking the raw document SHA-256 hash to the Flowable ProcessInstanceId:
{
"auditId": "AUDIT-KYC-2026-881204",
"flowableProcessInstanceId": "PROC-901283",
"bianDomain": "IdentityVerification",
"documentHash": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"aiModelId": "xenon-vision-kyc-v1",
"extractedPayload": {
"idNumber": "123456789",
"fullName": "John Smith",
"expirationDate": "2028-05-12",
"mrzChecksumPassed": true
},
"confidenceScore": 0.991,
"gatewayOutcome": "AUTOMATED_APPROVAL",
"timestamp": "2026-09-16T12:45:00Z"
}
3. Zero Data Retention at LLM Inference Layer
To comply with GDPR and local data residency laws, calls to third-party or internal multi-modal LLM gateways must use Zero Data Retention (ZDR) endpoints. Document image payloads are processed in-memory during inference and immediately purged from model memory without training log retention.
Embedding Multi-Modal LLM vision intelligence directly into Flowable BPMN pipelines solves the dual challenge of manual KYC bottlenecks and unconstrained AI risk. By relegating multi-modal vision models to structured fact extraction and leveraging BPMN exclusive gateways for rule evaluation and human escalation, modern core banking architectures achieve high operational throughput, sub-second onboarding execution, and complete regulatory compliance across all BIAN service domains.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.