Building an Agentic Authorization Observer with OpenTelemetry SemConv
Building an Agentic Authorization Observer with OpenTelemetry SemConv What we're building An observability layer that captures every authorization decision your AI agent makes — trust scores, drift detection
Building an Agentic Authorization Observer with OpenTelemetry SemConv
What we're building
An observability layer that captures every authorization decision your AI agent makes — trust scores, drift detection, and FGA (Fine-Grained Authorization) context — using the proposed OpenTelemetry semantic conventions for agentic authorization.
Prerequisites
- Node.js 18+
- An existing AI agent (or use the one we'll build)
- TracePilot API key (free at tracepilotai.com)
- Basic understanding of OpenTelemetry spans
Step 1: Set up your agent with authorization hooks
First, let's create a simple agent that makes decisions and checks permissions before executing tools.
npm install openai tracepilot-sdk uuid
// agent.ts
import { TracePilot } from 'tracepilot-sdk';
import OpenAI from 'openai';
const tp = new TracePilot(process.env.TRACEPILOT_API_KEY!);
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
// Simulated authorization context
interface AuthContext {
userRole: string;
resourceType: string;
action: string;
trustScore: number;
driftScore: number;
}
async function checkAuthorization(auth: AuthContext): Promise<{
allowed: boolean;
reason: string;
fgaDecision: string;
}> {
// In production, this calls your FGA service (Auth0 FGA, OPA, etc.)
const fgaDecision = auth.userRole === 'admin' ? 'allow' : 'deny';
const allowed = fgaDecision === 'allow' && auth.trustScore > 0.7;
return {
allowed,
reason: allowed ? 'Authorized' : `Denied: role=${auth.userRole}, trust=${auth.trustScore}`,
fgaDecision
};
}
async function runAgent(userInput: string) {
await tp.startTrace('authorized-agent');
const authContext: AuthContext = {
userRole: 'editor',
resourceType: 'document',
action: 'delete',
trustScore: 0.85,
driftScore: 0.12
};
const { result } = await tp.wrapOpenAI(
() => openai.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: userInput }]
}),
[{ role: 'user', content: userInput }]
);
return result;
}
Step 2: Add authorization observability with SemConv attributes
Now we wrap every authorization check with structured observability. These attributes follow the proposed agent.trust_score, agent.drift_score, and fga.* conventions.
// authorization-observer.ts
import { TracePilot } from 'tracepilot-sdk';
interface AuthorizationSpan {
trustScore: number;
driftScore: number;
fgaType: string;
fgaDecision: string;
fgaReason: string;
resourceType: string;
action: string;
userRole: string;
}
export class AuthorizationObserver {
private tp: TracePilot;
constructor(tp: TracePilot) {
this.tp = tp;
}
async observeAuthorization(
authCheck: () => Promise<{ allowed: boolean; reason: string; fgaDecision: string }>,
context: AuthorizationSpan
): Promise<{ allowed: boolean; reason: string }> {
// Record authorization decision as a span with SemConv attributes
const { result, spanId } = await this.tp.wrapToolCall(
'authorization-check',
authCheck,
undefined, // parent span — will be linked when called from agent
1, // step order
false // not destructive
);
// Add custom attributes to the span
// These follow the proposed SemConv naming convention
await this.tp.addSpanAttributes(spanId, {
'agent.trust_score': context.trustScore,
'agent.drift_score': context.driftScore,
'fga.type': context.fgaType,
'fga.decision': result.fgaDecision,
'fga.reason': context.fgaReason || result.reason,
'fga.resource_type': context.resourceType,
'fga.action': context.action,
'fga.subject_role': context.userRole
});
return { allowed: result.allowed, reason: result.reason };
}
}
Step 3: Wire it all together — agent with authorization observability
// authorized-agent.ts
import { TracePilot } from 'tracepilot-sdk';
import OpenAI from 'openai';
import { AuthorizationObserver } from './authorization-observer';
const tp = new TracePilot(process.env.TRACEPILOT_API_KEY!);
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const authObserver = new AuthorizationObserver(tp);
interface ToolCall {
name: string;
args: Record<string, unknown>;
userRole: string;
trustScore: number;
driftScore: number;
}
async function executeToolWithAuthorization(toolCall: ToolCall) {
// Step 1: Check authorization
const authResult = await authObserver.observeAuthorization(
() => checkAuthorization({
userRole: toolCall.userRole,
resourceType: toolCall.name,
action: 'execute',
trustScore: toolCall.trustScore,
driftScore: toolCall.driftScore
}),
{
trustScore: toolCall.trustScore,
driftScore: toolCall.driftScore,
fgaType: 'role_based',
fgaDecision: 'pending',
fgaReason: '',
resourceType: toolCall.name,
action: 'execute',
userRole: toolCall.userRole
}
);
// Step 2: Execute or deny based on authorization
if (!authResult.allowed) {
return { error: authResult.reason, authorized: false };
}
// Step 3: Execute the tool (wrapped for observability)
const { result } = await tp.wrapToolCall(
toolCall.name,
() => simulateToolExecution(toolCall),
undefined,
2,
true // destructive — marks with ⚠ in dashboard
);
return { data: result, authorized: true };
}
async function simulateToolExecution(toolCall: ToolCall): Promise<string> {
// Simulate actual tool execution
return `Executed ${toolCall.name} with args: ${JSON.stringify(toolCall.args)}`;
}
// Usage
async function main() {
await tp.startTrace('authorized-agent-run');
const toolCall: ToolCall = {
name: 'delete-document',
args: { documentId: 'doc-123' },
userRole: 'editor',
trustScore: 0.85,
driftScore: 0.12
};
const result = await executeToolWithAuthorization(toolCall);
console.log(result);
// → { data: "Executed delete-document with args: {\"documentId\":\"doc-123\"}", authorized: true }
}
main();
Adding observability with TracePilot
Here's where TracePilot makes debugging authorization decisions obvious. Install the SDK and add one line to see every authorization check in your dashboard.
npm install tracepilot-sdk
The AuthorizationObserver class we built above already uses TracePilot's wrapToolCall and addSpanAttributes methods. Open your TracePilot Dashboard and you'll see:
- Every authorization decision as a separate span with full context
- Trust and drift scores tracked per decision
- FGA attributes (type, decision, reason, resource, action, subject)
- The full execution tree — which agent step triggered which authorization check
To see a failed authorization in action, change the trust score:
typescript
const failedAuth = await executeToolWithAuthorization({
name: 'delete-document',
args: { documentId:
---
**Debugging AI agents shouldn't feel like reading The Matrix.**
Join other engineers who are building reliable autonomous workflows in our community: [TracePilot Discord](https://discord.gg/KzXRAXFM8)
📰 Read the original article on Dev.to AI
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.