PowerAI Labs logo

PowerAI Labs

Category: AI Engineering

  • Deploying, evaluating, and calling models in Microsoft Foundry: a production guide for architects

    Deploying, evaluating, and calling models in Microsoft Foundry: a production guide for architects

    I have spent the last few programmes wiring Microsoft Foundry models into real workloads, and the same confusions keep surfacing in design reviews. People conflate the model with the deployment, pick a deployment type by habit, and discover the cost implications only when the first invoice lands. This post is the guide I wish my teams had read first.

    I have deliberately left out resource and project creation, RBAC setup, and quota-increase mechanics, because those sit in the getting-started post. Here I focus on the four decisions that actually move cost and behaviour: what a deployment is, which deployment type to choose, how to evaluate before you commit, and how to call the model cleanly from the SDK.

    One caveat up front. Foundry naming and feature status move quickly, and pricing on Azure renders dynamically. Treat every number below as something to verify on the Azure pricing calculator before you budget, and check the (preview) tag on the specific Learn page before you depend on a feature.

    Resource, project, model, deployment. The deployment name, not the model name, is the addressable unit for inference.
    Resource, project, model, deployment. The deployment name, not the model name, is the addressable unit for inference.

    What “deploying a model” actually means

    The mental model that saved my teams the most time is to separate four things. The resource is the Microsoft.CognitiveServices account of kind AIServices: it is the governance, quota, and networking boundary. The project is the RBAC and isolation boundary inside it, and it owns agents, connections, evaluations, and threads.

    The model is an item in the catalogue, whether an Azure OpenAI model, a Microsoft model, or a partner model from Anthropic, Meta, Mistral, Cohere, DeepSeek, xAI, and others. The deployment is what you get when you deploy a model into the resource with a chosen deployment type (the SKU) and quota. It has a deployment name that you choose, and that name, not the underlying model name, is the addressable unit for inference.

    This matters in code. Microsoft’s quickstart is explicit: the model parameter requires the model deployment name, and if your deployment name differs from the underlying model name you adjust your code accordingly. An agent definition references the same deployment name through its model field.

    There is a useful exception since early 2026. Instant models (preview) let you call a supported model by name with no deployment at all, drawing on a separate global quota pool. They route to the latest evergreen version by default (pin a version by appending a date suffix), and during preview they are available only in West US 3 projects. Microsoft frames deployments as something you level up to, not a gate you must pass first. I reach for a deployment when I need reserved throughput, custom content filters, data residency, or enterprise configuration.

    Deployment types: the decision that fixes everything else

    The deployment type is the single biggest choice. It fixes data residency, latency variance, the quota model, and how you pay. Microsoft groups the types into standard (pay-per-token), provisioned (reserved PTU), and batch (async, 50% off), each available at global, data-zone, or regional scope. Data stored at rest always remains in the designated Azure geography: the differences below are about where inference is processed and how throughput is guaranteed.

    Deployment typeSKU codeData processing scopeBilling modelSLA / latencyBest for
    Instant (preview)N/A (no deployment)Any Azure regionPay-per-token (global quota pool)Best-effort, no SLAGetting started, prototyping
    Global StandardGlobalStandardAny Azure regionPay-per-tokenBest-effort, highest default quotaGeneral workloads, highest quota
    Data Zone StandardDataZoneStandardWithin US or EU data zonePay-per-tokenBest-effort, higher quota than regionalEU/US data-zone compliance
    Standard (regional)StandardSingle deployment regionPay-per-tokenBest-effort, limited regional capacityRegional compliance, low to medium volume
    Global ProvisionedGlobalProvisionedManagedAny Azure regionReserved PTU (hourly or reservation)Guaranteed throughput, low latency variancePredictable high throughput
    Data Zone ProvisionedDataZoneProvisionedManagedWithin US or EU data zoneReserved PTUGuaranteed throughput plus data-zoneData-zone plus predictable throughput
    Regional ProvisionedProvisionedManagedSingle deployment regionReserved PTUGuaranteed throughput, strict residencyRegional compliance plus throughput
    Global BatchGlobalBatchAny Azure region50% off Global StandardNo real-time SLA, 24-hour targetLarge async jobs
    Data Zone BatchDataZoneBatchWithin US or EU data zone50% offNo real-time SLA, 24-hour targetLarge async jobs with data-zone
    DeveloperDeveloperTierAny Azure regionPay-per-tokenNo SLA, no residency guarantee, 24-hour lifetime then auto-deletedFine-tuned model evaluation only

    A few notes from Learn that bite teams in production. Not all models support all types, so check “Foundry Models sold by Azure” for availability. With Global Standard and Data Zone Standard, a primary-region interruption affects all traffic initially routed there. Developer deployments self-delete after 24 hours, so they are for evaluating fine-tuned models, not for anything that needs to persist.

    Choosing a deployment type: residency requirement, then traffic pattern, then volume.
    Choosing a deployment type: residency requirement, then traffic pattern, then volume.

    Choosing by requirement is usually faster than reading the full matrix.

    If you needUse
    No residency restrictionGlobal Standard or Global Provisioned
    EU or US data-zone complianceData Zone Standard / Data Zone Provisioned
    Single-region residencyStandard or Regional Provisioned
    Quick start or prototypeInstant models (preview)
    Variable, bursty trafficStandard or Global Standard (pay-per-token)
    Consistent high volumeProvisioned types
    Large, non-time-sensitive jobsGlobal Batch or Data Zone Batch
    Low latency varianceProvisioned types
    Fine-tuned model evaluationDeveloper

    Three platform features are worth knowing before you commit. Spillover (GA, 2026) routes overflow from a provisioned deployment (a 429 when PTUs are exhausted, for example) to a matching Standard deployment in the same resource, billed at the standard per-token rate. The data-processing level must match (global provisioned to global standard), and it works with the Foundry Agent Service but not the Responses API, so plan gateway-level fallback there. Priority processing (GA, 2026) is a pay-per-call fast lane for latency-sensitive Standard workloads at a premium over Standard. Model router is a deployable chat model that picks an underlying model per prompt, and it now supports the GPT-5 series.

    Spillover: a provisioned deployment hits a 429, overflow routes to a matching Standard deployment in the same resource, billed per token. Does not cover the Responses API.
    Spillover: a provisioned deployment hits a 429, overflow routes to a matching Standard deployment in the same resource, billed per token. Does not cover the Responses API.

    Cost: the three levers, and the free money teams forget

    Per-token rates on Azure match OpenAI’s direct API. The premium you pay buys compliance, private networking, Entra authentication, support, and a single invoice. The deployment type changes how you pay, not the underlying token rate for a given scope.

    Indicative pay-as-you-go rates follow, for Global Standard, per 1M tokens. Verify these on the Azure pricing calculator: they are correct as of June 2026 per third-party aggregators (PricePerToken.com, last updated 14 June 2026) consistent with OpenAI list prices, not guaranteed Microsoft figures.

    ModelInput / 1MOutput / 1MNotes
    GPT-4.1\$2.00\$8.001M-token context
    GPT-4.1-mini\$0.40\$1.60strong cost/quality for routers
    GPT-5\$1.25\$10.00flagship reasoning, ~272,000-token context
    GPT-5-mini\$0.25\$2.00
    GPT-5-nano\$0.05\$0.40cheapest
    Pay-as-you-go vs PTU vs Batch across volume, with the 150 to 200M tokens/month break-even band marked. Verify on the Azure pricing calculator.
    Pay-as-you-go vs PTU vs Batch across volume, with the 150 to 200M tokens/month break-even band marked. Verify on the Azure pricing calculator.

    The three cost levers are pay-as-you-go, Provisioned Throughput Units (PTU), and Batch. Pay-as-you-go wins for variable traffic. Batch runs at 50% of Global Standard with a 24-hour target turnaround and a separate enqueued-token quota, so async jobs do not disrupt online traffic. Input is JSONL, one request per line with a unique custom_id, and you pay only for completed work.

    PTU is reserved capacity, billed hourly per deployed unit regardless of tokens consumed. The GPT-4o-class Global provisioned rate is roughly \$1/hour per PTU. Reservations give large term discounts: per Microsoft Learn’s onboarding page, a 1-month reservation is around 64% off and a 1-year around 70% off for GPT-4o-class, with example rates stamped “Azure pricing as of January 1, 2025”. Minimums matter too: Global and Data Zone Provisioned require 15 PTU, Regional Provisioned requires 25 PTU for mini/nano-class and 50 PTU for larger models.

    Two scope and discount rules round this out. For a given model, Data Zone is roughly +10% over Global and Regional is roughly +10% to +25%. Cached input tokens receive an automatic discount (roughly 50% to 90% off the input rate on repeated prefixes), so keep system prompts byte-identical across requests to trigger it.

    On the PTU break-even, do not commit on a calculator estimate. Third-party analysis (AZ365.ai, 2026) puts break-even at roughly 150 to 200 million tokens per month for GPT-5, but that assumes 100% sustained utilisation. I run pay-as-you-go for 30 to 60 days, measure P95 hourly throughput, then size PTU against real telemetry. One more trap: rate limiting estimates max processed tokens at request time including max_tokens, so an over-large max_tokens can self-throttle you.

    Evaluating a model before you commit

    Before deployment I shortlist with the Foundry model leaderboard (preview), which ranks catalogue models on quality, safety, cost, and throughput with trade-off charts and side-by-side comparison of up to three models. Cost benchmarks assume a 3:1 input-to-output ratio. This narrows the field cheaply before I spend on my own evaluation.

    Then I evaluate on my own data. Model and dataset evaluation is GA (agent evaluation remains preview), runnable from the portal or via the azure-ai-evaluation SDK. AI-assisted evaluators need an Azure OpenAI deployment as the judge, and Microsoft recommends gpt-5-mini for a good cost/quality balance.

    For a RAG system, GroundednessEvaluator is the first one I set up, because it is the leading indicator of hallucination risk. I pair it with RelevanceEvaluator and the safety evaluators (ViolenceEvaluator, SelfHarmEvaluator, HateUnfairnessEvaluator, and similar). Quality evaluators such as CoherenceEvaluator and FluencyEvaluator use a 1 to 5 Likert scale with a default pass threshold of 3. Different evaluators have different data needs: groundedness needs the source context, ROUGE-style evaluators need ground-truth references, and tool-call accuracy needs the full agent message trace. Results publish to Azure Monitor and Application Insights, so I alert on groundedness regressions. The decision rule is simple: shortlist with the leaderboard, evaluate the top two or three on your own data, and pick the cheapest model that clears your thresholds.

    Using a deployed model in an agent

    An agent references the deployment by name through its model field. In Azure AI Projects 2.x:

    from azure.identity import DefaultAzureCredential
    from azure.ai.projects import AIProjectClient
    from azure.ai.projects.models import PromptAgentDefinition
    project = AIProjectClient(
        endpoint="https://<resource>.services.ai.azure.com/api/projects/<project>",
        credential=DefaultAzureCredential(),
    )
    agent = project.agents.create_version(
        agent_name="my-agent",
        definition=PromptAgentDefinition(
            model="gpt-5-mini",   # the DEPLOYMENT name (or an instant-model name)
            instructions="You are a helpful assistant that answers general questions",
        ),
    )

    To converse, create an OpenAI-compatible client and use the Responses API with an agent reference:

    openai = project.get_openai_client()
    conversation = openai.conversations.create()
    response = openai.responses.create(
        conversation=conversation.id,
        extra_body={"agent_reference": {"name": "my-agent"}},
        input="...",
    )

    The Foundry Agent Service went GA in March 2026. The teaching point worth repeating in reviews: the agent’s model is the deployment name, which you find under Models + Endpoints in the portal.

    Calling the model via the Foundry SDK

    The SDK consolidated. azure-ai-projects 2.x is now the single Foundry SDK, covering agents, inference, evaluations, and memory, with the standalone azure-ai-agents dependency folded in. The 2.0.0 stable release shipped on 6 March 2026 and current PyPI is 2.2.0. Code written for 2.x is incompatible with 1.x: the old from_connection_string and .inference.get_chat_completions_client() patterns were removed, so budget a refactor sprint if you built against the beta.

    Authenticate with DefaultAzureCredential from azure-identity. The recommended current pattern is to get an OpenAI-compatible client from the project:

    from azure.identity import DefaultAzureCredential
    from azure.ai.projects import AIProjectClient
    project = AIProjectClient(
        endpoint="https://<resource>.services.ai.azure.com/api/projects/<project>",
        credential=DefaultAzureCredential(),
    )
    with project.get_openai_client() as client:
        response = client.responses.create(
            model="gpt-5-mini",            # deployment name
            input="What is the size of France in square miles?",
        )
        print(response.output_text)

    For OpenAI-style chat completions specifically:

    with project.get_openai_client() as client:
        resp = client.chat.completions.create(
            model="gpt-4.1",               # deployment name
            messages=[{"role": "user", "content": "How many feet are in a mile?"}],
            temperature=0.7,
            max_tokens=500,
        )
        print(resp.choices[0].message.content)

    Use the project endpoint and Responses API for Foundry features (agents, evaluations, tracing, content filters). Use the direct /openai/v1 endpoint for maximum OpenAI compatibility, lowest latency, or embeddings, which the project endpoint does not currently route. On .NET the equivalents are Azure.AI.Projects, Azure.AI.Extensions.OpenAI, and Azure.Identity, with the same pattern: construct AIProjectClient, get a chat or responses client, and pass the deployment name. One gotcha: do not install the preview Azure.AI.Projects.OpenAI alongside the GA Azure.AI.Extensions.OpenAI, because duplicate types cause ambiguous references.

    Inference parameters, and why reasoning models break the rules

    These are standard OpenAI parameters on the chat-completions and responses calls.

    ParameterWhat it doesRange / defaultNotes
    temperatureScales the whole distribution0 to 2, default 1.0Tune this or top_p, not both
    top_pNucleus sampling0 to 1, default 1.00.9 a common safety net; no top_k exposed
    max_tokens / max_completion_tokensCaps output tokensset conservativelyReasoning models require max_completion_tokens
    frequency_penaltyPenalises repeated tokens-2.0 to 2.0, default 0Leave at 0 for code/JSON
    presence_penaltyEncourages new topics-2.0 to 2.0, default 0Harmful for structured output
    stopStop sequenceslist of strings
    seedBest-effort reproducibilityintegerNot guaranteed, pin model version too
    response_formattext, json_object, or JSON schema
    reasoning_effortReasoning models onlylow / medium / highHigher means more tokens, latency, cost

    The guidance on temperature versus top_p is consistent across Microsoft and OpenAI: alter one or the other, not both. My default is to leave top_p at its default and tune temperature, reaching for top_p only when I want to keep temperature fixed for style but trim the occasional weird token. A common enterprise default for GPT-style chat is temperature 0.2 to 0.3 with top_p 0.8 to 0.95.

    Reasoning models behave differently and this catches teams out. The GPT-5 series and o-series (o1, o3, o3-mini, o4-mini) reject temperature, top_p, presence_penalty, frequency_penalty, logprobs, logit_bias, and max_tokens. Sending temperature typically returns a 400 “Unsupported parameter”. Instead you use reasoning_effort (low/medium/high, with newer models adding none/minimal/xhigh) and max_completion_tokens on chat completions or max_output_tokens on the Responses API. A wrinkle to watch: gpt-5.1 defaults reasoning_effort to none, so migrating from an earlier reasoning model may require you to pass an effort level explicitly to get any reasoning at all. System messages are treated as developer messages on the o-series.

    Reasoning models reject temperature, top_p, and the penalties, and substitute reasoning_effort and max_completion_tokens.
    Reasoning models reject temperature, top_p, and the penalties, and substitute reasoning_effort and max_completion_tokens.

    The practical fix is a shared wrapper that branches on model family and strips unsupported parameters before the call. That one piece of plumbing prevents most of the 400 errors that bite teams moving to GPT-5.

    What I would actually do

    Default to Global Standard, then narrow only for a reason: Data Zone when EU or US residency is required (accept roughly +10%), Regional only for strict single-region residency (accept +10% to +25% and smaller capacity). Do not buy PTU on a guess: run pay-as-you-go for 30 to 60 days, measure P95 hourly throughput, and commit to a 1-year reservation only once sustained volume sits in the 150 to 200M tokens/month range for GPT-5-class.

    Turn on the free discounts. Route anything async to Batch, make system prompts byte-identical for cached-input savings, and send routing, extraction, and classification to a mini or nano model while reserving flagships for the hard cases. Add spillover to any provisioned customer-facing deployment, but remember it does not cover the Responses API.

    Gate model choice on evaluation, not vibes, and make parameter handling model-aware. Those two habits, plus standardising on azure-ai-projects 2.x with DefaultAzureCredential, have removed most of the surprises my teams used to hit in production.

    References

    Image credits

    All diagrams in this post are my own. They illustrate concepts documented on Microsoft Learn (linked in the References above); pricing figures shown are indicative and should be verified on the Azure pricing calculator.

  • Copilot Cowork: the agent that does the work — and the extensibility model architects should actually study

    Copilot Cowork: the agent that does the work — and the extensibility model architects should actually study

    Most Copilot feature announcements I read the way I read a release note: skim the capability list, note what changed, move on. Microsoft 365 Copilot Cowork was the first one in a while that made me stop and read the developer documentation twice. Not because of what it does — the "it sends your emails and builds your decks" story is everywhere — but because of what sits underneath it.

    Chat assistants describe work. Cowork does it. That shift is the headline, and it is real. But the part worth an architect’s attention is the extensibility model: Cowork adopts an open skills standard, runs a multi-model architecture, and packages capability as standard Microsoft 365 app packages. Read that way, Cowork is less a product and more a distribution channel for agent capabilities you may already have built.

    This post is a companion to my Microsoft IQ pillar post — Cowork is the most visible consumer of Work IQ to date. Here I want to cover the architecture, the extensibility model in detail, and what is worth doing (and not doing) with it while it is still preview.

    What Cowork is — and the preview status everyone keeps getting wrong

    Copilot Cowork carries out tasks across your Microsoft 365 environment rather than just answering questions about them. It drafts and sends email through Outlook, schedules meetings and manages your calendar, creates Word, Excel, PowerPoint, and PDF files, posts to Teams channels and chats, searches across your organisation, runs deep research, and can run prompts on a schedule for recurring work. Every step is visible in the conversation as it happens.

    The control model is the important design choice. Before any sensitive action — sending, posting, creating — Cowork pauses and asks. Medium- and high-risk actions carry a risk-level indicator. The approval button is labelled for the specific action (Send, Post, Create), and you can pause, resume, or cancel at any point. Microsoft announced Cowork on 9 March 2026 and made it available through the Frontier preview programme; it runs in the browser at m365.cloud.microsoft, in the desktop app for Windows and Mac, and — since the May 2026 update — on iOS and Android through the Microsoft 365 Copilot app.

    Here is the correction worth making early, because some third-party coverage has it wrong: Cowork is not generally available. Several write-ups have claimed a GA milestone. The Microsoft Learn documentation says the opposite, on every page, in a banner: this is prerelease documentation, the feature is in Frontier preview, and capabilities may change. Your admin account also has to be Frontier-enrolled (Copilot → Settings → Frontier) or Cowork will not even appear in Admin Center agent management. Treat anything you read about Cowork being "shipped" with that banner in mind.

    The architecture underneath

    Copilot Cowork architecture: connectors and integrations (Fabric IQ/Power BI, Dynamics 365, third-party MCP) feed the skills system, which feeds capability into Cowork; Work IQ grounds the plan; desktop and iOS/Android client surfaces feed in; every action passes a per-action approval gate before reaching Microsoft 365 surfaces.
    Cowork architecture — Work IQ grounds the agent, the skills system feeds capability, and every action passes a per-action approval gate before it reaches a Microsoft 365 surface.

    Cowork is built on Work IQ. The documentation is explicit that it "browses your entire Work IQ" to pull in the content it needs — emails, meetings, messages, files, and data across Outlook, Teams, Excel, SharePoint, and the rest of Microsoft 365. That grounding layer is what lets the agent act with context rather than starting cold from a prompt. If you want the deeper treatment of what that intelligence layer is and what it changes, that is the subject of the IQ pillar post.

    The second architectural fact is that Cowork is multi-model. It uses Microsoft’s own models alongside Anthropic’s Claude — the model selector currently exposes Claude Opus 4.7 as an option, and Microsoft documents that it uses Anthropic models as a subprocessor. One consequence that belongs in your governance notes: access to the Anthropic models is limited to Anthropic-supported regions, and Cowork is not exempt from that restriction. If your tenant spans regions, verify coverage before you assume availability.

    The third fact — the one that carries the rest of this post — is that the unit of capability is the skill. Cowork ships with built-in skills (Word, Excel, PowerPoint, PDF, Email, Scheduling, Calendar Management, Meetings, Daily Briefing, Enterprise Search, Communications, Deep Research, and Adaptive Cards), and it loads them dynamically during a conversation, showing you which are active in a side panel. Everything you can add to Cowork is expressed as a skill or a connector. So the extensibility model is the architecture that matters.

    The extensibility model

    There are two tiers, and they map cleanly to two audiences.

    Tier one: OneDrive custom skills (no-code)

    Any user can create up to 50 custom skills by dropping a SKILL.md file into a subfolder of their OneDrive at /Documents/Cowork/skills/<skill-name>/SKILL.md — for example /Documents/Cowork/skills/weekly-report/SKILL.md. Cowork discovers them automatically at the start of each conversation. A skill is a YAML frontmatter block (a name and a description) followed by a Markdown body of instructions: structure, tone, the steps you want followed, the output format you expect. No deployment, no packaging, no admin involvement. This is the path to test the model on a real recurring task this week.

    Tier two: plugin packages (skills + MCP connectors)

    The developer path uses the same distribution mechanism as Teams apps, Copilot agents, and Office add-ins: the Microsoft 365 app package. A Cowork plugin is a .zip containing a manifest.json (unified manifest v1.28), the two app icons, and a skills/ folder. It can carry two extension types. Skills are the prompt-based workflows already described. Connectors are remote MCP servers that give Cowork access to external data and APIs — Streamable HTTP over HTTPS (TLS 1.2+), JSON-RPC 2.0 message format, and support for tools/list and tools/call. The package limits are firm: a maximum of 20 skills and 10 connectors per package.

    Packages are distributed through the Microsoft 365 App Store (submitted via Partner Center) or deployed by an admin. Connector credentials never live in the manifest or the skill files — they reference the Microsoft Enterprise Token Store, using OAuthPluginVault for OAuth 2.0 APIs or ApiKeyPluginVault for API-key services, with the secret held in the vault and only a reference ID in the package.

    The Agent Skills open standard — and the Claude conversion path

    This is the detail almost no one is covering properly. Cowork’s skills are not a proprietary Microsoft format. They use the Agent Skills open standard — the same SKILL.md format supported by Claude Code, Claude.ai projects, Visual Studio Code and GitHub Copilot, Gemini CLI, Cursor, JetBrains Junie, OpenAI Codex, and, per Microsoft’s own count, 30+ other AI tools. The skill text you write for one is the skill text you can use across all of them.

    Microsoft leans into this with a PowerShell conversion script, Convert-ClaudePluginToMOS3.ps1, that turns an existing Claude Code plugin into a valid M365 package — Microsoft quotes roughly five minutes. It reads the plugin’s .claude-plugin/plugin.json, its .mcp.json, and its skills/ directory and emits a .zip with a generated manifest. The mapping is clean, but it is not complete. What converts and what does not is worth keeping in front of you:

    Conversion path: a Claude Code plugin runs through the conversion script into an M365 app package, then App Store or admin deployment into a Cowork session. Slash commands, sub-agents, and hooks are not converted.
    The Claude plugin to M365 package conversion path. Skills copy verbatim and MCP servers map to connectors; slash commands, sub-agents, and hooks do not convert.
    Claude plugin artifactM365 equivalentStatus
    plugin.jsonmanifest.jsonName, description, author mapped; GUID auto-generated (deterministic UUID v5)
    skills/*/SKILL.mdagentSkills[] + skills/ folderCopied verbatim — identical format
    .mcp.json serversagentConnectors[]URL and auth type autodetected
    commands/ (slash commands)—Not yet supported
    agents/ (sub-agents)—Not yet supported
    hooks/ (event handlers)—Not yet supported

    Skills copy across verbatim and MCP servers map to agentConnectors. Slash commands, sub-agents, and hooks do not convert. Hold that thought — it matters for the "portability" claim later.

    How a skill is loaded — the three-layer context model

    The skill format is designed around a context budget, and understanding it is the difference between a skill that triggers reliably and one that quietly never fires. The system loads a skill in three layers:

    LayerWhen loadedTarget size
    Frontmatter (name + description)Always — at startup~100 tokens
    SKILL.md bodyWhen the skill triggersUnder 5,000 tokens (1,500–2,000 words)
    references/On demand, by the agentUnlimited
    scripts/Executed, not loaded into contextN/A

    The frontmatter is always resident, so the description is doing real work on every conversation. The body loads only when the skill triggers. References are pulled in when the agent decides it needs them, and scripts are run rather than read into the window at all. Each skill can carry up to 20 companion files, 5 MB each, 10 MB total. Keep the body lean and push depth into references/.

    One gotcha causes more failures than any other, and Microsoft says so directly: the folder name must match the name field in the frontmatter, exactly. skills/contract-analysis/SKILL.md with name: contract-analysis works. The same folder with name: ContractAnalysis does not. Kebab-case only — lowercase alphanumerics and single hyphens, no underscores, no leading or trailing or consecutive hyphens. If a skill never activates and you have ruled out the description, check this first.

    The connector surface is widening fast

    The May 2026 update makes the distribution-channel framing more concrete. Microsoft is shipping native integrations into Cowork — Fabric IQ with Power BI for data, and Dynamics 365 across sales, customer service, and ERP for scenarios like pipeline reviews, case resolution, and order approvals — alongside connectors to third-party systems including LSEG, Miro, monday.com, and S&P Global Energy. Cowork has also moved onto iOS and Android, so a delegated task can run in the cloud while you are away from your desk.

    Two things follow for architects. The reach is broadening quickly, which strengthens the case for treating Cowork as a delivery surface rather than a destination. But the pace also reinforces the preview caveat: Microsoft describes itself as “still early and moving fast,” with capabilities rolling out continuously. A connector that exists this month is not a contract for next month — verify availability against the live docs before you design around any specific integration.

    Governance and control

    The governance story is reassuringly standard, because Cowork reuses the Microsoft 365 controls you already operate. Cowork inherits the signed-in user’s Entra identity and permissions — it can only reach files and mail the user can already reach. Admins get tenant-level allow and block lists, can deploy plugins on behalf of users, and can apply compliance policies. When a plugin is revoked, its skills and connectors are removed from the user’s session on the next sync; active conversations are not interrupted, but new ones lose the capability.

    Two things to note on the current preview surface. Purview sensitivity labels are surfaced in responses and citations, showing the highest-priority label across the data used — useful, but a display behaviour rather than a full enforcement story yet. And the May 2026 update brought Agent 365 integration, which is how Microsoft intends Cowork to come under enterprise observability, security, and governance through a single control plane. That integration is the direction of travel; it is not a substitute for verifying what is actually auditable in your tenant today.

    Where this falls short today

    It is preview, and the documentation is explicit that capabilities may change. That is not a disclaimer to skim past. Do not build production process dependencies on a Frontier preview feature whose behaviour Microsoft reserves the right to alter. The correction on the false GA claims matters precisely because someone, somewhere, is about to wire a business process to this on the assumption that it has shipped.

    Per-action approval is the right default and a real friction at the same time. Human-in-the-loop on every send and post is exactly what enterprise governance requires. It also caps the autonomy story. A "delegate and walk away" workflow that pings you eight times for approval is supervised execution, not delegation — and that is fine, as long as you are honest about which one you are buying. There is a "don’t ask again" option, scoped to the current conversation, and an "Approve All" for batching pending approvals; both move risk from the system to the user, and that trade should be a conscious choice, not a reflex click.

    Skill triggering is probabilistic, not configured. The description field is how the agent decides whether to activate a skill, which makes activation reliability a prompt-engineering problem rather than a guarantee. The docs push explicit trigger phrases ("use when the user asks to…") for exactly this reason. Anyone who has run function-calling agents in production will recognise the failure mode immediately; in my own work, intermittent tool selection was consistently the hardest class of bug to reproduce, precisely because it was non-deterministic. Plan for it: write specific descriptions, name your connector tools explicitly in the skill body, and measure activation rather than assuming it.

    The open standard cuts both ways. Adopting Agent Skills is genuinely good — portable skill text, no proprietary lock-in on the part that holds your domain logic. But the conversion is lossy today: slash commands, sub-agents, and hooks do not come across. And the manifest, the store review process, and the Enterprise Token Store are all Microsoft-specific. Portability of the skill text is not portability of the whole solution; the wrapper stays platform-bound even when the contents travel.

    Finally, cost clarity is thin. Cowork requires a Microsoft 365 Copilot licence and Frontier enrolment, and the consumption implications of agentic execution at scale — an agent that plans, calls tools, and acts across many steps — are not yet documented. This is the question to take to your account team before any pilot grows into something people depend on.

    What I’d do this month

    Concrete and bounded. Enrol a sandbox tenant in the Frontier programme so you are evaluating on real infrastructure rather than reading about it. Write one OneDrive custom skill for a genuine recurring task — a weekly status roll-up, a standard document format — and then measure how reliably it triggers across a dozen real prompts, not how well it works once when you phrase the prompt perfectly. That single measurement tells you more about production-readiness than any feature list.

    If you already maintain Claude Code skills, run the conversion script against one and inventory exactly what is lost — the slash commands and hooks you relied on will not survive, and it is better to know that now. And hold production dependencies until GA. Build familiarity, build a couple of skills, build an opinion. Do not build a process that breaks the next time Microsoft changes a preview behaviour.

    Where Cowork sits in the bigger picture

    Cowork is the most visible thing Microsoft has built on top of Work IQ, and that is the right way to read it. The execution capabilities will get the attention, but the durable architectural story is the combination underneath: an intelligence layer that grounds the agent, a multi-model engine, and an extensibility model that adopts an open standard and rides the existing Microsoft 365 distribution rails. For an architect, the question is not "what can Cowork do" — that list will keep changing through preview. It is "what is the unit of capability, and how does it travel." The answer, for once, is a portable, open-standard skill. That is worth studying now, even while the product around it is still moving.

    References

  • Learn Fine-Tuning — the hands-on course I’m building during my master’s

    Learn Fine-Tuning — the hands-on course I’m building during my master’s

    Why I built this

    I was halfway through my master’s when I realised I didn’t actually understand fine-tuning. I could call trainer.train(). I could read a loss curve. I could tell you the difference between LoRA and full fine-tuning at a cocktail-party depth. But if you’d asked me to explain why low-rank adaptation works — what the rank actually constrains, why some target modules matter more than others — I’d have hand-waved you through it and hoped you didn’t follow up.

    So I went looking for something that would close the gap. I read papers. I watched courses. I ran a dozen tutorials. Half of them assumed I already knew the maths and just walked me through API calls. The other half wrapped everything in a hosted notebook with a button that said “Run cell” and skipped the maths entirely. Nothing I found taught the intuition and the code together in a way that respected the fact that I wanted to understand what I was running, not just watch it run.

    So I built what I wanted to read. Nineteen lessons across nine modules, each one in three formats — a long-form markdown explanation, a clean Python script, and a runnable notebook. Every script generates its own synthetic data and runs on a laptop. I’m publishing it because the next person learning this shouldn’t have to start where I started.

    The four design principles

    Self-contained. No API keys, no cloud accounts, no external datasets. The single biggest reason fine-tuning tutorials fall over is that the dataset link rots, the API quota expires, or the notebook assumes a paid runtime. Every lesson here generates its own synthetic data and runs end-to-end on whatever machine you already own. The friction between “I want to learn this” and “I’m seeing a result” is as close to zero as I could make it.

    Concept first, code second. Every lesson opens with the theory — the maths, the trade-offs, the analogies, the ASCII diagrams — and only then introduces code. This was the principle I worked hardest on. The temptation when writing a fine-tuning lesson is to lead with from peft import LoraConfig and explain as you go. I forced myself to do the opposite: explain what a low-rank decomposition is, why it works as an approximation, what you’re giving up in exchange for the parameter savings — and only then write the line that imports the library.

    Three formats per lesson. Markdown for reading, Python script for skimming the clean code, Jupyter notebook for running cell-by-cell. The three formats aren’t redundancy. They map to three different learning modes — reading to understand, running to see, and editing to internalise — and I wanted each lesson to support all three without asking the learner to context-switch between sources.

    Small models, real patterns. Every lesson uses a model between 60M and 124M parameters — distilbert-base-uncased, bert-base-uncased, gpt2, t5-small. You can train all 19 lessons on a CPU. The point isn’t that you’d fine-tune a 66M-parameter encoder in production; the point is that the patterns — LoRA, QLoRA, DPO, the SFTTrainer pipeline — are identical at 66M and at 70B. Learn them on something that fits in your laptop’s RAM, then apply them where they need to go.

    What’s in the course (at a glance)

    The shape of it: foundations → transfer learning → supervised fine-tuning → PEFT (LoRA, QLoRA) → prompt tuning and few-shot → alignment (RLHF, DPO) → data engineering → evaluation → production. Nine modules, nineteen lessons, each one building on the last.

    I’m deliberately not walking through them one by one here — that’s what the Project page on PowerAI Labs is for, and the repo README has the full lesson-by-lesson breakdown with topics, models, and the papers behind each one.

    The three things I learned that surprised me

    LoRA’s rank is less sensitive than the papers suggest — but the target modules are everything

    I expected rank to be the lever I’d spend the most time tuning. It isn’t. On the tasks I worked through in the course, rank 4, rank 8, and rank 16 produced results that were within noise of each other. Above rank 16 the gains were small enough that I struggled to justify the extra parameters; below rank 4 the model would start to underfit, but the transition wasn’t dramatic.

    What did matter, by a long way, was which modules the LoRA adapters were attached to. Adapting only q_proj left obvious capacity on the table. Adapting q_proj and v_proj — the original LoRA paper’s recommendation — was a meaningful step up. Adapting all linear layers was a further step up again, at a parameter cost that was still tiny relative to full fine-tuning. The rank-vs-target-modules trade-off is the one I now reach for first when a LoRA run isn’t doing what I want, and it’s the opposite of what I’d have guessed before I built the course.

    DPO is genuinely simpler than RLHF, and the implicit reward is the real insight

    I’d read the DPO paper before I built Module 6, and I thought I understood it. I didn’t, not properly. The insight that survives once you’ve worked through both a full RLHF pipeline and a DPO pipeline back-to-back is that DPO doesn’t replace the reward model — it absorbs it. The Bradley-Terry preference equation can be rearranged so that the reward score is expressed as a log-ratio of policy probabilities to a reference policy, and once you make that substitution the entire reward-model-then-PPO machinery collapses into a single supervised loss over preference pairs.

    The practical consequence is that DPO is dramatically less code than RLHF, has no reward-model overfitting failure mode, and trains stably with a single hyperparameter — beta — that you can actually reason about. The conceptual consequence is harder to express but more important: once you see that the reward signal is implicit in the policy itself, you start to see alignment as a property of the model rather than a separate system bolted on top. You cannot unsee it.

    Quantisation and PEFT compose better than I expected

    QLoRA’s claim — fine-tune a billion-parameter model on a single consumer GPU at near-LoRA accuracy — sounded like marketing. It isn’t. In the lessons where I ran the comparison properly, QLoRA was within a percentage point of standard LoRA on the same task at a fraction of the VRAM. The two ideas — 4-bit NF4 quantisation of the base model, low-rank adaptation on top — compose almost orthogonally, and you genuinely lose very little to the quantisation when the adapters are doing the work.

    The practical implication isn’t subtle. Production-ready LoRA fine-tuning on a single consumer GPU is real, today, with the libraries on the install list at the top of the course. That was true in research a year ago and it’s true on a laptop now, and it changes the economics of what an individual engineer can do without asking finance for a cluster.

    Who this is for

    For engineers who want to learn by running code. For architects who need to understand what the abstractions hide before they sign off on a design that depends on them. For master’s students working on adjacent topics who want a concrete codebase next to the papers. For anyone who has felt the gap between “the code works” and “I understand what it did” and wants to close it.

    Not for people who want to call an API and move on — there are great products for that, and you don’t need this course to use them. Not for people who want a polished, certificate-bearing online course with video lectures and a discussion forum. This is self-paced, open-source, and rough-edged. The rough edges are part of the learning.

    What’s next

    The course lives at its Project page on PowerAI Labs, with the code on GitHub. Clone it, star it, fork it, file issues with corrections — I read every one, and corrections from people working through the material are the single best signal I get on what to tighten next.

    Over the next few months I’ll publish a deep-dive blog post per module on PowerAI Labs, starting with LoRA — the rank-versus-target-modules result above is going to need its own post to do it justice. Subscribe if that’s useful and I’ll link them here as they go live.

  • Authentication patterns for Microsoft Foundry — beyond DefaultAzureCredential

    Authentication patterns for Microsoft Foundry — beyond DefaultAzureCredential

    DefaultAzureCredential is the right default, and I said as much in the getting-started guide that this post follows. It walks an ordered chain — environment variables, managed identity, Azure CLI, VS Code, interactive browser — and the same line of code works on a laptop, in CI, and on production compute. That is exactly why it earns its place on day one.

    The trouble starts by the time you hit production, when the questions get more specific. Your production workload needs to authenticate as something stronger than “whichever managed identity the host happens to provide.” Your CI/CD pipeline has to deploy agents, model deployments, and role assignments without a client secret sitting on the build agent. Your app calls Foundry on behalf of a signed-in user, and the user’s own identity has to reach Foundry — both for RBAC and for audit. And a security review asks for a complete inventory of who can call what, and “DefaultAzureCredential” is not an answer to that question.

    What follows is the auth pattern catalogue I wish I had when I went from prototype to production on Foundry. Five patterns, a per-environment role assignment model, the multi-environment story, and the four things that will bite you.

    The big picture — one diagram

    Before the catalogue, the one diagram that summarises the relationships. Every identity — a developer’s laptop, a signed-in end user, a workload on Azure compute, a CI/CD pipeline — reaches Foundry by way of an Entra-issued access token. The pattern you pick determines how that token is minted, not whether Entra is in the loop.

    Authentication architecture for Microsoft Foundry — calling identities flow through Entra ID via one of five auth patterns to reach the Foundry project and its endpoints.
    Authentication architecture for Microsoft Foundry. Every calling identity reaches Foundry via an Entra-issued access token.

    1. The auth pattern catalogue

    1.1 System-assigned managed identity for single-resource workloads

    When to use it. A single App Service, Function, or Container App that calls one Foundry resource, has no shared identity needs with anything else, and never has to outlive its host.

    When not. Anything where two compute resources need the same identity, or where the identity must persist across redeploys.

    Trade-off. System-assigned managed identities are created and deleted with their host. Zero lifecycle work, zero secrets, and zero portability. If you delete the App Service, the identity is gone — along with every role assignment that ever referenced it.

    resource app 'Microsoft.Web/sites@2023-12-01' = {
      name: 'app-foundry-prod'
      location: location
      identity: { type: 'SystemAssigned' }
      properties: { serverFarmId: plan.id }
    }
    // Assign Foundry User on the project (not the resource)
    resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
      name: guid(project.id, app.id, foundryUserRoleId)
      scope: project
      properties: {
        principalId: app.identity.principalId
        principalType: 'ServicePrincipal'
        // Foundry User role ID — stable across the rename
        roleDefinitionId: subscriptionResourceId(
          'Microsoft.Authorization/roleDefinitions',
          '53ca6127-db72-4b80-b1b0-d745d6d5456d'
        )
      }
    }
    System-assigned managed identity lifecycle. The identity is created with the host, lives only as long as the App Service, and dies with it — taking every role assignment with it.
    System-assigned managed identity lifecycle. The identity is created with the host and deleted with it — taking every role assignment with it.

    1.2 User-assigned managed identity for shared and durable workloads

    When to use it. Multiple compute resources sharing one identity (App Service plus a Function, two AKS workloads, a Container App plus a Logic App). Or anywhere the identity must survive a redeploy of the compute.

    When not. A single transient workload — system-assigned is simpler, and you do not have an identity hanging around with no host.

    Trade-off. Durable and shareable, but you own the lifecycle. Think of it as identity-as-a-resource: it gets its own Bicep module, its own naming convention, and its own teardown plan.

    resource uami 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
      name: 'id-foundry-app-prod'
      location: location
    }
    resource app 'Microsoft.Web/sites@2023-12-01' = {
      name: 'app-foundry-prod'
      location: location
      identity: {
        type: 'UserAssigned'
        userAssignedIdentities: { '${uami.id}': {} }
      }
      properties: { serverFarmId: plan.id }
    }
    resource projectRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
      name: guid(project.id, uami.id, foundryUserRoleId)
      scope: project
      properties: {
        principalId: uami.properties.principalId
        principalType: 'ServicePrincipal'
        roleDefinitionId: subscriptionResourceId(
          'Microsoft.Authorization/roleDefinitions',
          '53ca6127-db72-4b80-b1b0-d745d6d5456d'
        )
      }
    }
    User-assigned managed identity shared across App Service, Function, AKS, and Container Apps — one identity, one Foundry role assignment, multiple workloads.
    User-assigned managed identity shared across App Service, Function, AKS, and Container Apps — one identity, one role assignment, multiple workloads.

    For anything in production, my default is user-assigned. The first time you redeploy a Container App and discover every role assignment has gone with it, you will thank yourself.

    1.3 Workload identity federation for GitHub Actions and other federated CI/CD

    When to use it. Any pipeline that deploys Foundry agents, model deployments, role assignments, or any other RBAC-protected operation. GitHub Actions, Azure DevOps with OIDC, Terraform Cloud, AKS workload identity — all federated subjects.

    When not. There is not a good “when not.” If your GitHub Actions workflow still has AZURE_CLIENT_SECRET in its repository secrets, you should be migrating off it.

    Trade-off. A bit of configuration up front — a federated credential on the app registration with the right subject claim and audience. Zero credential rotation forever after. The external identity provider (GitHub, Kubernetes, etc.) is trusted to assert the workload’s identity, and Entra exchanges that assertion for a token. No client secret ever crosses the wire.

    # Create the federated credential on an app registration
    az ad app federated-credential create \
      --id $APP_ID \
      --parameters '{
        "name": "github-main-prod",
        "issuer": "https://token.actions.githubusercontent.com",
        "subject": "repo:my-org/my-repo:ref:refs/heads/main",
        "audiences": ["api://AzureADTokenExchange"]
      }'
    # .github/workflows/deploy.yml
    permissions:
      id-token: write   # required to mint the OIDC token
      contents: read
    jobs:
      deploy:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: azure/login@v2
            with:
              client-id: ${{ secrets.AZURE_CLIENT_ID }}
              tenant-id: ${{ secrets.AZURE_TENANT_ID }}
              subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
              enable-AzPSSession: false
          - run: az deployment group create ...
    Workload identity federation trust between GitHub Actions and Microsoft Entra ID. The runner sends an OIDC token, Entra validates it against the federated credential on the app registration, and returns a Foundry-scoped access token.
    Workload identity federation trust between GitHub Actions and Microsoft Entra ID. The runner sends an OIDC token, Entra validates it against the federated credential, and returns a Foundry-scoped access token.

    The pattern generalises. AKS workload identity uses the same federation primitive with the cluster’s OIDC issuer as the subject. Terraform Cloud has its own. The configuration changes; the model does not.

    1.4 On-Behalf-Of flow for apps that call Foundry as the signed-in user

    When to use it. A web app or API where the end user’s identity must reach Foundry — because the user’s own RBAC determines what they can see, because audit logs need the user not the app, or because a compliance regime requires per-user attribution all the way to the model call.

    When not. Pure machine-to-machine workloads. If there is no signed-in human in the loop, you want a managed identity, not OBO.

    Trade-off. More moving parts. The user signs into the front end, the front end calls your API with their access token, the API exchanges that token for a downstream token scoped to Foundry, and only then does the call go through. It is the only correct answer for user-scoped operations.

    # Middle-tier API: exchange the incoming user token for a Foundry-scoped token
    import msal
    app = msal.ConfidentialClientApplication(
        client_id=API_CLIENT_ID,
        client_credential=API_CLIENT_SECRET,  # or a certificate / federated credential
        authority=f"https://login.microsoftonline.com/{TENANT_ID}",
    )
    # incoming_user_token comes from the Authorization header on the request
    result = app.acquire_token_on_behalf_of(
        user_assertion=incoming_user_token,
        scopes=["https://ai.azure.com/.default"],
    )
    foundry_access_token = result["access_token"]
    On-Behalf-Of flow sequence — end user signs into the front end, the middle-tier API exchanges the user token for a Foundry-scoped token, and the call to Foundry runs under the user identity with their RBAC and Conditional Access applied.
    On-Behalf-Of flow. The middle-tier API exchanges the user token for a Foundry-scoped token, and the call runs under the user identity with their RBAC and Conditional Access applied.

    One implication worth calling out: any Conditional Access policy on the user’s original sign-in propagates through the OBO exchange. If your CA policy says “no Foundry access from non-compliant devices,” the downstream Foundry call inherits that. That is almost always what you want.

    1.5 Application registrations with client secrets — when (rarely) still appropriate

    When to use it. Local developer machines that are not on a corporate-managed laptop with Entra-joined credentials. Genuinely headless scripts that cannot use a managed identity or federated workload identity. Third-party integrations that do not yet support OIDC federation. That is it.

    When not. Anything in production on Azure compute — use a managed identity. Anything in CI/CD on a platform that supports federation — use workload identity federation. Anything an auditor will ever look at.

    Trade-off. Simplest to set up, hardest to govern. Secrets rotate, they leak, they accumulate. If you have more than a handful, you have a secret-sprawl problem and you do not yet know it.

    If you must use one: short expiry (90 days), stored in Key Vault, never in a .env checked into a repo, and the role assigned to the app’s service principal is the minimum it needs — Foundry User scoped to the project, never Contributor scoped to the subscription.

    The hard line: if you are putting a client secret on a production workload, you have taken a wrong turn. Go back and use one of the four patterns above.

    Client secrets as an anti-pattern in production — secrets leak via .env files, copied CI variables, and expire without an owner. Replace with managed identity, workload identity federation, or On-Behalf-Of.
    Client secrets in production are an anti-pattern. Replace with managed identity for Azure compute, workload identity federation for CI/CD, or On-Behalf-Of for signed-in user apps.

    2. The role assignment model — least privilege without the spreadsheet

    Two principles. Roles are assigned to principals — managed identities, user accounts, Entra groups — at a scope. The scope can be project, Foundry resource, resource group, or subscription. Get the scope right and least privilege follows naturally. Get it wrong and you will be re-assigning Contributor every six months because somebody got blocked at a demo.

    In prose, here is the model I deploy:

    Application principals — the managed identity that the production app authenticates as, the federated workload identity the AKS pod assumes — get the Foundry User role, scoped to the project, not the resource. Project-scoped assignments mean a misconfigured app cannot accidentally see another project’s agents, threads, or connections.

    Build and deploy principals — the federated CI/CD identity that runs your GitHub Actions workflow — get Foundry Project Manager scoped to the project. If the same pipeline also creates projects, then it needs a resource-level role for that one operation; keep it as narrow as you can get away with.

    Human developers get Foundry Project Manager on the dev project, Foundry User on staging, and read-only on prod. Production changes go through the pipeline; they do not go through individual developer accounts.

    Resource-level roles — Foundry Account Owner and Foundry Owner — are platform-team territory, and even there they should be PIM-eligible rather than standing assignments. These are the roles that can create new projects, configure guardrails, and conditionally hand out other roles. Treat them accordingly.

    A few practical notes the docs are explicit about. Do not assign built-in roles that start with Cognitive Services for Foundry work — Microsoft’s RBAC documentation calls this out directly. Those roles are for accessing AI Services resources directly and do not apply to Foundry scenarios, even though Foundry sits on the Microsoft.CognitiveServices resource provider. Also avoid the Azure AI Developer role for Foundry — despite the name, it is scoped to Azure Machine Learning workspaces and Foundry hubs, not to Foundry projects or resources.

    One more practical note: reference role definition GUIDs in Bicep and Azure CLI, not display names. The Foundry roles were recently renamed from their Azure AI predecessors (Azure AI User → Foundry User, Azure AI Project Manager → Foundry Project Manager, Azure AI Account Owner → Foundry Account Owner). The GUIDs are stable; the display names are still mid-rollout across the portal and tooling.

    Role assignment model. Application principals get Foundry User at project scope, CI/CD and developer principals get Foundry Project Manager at project scope, and resource-level Foundry Account Owner and Foundry Owner roles stay with the platform team. Avoid Cognitive Services * roles and Azure AI Developer for Foundry work.
    Role assignment model. Application principals get Foundry User on the project, CI/CD and developer principals get Foundry Project Manager, and resource-level Foundry Account Owner / Foundry Owner stay with the platform team. Avoid Cognitive Services * roles and Azure AI Developer for Foundry work.

    3. The multi-environment story

    Dev, staging, and prod each get their own Foundry resource — not just their own project. Quotas are resource-scoped. Network configuration is resource-scoped. The blast radius of a misconfigured role assignment is resource-scoped. All of those argue for full resource separation between non-prod and prod, even if it means three sets of Bicep modules and three Application Insights workspaces. The cost of running an under-utilised dev resource is far less than the cost of an intern accidentally pointing a load test at a prod deployment.

    Each environment gets its own user-assigned managed identity for the application principal, its own federated credential on the CI/CD app registration (one per environment, with a distinct subject claim — environment:dev, environment:prod — so prod deploys only run from protected branches and reviewed environments), and its own Entra group for human access. Group membership rather than direct user assignment, always — that is how you get clean joiner/mover/leaver flows without a quarterly spreadsheet review.

    Secrets that genuinely have to exist — third-party API keys, database connection strings — live in a per-environment Key Vault, accessed by the per-environment managed identity. Foundry credentials themselves are never in Key Vault. They are token exchanges via the patterns in Section 1.

    Elevated roles on the prod resource go through Privileged Identity Management. The platform team holds Foundry Owner on prod as PIM-eligible, not as a standing assignment. Activation requires justification, a time window, and an audit trail. If your auditor asks “who could have changed the prod guardrails on this date,” you want PIM logs to answer that, not Azure Activity Log archaeology.

    Multi-environment isolation. Dev, staging, and production each get their own Foundry resource, user-assigned managed identity, federated credential, and Key Vault. Elevated roles on prod are PIM-eligible only.
    Per-environment isolation. Dev, staging, and production each get their own Foundry resource, user-assigned managed identity, federated credential, and Key Vault. Elevated roles on prod are PIM-eligible only.

    4. The four things that will bite you

    Token caching. The Azure SDK clients cache tokens for the lifetime of the credential object. Long-lived processes — anything stateful, anything that processes a queue, anything with a connection pool — need to handle credential refresh correctly. The right pattern is usually to reuse a single credential instance across all clients in the process, not to recreate DefaultAzureCredential() (or its successor) per call. Recreating it per call defeats the cache and, on a busy worker, will get you rate-limited at the IMDS endpoint before you have shipped a single completion.

    Cross-tenant scenarios. Foundry resources live in a single tenant. If you have a partner tenant whose users need to call your Foundry workload, you are in B2B territory and the patterns above need adapting. Managed identities do not cross tenants without explicit federation, and OBO has its own constraints when the user is a guest. Do not discover this two weeks before a launch — design for the tenant model on day one.

    Private endpoints and DNS. Authentication works, the call still fails. If you have put Foundry behind a private endpoint, the DNS for the resource FQDN must resolve to the private IP from the calling network. Public DNS will look correct, your nslookup from a different network will look correct, and the call from inside the VNet will time out with no useful error. Always check resolution from the calling subnet, not from your laptop.

    Role propagation latency. New role assignments take up to ten minutes to propagate. Pipelines that create a user-assigned managed identity and immediately use it against Foundry will hit 403s on the first run. Options: insert a wait step after role assignment, retry with exponential backoff in the calling code, or assign roles ahead of provisioning the compute they are attached to. I prefer the third — the assignment is declarative and the compute picks it up when it comes online.

    Four gotchas to watch for: stale tokens in long-lived processes, cross-tenant scenarios needing multi-tenant app registrations, private-endpoint DNS resolution failures, and the up-to-ten-minute delay before new role assignments take effect.
    Four things that will bite you in production: stale tokens in long-lived processes, cross-tenant scenarios needing multi-tenant app registrations, private-endpoint DNS failures, and the up-to-ten-minute delay before new role assignments take effect.

    5. When NOT to add another auth pattern

    Counterweight, briefly. If your workload is one App Service calling one Foundry resource for one tenant’s users, deployed by one GitHub Actions workflow, you do not need four patterns. You need a user-assigned managed identity on the App Service and a federated workload identity for the pipeline. Stop there. Adding OBO, custom token exchange, or a second managed identity because “we might need it later” is the kind of architecture work that looks responsible in a design doc and creates three years of operational debt.

    And if you find yourself building a custom token-exchange layer — your own service that sits in front of Foundry and stamps tokens on requests — you are almost certainly reinventing something Entra already does. Read the workload identity federation and OBO docs again before you write more code. The thing you are about to build is probably a federated credential with the wrong subject claim.

    6. Closing

    DefaultAzureCredential is how you start. The patterns in this post are how you scale. Pick the right managed identity flavour for the workload’s lifecycle. Federate your CI/CD so no client secret ever lives on a build agent. Use OBO where the user’s identity has to reach Foundry, and do not use it where it does not. Get the role scope right at the project level. Separate environments by resource, not just by project.

    References

  • Why I built 6 agents instead of 1 mega-agent — lessons from TrafficIQ

    I had two design choices for TrafficIQ: one super-agent holding 56 tools, or six specialist agents sharing them. I picked six. Here is what the one-agent path gets right, where it breaks, and the six lessons I took into production.

    TrafficIQ went on to win Best Use of Microsoft Foundry at the AI Dev Days Hackathon — chosen from 401 projects and 2,041 registrants. The architecture choices below are what made that possible, and what I would actually defend in front of an enterprise architecture review board.

    Why one-agent is genuinely tempting

    The one-agent design is the simpler mental model. One assistant. One system prompt. One thread. One place to debug.

    When you are sketching the first prototype, this is almost always the right move. Orchestration is not free — you have to write a router, define handoff contracts, manage cross-agent state. Skipping all of that gets you to a working demo in an afternoon. Most enterprise teams default here, and for a 10-tool assistant, they are right to.

    The trouble starts later. It starts when the surface area grows past what a single model can hold in its head.

    Where one-agent breaks

    In my experience tool-selection accuracy degrades non-linearly past around 15 to 20 tools. The model does not fail loudly. It fails subtly. It picks get_shipment_status when the user clearly needed check_shipment_status, because the names overlap and the descriptions rhyme. It calls track_shipment when the right answer was get_proof_of_delivery.

    The system prompt becomes the second symptom. To compensate for the confusion, you add disambiguation rules. “Use tool X only when the user mentions Y.” The prompt grows. By the time you have 40 tools, you are nursing a 4,000-token monolith that nobody on the team wants to touch.

    And then there is context-window pressure. Every tool’s JSON schema, every parameter description, every example — it all lives in the agent’s context on every turn. With 56 tools, that alone is enough to crowd out the actual conversation.

    A super-agent does not just get slower. It gets less correct. The failure mode is “looks plausible, called the wrong tool.”

    The architecture I chose

    Six specialist agents, each with a tight tool set scoped to its domain. One orchestrator on top. One router inside the orchestrator. GPT-4.1 under each agent. The whole orchestration layer is built on the Microsoft Foundry SDK — the MultiAgentOrchestrator, the specialists, and the RouterAgent are all SDK-native, using the Foundry Assistants pattern (agent, thread, message, run) end to end.

    TrafficIQ multi-agent architecture — 6 specialist agents and the orchestrator
    TrafficIQ multi-agent architecture — 6 specialist agents and the orchestrator.

    The split is the part most people skip past, so it is worth being concrete:

    • Traffic Agent — 17 tools. Routing, journeys, incidents, reroutes, weather, POI, isochrone, snap-to-road.
    • Supply Chain Agent — 11 tools. Shipments, deliveries, inventory, ETAs, KPIs, proof of delivery. Backed by D365 F&O via the MCP Server.
    • Fleet Agent — 7 tools. Vehicle positions, driver performance, health, maintenance.
    • Operations Agent — 7 tools. Work orders, technician availability, schedule optimisation, returns.
    • Field Service Agent — 7 tools. Service requests, customer assets, SLAs, dispatch, parts.
    • IoT & Logistics Agent — 7 tools. Device health, geofences, driving behaviour, connectivity, batch route alternatives.

    Plus 2 shared tools (navigate_to_page, show_input_form) that every agent can call. That is 56 tools total, none of which any single agent actually has to reason over.

    Coordination sits in a MultiAgentOrchestrator. It runs a three-tier router: sticky → keyword → LLM classifier (the RouterAgent). Each specialist holds its own Foundry thread so its context stays clean. The orchestrator handles handoff when the user pivots from one domain to another.

    Broader TrafficIQ architecture — agents, MCP, Azure services, Dataverse
    Broader TrafficIQ architecture — agents, MCP, Azure services, Dataverse.

    The rest of this post is the six lessons that fell out of building it.

    Lesson 1 — route in tiers, not in one LLM call

    The naive multi-agent router is “ask GPT which agent should handle this.” It works. It is also slow and expensive on every single turn, including the easy ones.

    I run three tiers in order. First, sticky: if the user is mid-thread with the Supply Chain Agent and the next message is “and the one after that?”, stay put. Conversations are usually continuous. The default should be continuity, not re-evaluation.

    Second, keyword. Each agent registers a small set of high-signal terms — “shipment”, “warehouse”, “geofence”, “technician”. A keyword match is effectively free. For roughly the queries you would expect — the obvious ones — this resolves the routing decision in microseconds with no token spend.

    Only when both tiers miss do I fall back to the LLM classifier. That is the RouterAgent, and it is the only model call dedicated to routing. The result is a router that is fast on the common path, accurate on the ambiguous one, and cheap in aggregate. Putting the cheap checks first is the entire trick.

    Lesson 2 — each agent owns its own thread

    This one took me a while to land on, and I think it is the most underrated decision in the whole architecture.

    The obvious approach is to share a single conversation thread across all agents, and have the orchestrator switch which agent reads from it. Do not do this. It is the worst of both worlds. Each agent now sees every tool’s history, including tools it does not own. The tool-set bleed contaminates selection. You also get token bloat: every agent re-reads the entire shared history on every run.

    In TrafficIQ each specialist owns its own thread via the Microsoft Foundry SDK. The Supply Chain Agent’s thread only ever contains Supply Chain turns. Its tool schemas, its system prompt, its prior tool calls — none of it touches the Fleet Agent’s context. Each agent is, effectively, a tightly scoped assistant that does not know the others exist. The SDK’s thread primitive is what makes that isolation cheap to enforce.

    The orchestrator is the only component that knows there are multiple agents. The agents themselves are blissfully ignorant. That isolation is what makes them stay accurate as the system grows.

    Lesson 3 — context handoff is the hard problem, not routing

    Once you have isolated threads, the next question is the obvious one: what happens when the user pivots? “What’s the ETA on that shipment?” — Supply Chain handles it. Then: “And dispatch a tech to the warehouse.” — that is Field Service, and Field Service has no idea what “that shipment” refers to.

    You cannot dump the entire Supply Chain thread on Field Service. That would re-introduce every problem isolated threads were meant to solve. You also cannot hand over nothing — the user is mid-thought and expects continuity.

    What I settled on is a small, deliberate handoff payload: a summary of the last N messages from the source agent, written into the destination agent’s thread as a context message before the user’s new turn lands. Enough grounding to resolve “that shipment”. Not enough to confuse tool selection. The summary is generated by the same Azure OpenAI deployment the agents use, with a tight system prompt — give me entities, IDs, and the last user intent. No prose.

    Routing gets the headlines. Handoff is what actually breaks in production if you get it wrong.

    Lesson 4 — tools must be MECE within an agent, not across all agents

    MECE — mutually exclusive, collectively exhaustive. It is the rule I borrowed from consulting, and it is the cleanest way to think about tool design in a multi-agent system.

    Across the whole platform, similar-sounding tools exist. Traffic’s plan_journey and Supply Chain’s optimize_delivery_route both compute routes. That is fine. They live in different agents and serve different intents — a personal commute is not a multi-stop delivery plan. The router decides which world the user is in. The agent never has to choose between them.

    The rule that actually matters: within one agent, no two tools should be confusable. The Traffic Agent has 17 tools, and I spent more time on their names and descriptions than on any other part of the system. get_traffic_incidents queries an area. monitor_saved_journey watches a specific route. suggest_reroute triggers a recompute. Different verbs, different objects, no overlap.

    If you cannot explain to a junior engineer in one sentence what makes two tools different, the model will not get it right either.

    Lesson 5 — make agents observable from day one

    You cannot debug a multi-agent system from the response text alone. You need to see which agent answered and which tool fired. So the chat panel in TrafficIQ shows both.

    TRAFI chat panel with agent badges and tool-call indicators
    TRAFI chat panel with agent badges and tool-call indicators.

    Every message carries an agent badge — colour-coded per domain. Every tool call streams in real time as a small inline indicator: tool name, parameters, status. When something looks off, I can see immediately whether the routing was wrong, the tool selection was wrong, or the tool itself returned bad data. Three different failure modes, three different fixes, and you cannot tell them apart without the visibility.

    This is not UI polish. I would argue it is the single most important user-trust feature in the product. Users are sceptical of agents — rightly. When they can see “Supply Chain Agent → check_shipment_status → D365 F&O”, the agent stops being a black box. It becomes a transparent process they can audit.

    Build the observability before you build the second agent. You will need it the moment routing decisions start mattering.

    Lesson 6 — ground on enterprise data, not the LLM’s memory

    Every tool in TrafficIQ resolves against a real system of record. D365 F&O via the MCP Server for shipments, inventory, work orders. Azure Maps for routing, traffic, weather, POI. Azure IoT Hub for device health and telemetry. Dataverse for application state.

    The agents never “remember” entities. They look them up. If the user asks about shipment SH-10042, the agent does not summarise what it thinks it knows — it calls check_shipment_status and reads the live record. If GPT-4.1 hallucinates an ETA, the tool result overwrites it.

    That single discipline is what separates a hackathon demo from something an enterprise IT team can own. The model is the reasoning surface. The tools are the truth surface. Keep them strictly separated and the agent’s answers become defensible, auditable, and — most importantly — refreshable when the underlying data changes.

    What I would do differently next time

    Two honest ones.

    First, I would build the router evaluation harness before writing the router. I built it last. I now have a CSV of representative queries with the expected target agent, and it runs as a test suite — but I had to retrofit it after the architecture was already set. If I had started with the eval, I would have caught two keyword collisions weeks earlier.

    Second, I would put a hard token budget on per-agent system prompts from day one. The Traffic Agent’s prompt drifted from 600 tokens to nearly 1,400 over the course of the build, because every new tool came with “and remember to use this when…” instructions. A budget forces the discipline of writing better tool descriptions instead of patching the prompt. Treat the system prompt like a constitution, not a notepad.

    Closing

    The headline is small but the implication is large: when a single agent’s tool surface grows past where its selection accuracy holds, the answer is not a smarter prompt. It is a smaller agent.

    Six specialists with clear scopes, isolated threads, tiered routing, MECE tools, visible execution, and grounded data — that is the recipe that survived production hardening in TrafficIQ. None of it is exotic. All of it is boring engineering applied carefully.

    If you want to see the code, the TrafficIQ repo is on GitHub. The Microsoft winner announcement is here. And the full demo video walks the router, the handoffs, and the tool execution in real time.

    TrafficIQ operational dashboard
    TrafficIQ operational dashboard.
  • 🏆 Winning Best Use of Microsoft Foundry at AI Dev Days Hackathon — TrafficIQ

    🏆 Winning Best Use of Microsoft Foundry at AI Dev Days Hackathon — TrafficIQ

    🏆 Best Use of Microsoft Foundry — Microsoft AI Dev Days Hackathon · 2026

    I am honored to share that TrafficIQ — Supply Chain Transport Intelligence won the Best Use of Microsoft Foundry Project award at Microsoft’s AI Dev Days Hackathon. The hackathon brought together a global community of 2,041 registrants and 401 submitted projects, with winners selected across two Grand Prize categories and four special category awards.

    TrafficIQ Dashboard
    TrafficIQ Dashboard — the operational command centre.

    What TrafficIQ does

    TrafficIQ is an enterprise-grade multi-agent AI platform built entirely on the Microsoft AI Platform. It brings real-time traffic intelligence into fleet, logistics and supply-chain workflows — so dispatchers, drivers and operations leaders can make smarter routing and delivery decisions before disruptions hit the bottom line.

    The Microsoft stack underneath

    • Azure AI Foundry — model hosting, agent orchestration and evaluation
    • Microsoft Agent Framework — multi-agent coordination and tool calling
    • Azure Maps — routing, traffic incidents and geospatial intelligence
    • Azure IoT Hub — fleet GPS telemetry and vehicle sensor streams
    • Dynamics 365 Finance & Operations — orders, shipments and field service
    • MCP (Model Context Protocol) — standardised tool integration across agents
    • Dataverse, Power Apps & Power Automate — the human-in-the-loop UI and workflow layer
    TrafficIQ Multi-Agent Architecture
    The TRAFI multi-agent architecture — 6 specialist agents, 49 composable tools.

    TRAFI — the multi-agent core

    At the heart of TrafficIQ is TRAFI, a multi-agent AI orchestration system with 6 specialist agents and 49 composable tools. The agents proactively monitor traffic incidents, optimise delivery routes, and reduce operational disruptions before they impact supply chains. Each agent owns a clear responsibility — incident monitoring, route planning, ETA recalculation, fleet health, customer notifications, escalation — and they coordinate through the Microsoft Agent Framework.

    What the platform delivers

    • ✅ Real-time traffic awareness across the fleet
    • ✅ Intelligent route optimisation with live re-planning
    • ✅ Fleet GPS visibility and IoT telemetry
    • ✅ Predictive maintenance insights
    • ✅ Automated ETA updates to customers
    • ✅ Field service & inventory management
    • ✅ Enterprise notifications and operational dashboards
    TrafficIQ Delivery Planner
    Delivery Planner — AI-assisted scheduling and route optimisation.
    TrafficIQ Fleet Management
    Fleet Management — live vehicle health and GPS telemetry.
    TrafficIQ Analytics
    Operational analytics — KPIs that decision-makers actually read.
    TrafficIQ AI Chat Agent
    The TRAFI chat agent — natural-language ops co-pilot for dispatchers.

    Why this project

    This project was focused on solving practical enterprise challenges using agentic-AI patterns and Microsoft technologies in a production-oriented architecture. Hackathon code often optimises for the demo. With TrafficIQ I tried to optimise for what would survive a 3-month production hardening cycle: typed contracts between agents, explicit human-in-the-loop checkpoints, and a Dataverse-backed operational model that an enterprise IT team could actually own.

    Links

  • Building LocalRAG — a fully local AI document search

    LocalRAG is a fully local Retrieval-Augmented Generation application I built to answer one question: how much of a useful enterprise RAG can you run without sending a single byte to a cloud LLM?

    The problem

    Most “build a chatbot over your documents” tutorials assume an OpenAI key, a managed vector database and a cloud orchestrator. That’s fine for prototypes — and a dead end the moment you talk to a customer in regulated banking, healthcare or government. They want answers on their data, on their hardware, with no egress.

    The shape of the solution

    LocalRAG uses local Ollama models for both embeddings and generation, FAISS for the vector index, and a content-type-aware ingestion pipeline that handles PDF, DOCX, CSV, Excel, XML and images. Everything runs on a laptop. The full demo is on YouTube.

    • Ingestion: multi-format extractors that preserve enough structure to chunk intelligently — tables stay together, lists stay together, headings become metadata.
    • Indexing: FAISS index with content-type tags so retrieval can prefer the right shape of content for the question.
    • Retrieval: semantic top-k with rate-limited retries and a simple fallback when a model is overloaded.
    • Generation: a local Ollama model with grounded prompts and source citations.

    What I’d do differently next time

    Two things. First, evaluation should be a first-class subsystem from day one, not bolted on later — even a small golden-question set saves you from regression panic during refactors. Second, content-type awareness is more important than fancy reranking; a boring extractor that respects document structure beats a clever reranker that received bad chunks.

    Repo: github.com/PowerAI-Labs/LocalRAG. Feedback and PRs welcome.

  • Welcome to PowerAI Labs

    Welcome to PowerAI Labs — my engineering notebook in public. This is where I’ll write down what I’m building, what I’m breaking, and what survives contact with production on the Microsoft AI stack.

    Why this site exists

    After more than fifteen years shipping enterprise solutions across Dynamics 365, Power Platform, Copilot Studio and Azure AI, I’ve collected a lot of architecture decisions, hard-won lessons and reusable patterns. Most of them live in private Confluence pages, customer engagements and my own notes. PowerAI Labs is where I’m pulling the ones I can share into the open.

    What you’ll find here

    • Architecture deep-dives — reference architectures, decision logs and trade-off analyses for AI agents, RAG, and Power Platform solutions at enterprise scale.
    • Lessons from production — the things that don’t make it into vendor docs: cost surprises, throttling, governance, prompt drift, eval pipelines.
    • Tutorials and walkthroughs — hands-on guides for Copilot Studio, Microsoft 365 Copilot agents, Azure AI Foundry and the rest of the stack.
    • Projects — open-source experiments like LocalRAG, TrafficIQ and MakeLifeEasy.

    The point of view

    AI in the enterprise is exciting and chaotic at the same time. The vendor demos are perfect; the customer environments are not. My bias is towards architectures that are boring on purpose — strongly typed contracts, explicit data lineage, evaluable agents, and a default of “make it observable before you make it autonomous”. Everything I publish here is filtered through that lens.

    Stay in touch

    If a post is useful, share it. If something is wrong, tell me — I’d rather be corrected than confident. You can reach me at contact@powerailabs.dev, on LinkedIn, or via GitHub.

    — Raghav