Here’s a reference architecture for a banking chatbot on Azure OpenAI that’s designed for security, grounding, auditability, and human handoff.

Architecture

Customers / Bank staff
        ↓
Web / Mobile / Contact-center UI
        ↓
API Gateway / WAF
        ↓
Chat Orchestrator (App Service / AKS)
   ├─ Microsoft Entra ID auth
   ├─ session state + rate limiting
   ├─ prompt assembly + policy checks
   ├─ tool calling / workflow engine
   ├─ PII masking / redaction
   └─ escalation to human agent
        ↓
+---------------------------+---------------------------+
|                           |                           |
v                           v                           v
Azure OpenAI           Azure AI Search            Core banking tools/APIs
(chat + embeddings)    (hybrid RAG index)         (CRM, accounts, cards,
                                                 loans, fraud, ticketing)
|                           |
|                           v
|                    Indexed bank knowledge
|                    ├─ policies / FAQs
|                    ├─ product docs
|                    ├─ procedures / SOPs
|                    └─ secure document ACLs
|
v
Response composer
   ├─ citations
   ├─ confidence scoring
   ├─ compliance banners
   └─ allowed action filtering
        ↓
Customer response / human handoff
Supporting services:
- Azure Key Vault
- Azure Monitor / App Insights / Log Analytics
- Microsoft Sentinel
- Private endpoints / VNet integration
- Blob / SharePoint / SQL ingestion pipeline

This structure follows Microsoft’s current baseline enterprise chat architecture for Azure, where the application layer sits in front of the model and retrieval services, uses private networking, and keeps orchestration separate from the model itself. Azure also recommends Azure AI Search as the retrieval layer for RAG, with support for hybrid retrieval, document-level security trimming, and private endpoints. (Microsoft Learn)

What each layer does

1. Channels and identity
Customers access the bot through mobile banking, web banking, or a contact-center console. Use Microsoft Entra ID for workforce users and your bank’s customer identity stack for retail users, then pass identity and entitlement context to the orchestrator. Azure AI Search recommends Entra-based auth and role-based access because it gives centralized identity, conditional access, and stronger audit trails. (Microsoft Learn)

2. Chat orchestrator
This is the most important layer. It handles conversation memory, prompt templates, rate limiting, policy checks, tool access, and handoff to a human agent. Microsoft’s baseline Azure chat reference architecture puts this orchestration layer between the UI and Azure OpenAI rather than letting clients call the model directly. (Microsoft Learn)

3. Azure OpenAI
Use one deployment for the chat model and another for embeddings. The chat model generates answers; the embedding model helps retrieve relevant knowledge chunks. Azure documents content filtering and abuse monitoring as built-in safety controls, which is especially important for regulated customer-facing use. (Microsoft Learn)

4. Azure AI Search for grounding
For banking, do not rely on the model’s memory for policies, fees, disclosures, or procedures. Put approved content into Azure AI Search and use hybrid retrieval so the chatbot answers with grounded content and citations. Microsoft’s current guidance explicitly recommends Azure AI Search for RAG and notes support for security trimming and private network isolation. (Microsoft Learn)

5. Banking systems and tools
The chatbot should not directly expose raw core banking systems to the model. Instead, the orchestrator should call tightly scoped internal APIs for approved actions like “show recent transactions,” “freeze card,” or “open a support case.” That way the model suggests the action, but the backend enforces the rules.

Banking-specific design principles

Grounded answers only for policy and product questions
Use RAG for fees, terms, product comparisons, and internal procedures. This reduces hallucinations and supports citations. Microsoft’s RAG guidance for Azure AI Search emphasizes grounding, citations, and security-aware retrieval. (Microsoft Learn)

Document-level access control
If the chatbot is used by employees, access trimming matters a lot. A branch employee should not retrieve internal audit documents just because they ask. Azure AI Search supports document-level access control and security trimming patterns tied to identity. (Microsoft Learn)

Private networking by default
For a bank, expose as little as possible publicly. Microsoft’s baseline Azure chat architecture uses private endpoints and VNet integration, and Azure OpenAI On Your Data guidance also calls out private networking and restricted access paths. (Microsoft Learn)

Human handoff for sensitive cases
For fraud claims, hardship, complaints, suspicious activity, or low-confidence responses, the bot should escalate to a human banker or contact-center agent instead of improvising.

Audit everything
Send logs, prompts, tool calls, retrieval events, and security events to Azure Monitor and Microsoft Sentinel. Sentinel is Microsoft’s cloud-native SIEM for detection, investigation, and response, which fits banking operational monitoring well. (Microsoft Learn)

Recommended banking use cases

Good first-wave use cases:

• product FAQs
• branch and ATM help
• card controls like freeze/unfreeze
• loan application status
• internal employee knowledge assistant
• secure document Q&A for policies and procedures

Use more caution with:

• personalized financial advice
• transaction disputes
• fraud investigations
• credit decisions
• anything that creates legal or regulatory commitments

Suggested deployment pattern

For a bank, I’d recommend this split:

Customer bot

• retail/mobile/web channels
• heavily restricted tools
• strict content policy
• human handoff early

Employee copilot

• internal knowledge access
• stronger retrieval permissions
• workflow tools for CRM/ticketing
• document-level access trimming

This split reduces risk because customer-facing and employee-facing requirements are usually very different.

Minimal Azure stack

• Frontend: Web app, mobile app, or contact-center console
• Orchestrator: Azure App Service or AKS
• LLM: Azure OpenAI
• Retrieval: Azure AI Search
• Identity: Microsoft Entra ID
• Secrets: Azure Key Vault
• Monitoring: Azure Monitor + Application Insights + Log Analytics
• Security ops: Microsoft Sentinel
• Documents: Blob / SharePoint / SQL ingestion pipeline

This aligns closely with Microsoft’s baseline Foundry chat architecture and Azure AI Search RAG guidance. (Microsoft Learn)

Practical request flow

1. User asks: "What is my mortgage payoff amount?"
2. Orchestrator authenticates user and checks entitlements.
3. If answer needs bank data, orchestrator calls approved internal API.
4. If answer needs policy text, orchestrator queries Azure AI Search.
5. Azure OpenAI generates a grounded response using retrieved data.
6. Response includes citation or disclosure.
7. If confidence is low or request is high-risk, escalate to human agent.

What I would avoid

• letting the frontend call Azure OpenAI directly
• storing sensitive long-term memory in prompts
• giving the model direct unrestricted access to core banking systems
• answering regulated policy questions without retrieval/citations
• using only vector search when hybrid search is available
• treating safety filters as the only compliance control

Best one-line summary

For a banking chatbot on Azure, the safest reference architecture is:

customer/app channel → secure orchestrator → Azure OpenAI + Azure AI Search → tightly scoped banking APIs, all behind private networking with identity-aware retrieval, full logging, and human escalation. (Microsoft Learn)

A solid GenAI chatbot on Azure OpenAI usually looks like this:

User
  ↓
Web / Mobile / Teams UI
  ↓
Backend API / Orchestrator
  ├─ Auth (Microsoft Entra ID)
  ├─ Prompt assembly + guardrails
  ├─ Conversation state
  └─ Tool calling / business logic
        ↓
Azure OpenAI
  ├─ Chat model
  └─ Embedding model
        ↓
Azure AI Search
  ├─ Keyword + vector + semantic retrieval
  └─ Citations / grounding docs
        ↓
Enterprise data sources
  ├─ Blob / SharePoint / SQL / Cosmos DB
  └─ Ingestion + chunking pipeline

For Azure, the most common production pattern is RAG: the app retrieves relevant chunks from your data with Azure AI Search, then sends those chunks to Azure OpenAI so answers stay grounded instead of relying only on model memory. Microsoft specifically recommends Azure AI Search as an index store for RAG, and its current docs distinguish classic RAG from newer agentic retrieval patterns. (Microsoft Learn)

Core components

Frontend
A web app, mobile app, or Teams app handles chat UI, file uploads, citations, and feedback.

Backend / orchestrator
This is the “brain” of the app. It manages auth, session history, prompt templates, retrieval calls, tool use, rate limiting, and logging. In Microsoft’s baseline enterprise chat architecture, the app layer sits in front of the model and retrieval services rather than having the client talk to the model directly. (Microsoft Learn)

Azure OpenAI
Use one deployment for chat and usually another for embeddings. The chat model generates the answer; the embedding model converts documents and queries into vectors for retrieval. Azure OpenAI “On Your Data” exists as a simpler way to ground answers in enterprise content, though Microsoft labels that path as “classic.” (Microsoft Learn)

Azure AI Search
This is the retrieval layer. It supports vector search, semantic ranking, hybrid search, enrichment, and newer agentic retrieval features for chatbot scenarios. Microsoft’s current guidance says Azure AI Search is a recommended retrieval/index layer for RAG workloads. (Microsoft Learn)

Data ingestion pipeline
Documents from Blob, SharePoint, SQL, PDFs, and other sources get extracted, chunked, enriched, and indexed. Azure AI Search supports enrichment for content such as PDFs and images that are not searchable in raw form. (Microsoft Learn)

Best-practice architecture

1. Start with RAG, not pure prompting

For an enterprise chatbot, keep company docs outside the prompt until query time. Store them in Azure AI Search, then retrieve only the relevant chunks for each question. Microsoft’s RAG guidance says this improves grounding and supports citations and better relevance. (Microsoft Learn)

2. Use hybrid retrieval

Use vector + keyword + semantic ranking together. Azure AI Search supports this combination, and it is usually stronger than relying on vectors alone for real-world business documents. (Microsoft Learn)

3. Add identity-aware filtering

If different users should see different documents, put Microsoft Entra ID in front of the app and apply Azure AI Search security filters or document-level access trimming. Microsoft documents this specifically for Azure OpenAI On Your Data with Azure AI Search. (Microsoft Learn)

4. Separate conversation memory from knowledge retrieval

Keep short-term chat history in app storage, but keep source-of-truth business content in the search index. This avoids bloated prompts and makes updates to your knowledge base easier. Microsoft’s baseline chat architecture separates the app/orchestration layer from the grounding data layer. (Microsoft Learn)

5. Prefer managed identity where possible

Microsoft’s Azure App Service RAG tutorial uses managed identities for passwordless authentication between services. That is the cleaner production pattern versus storing secrets in code. (Microsoft Learn)

Two good Azure patterns

Pattern A: Simpler RAG app

Use this when you want a straightforward chatbot fast.

App Service / AKS
  ↓
Backend API
  ↓
Azure AI Search
  ↓
Azure OpenAI

This is the easier option and matches Microsoft’s tutorial-style architecture for grounded chat apps. (Microsoft Learn)

Pattern B: Agent-style chatbot

Use this when you need tool use, more complex reasoning, or multi-step workflows.

UI
  ↓
Foundry Agent Service / custom orchestrator
  ├─ retrieval
  ├─ tools
  ├─ memory
  └─ policy checks
      ↓
Azure OpenAI + Azure AI Search + enterprise APIs

Microsoft’s current architecture guidance includes Foundry Agent Service and a baseline Foundry chat reference architecture for enterprise chat applications. (Microsoft Learn)

What I’d recommend

For most teams:

• Frontend: React or Teams app
• Backend: App Service or AKS
• LLM: Azure OpenAI
• Retrieval: Azure AI Search
• Identity: Entra ID
• Secrets: Key Vault
• Telemetry: Application Insights / Azure Monitor
• Documents: Blob + ingestion pipeline

That gives you a practical, scalable architecture without too much complexity. Azure AI Search is the natural retrieval layer, and Azure’s current enterprise chat reference architectures are built around that same idea. (Microsoft Learn)

Common mistakes

• Letting the frontend call the model directly
• Sending entire documents to the model instead of retrieving chunks
• Skipping citations
• Mixing access control into prompt text instead of enforcing it in retrieval
• Using only vector search when hybrid retrieval would work better
• Treating chat history as your knowledge base

Quick starter version

Users
  ↓
Azure App Service
  ↓
Backend API
  ├─ Entra ID auth
  ├─ prompt templates
  ├─ chat history store
  └─ calls Azure AI Search
        ↓
   top-k chunks + citations
        ↓
    Azure OpenAI
        ↓
   answer to user