Here’s a hands-on Azure AI Search setup using the Azure portal, which is the easiest way to learn it end to end.

The fastest path is:

1. create an Azure AI Search service,
2. put sample files in Blob Storage,
3. use the Import data wizard to create the data source, index, and indexer,
4. run test queries,
5. then optionally add vector search for RAG. Microsoft’s current portal quickstarts support both classic full-text search and integrated vectorization through the Import data wizard. (Microsoft Learn)

What you’re building

You’ll end up with:

• a search service,
• a storage container with documents,
• a search index,
• optionally an indexer that keeps the index updated,
• and a search UI in the portal to test queries. Azure AI Search indexes store searchable documents, and indexers automate ingestion from supported data sources like Blob Storage. (Microsoft Learn)

Option A: simplest hands-on setup with the portal

Step 1: Create an Azure AI Search service

In Azure portal, create a new Azure AI Search resource. Pick your subscription, resource group, region, and pricing tier. Microsoft’s service-creation guide notes you create the service directly in the portal and choose region and tier there. (Microsoft Learn)

Step 2: Prepare sample data in Blob Storage

Create or use an Azure Storage account, then create a container and upload sample files. Microsoft’s full-text portal quickstart uses a Blob container named hotels-sample-data and sample JSON content, but you can also use your own files. (Microsoft Learn)

A simple path is:

• Storage account
• Blob container
• upload JSON, PDFs, or text files

Step 3: Open the Import data wizard

Open your Azure AI Search service, then choose Import data. Microsoft documents this as the no-code path that can create the data source, optional enrichment/vectorization, inferred schema, and load content into an index. (Microsoft Learn)

Step 4: Choose your data source

Select Azure Blob Storage as the source and connect it to your container. Azure AI Search supports both push and pull ingestion models, and the Import wizard uses the pull/indexer path for supported sources. (Microsoft Learn)

Step 5: Pick search type

For a first lab, choose Keyword search. Microsoft’s current portal quickstart explicitly uses keyword search for the basic getting-started experience. (Microsoft Learn)

Step 6: Let the wizard create the index

The wizard will infer a schema and create an index. In Azure AI Search, the index is the searchable structure that stores documents and fields, and each document needs a unique key field. (Microsoft Learn)

Step 7: Run the indexer

The wizard can create and run an indexer, which automates loading content from the source into the index. Microsoft’s indexer documentation describes indexers as the automation layer for ingestion from supported data sources. (Microsoft Learn)

Step 8: Test queries

Go to Indexes, open your new index, and try some searches in Search Explorer. Azure AI Search supports full-text querying over the indexed content once documents are loaded. (Microsoft Learn)

What just got created

The portal flow usually creates:

• a data source: connection to Blob Storage,
• an index: searchable schema and content,
• an indexer: scheduled or on-demand ingestion job.

Those are the main building blocks for classic portal-based setup. (Microsoft Learn)

Option B: hands-on setup for RAG and vector search

If your goal is chatbot/RAG, use the vector quickstart in the portal instead of the basic keyword-only wizard.

Step 1: Start with the Import data wizard for vectors

Microsoft’s vector portal quickstart says the wizard can do integrated vectorization, including chunking content and calling an embedding model at indexing and query time. (Microsoft Learn)

Step 2: Provide supported files

The vector quickstart uses text PDFs and simple images from Microsoft sample data, but you can also use your own files. (Microsoft Learn)

Step 3: Configure embeddings

For vector search, Azure AI Search stores vectors at the field level and can combine vector and nonvector fields in the same index. Microsoft recommends hybrid search for many production scenarios because vector and keyword queries run in parallel and return one merged result set. (Microsoft Learn)

Step 4: Run test vector or hybrid queries

Once the vector index is built, test either:

• semantic similarity search,
• or hybrid search with both text and vectors.

Microsoft’s vector overview says hybrid search often gives better results than vector-only or keyword-only search. (Microsoft Learn)

When to use indexers vs push ingestion

Use the portal/indexer approach when:

• your content sits in supported sources like Blob Storage,
• you want a fast setup,
• you want scheduled ingestion.

Use push ingestion from code when:

• your data changes very frequently,
• you need tighter control over updates,
• or your application already owns the ingestion process.

Microsoft’s ingestion guidance explains that indexers are the pull model, while push lets your app send documents directly and is better for low-latency sync scenarios. (Microsoft Learn)

A good first lab

A practical first exercise is:

• create a search service,
• upload 10–20 PDF or text files to Blob Storage,
• run the Import data wizard,
• create a basic keyword index,
• test queries,
• then repeat with the vector wizard and compare results.

That mirrors Microsoft’s current quickstart progression from full-text search to vector search and semantic ranking. (Microsoft Learn)

Common mistakes

The biggest ones are:

• choosing the wrong tier or region when creating the service, since service creation is tied to those settings in the portal, (Microsoft Learn)
• expecting Azure AI Search to search your files before they are loaded into an index, because queries run over indexed content, (Microsoft Learn)
• skipping a unique key field in the index schema, which Azure AI Search requires for documents, (Microsoft Learn)
• using vector-only search when hybrid would usually retrieve better business results. (Microsoft Learn)

Best next step after setup

After the basic setup works, the next useful upgrade is:

• add semantic ranking, or
• add integrated vectorization for RAG.

Microsoft provides separate quickstarts for both. (Microsoft Learn)

Here’s a reference architecture for a banking chatbot on Azure OpenAI that’s designed for security, grounding, auditability, and human handoff.

Architecture

Customers / Bank staff
        ↓
Web / Mobile / Contact-center UI
        ↓
API Gateway / WAF
        ↓
Chat Orchestrator (App Service / AKS)
   ├─ Microsoft Entra ID auth
   ├─ session state + rate limiting
   ├─ prompt assembly + policy checks
   ├─ tool calling / workflow engine
   ├─ PII masking / redaction
   └─ escalation to human agent
        ↓
+---------------------------+---------------------------+
|                           |                           |
v                           v                           v
Azure OpenAI           Azure AI Search            Core banking tools/APIs
(chat + embeddings)    (hybrid RAG index)         (CRM, accounts, cards,
                                                 loans, fraud, ticketing)
|                           |
|                           v
|                    Indexed bank knowledge
|                    ├─ policies / FAQs
|                    ├─ product docs
|                    ├─ procedures / SOPs
|                    └─ secure document ACLs
|
v
Response composer
   ├─ citations
   ├─ confidence scoring
   ├─ compliance banners
   └─ allowed action filtering
        ↓
Customer response / human handoff
Supporting services:
- Azure Key Vault
- Azure Monitor / App Insights / Log Analytics
- Microsoft Sentinel
- Private endpoints / VNet integration
- Blob / SharePoint / SQL ingestion pipeline

This structure follows Microsoft’s current baseline enterprise chat architecture for Azure, where the application layer sits in front of the model and retrieval services, uses private networking, and keeps orchestration separate from the model itself. Azure also recommends Azure AI Search as the retrieval layer for RAG, with support for hybrid retrieval, document-level security trimming, and private endpoints. (Microsoft Learn)

What each layer does

1. Channels and identity
Customers access the bot through mobile banking, web banking, or a contact-center console. Use Microsoft Entra ID for workforce users and your bank’s customer identity stack for retail users, then pass identity and entitlement context to the orchestrator. Azure AI Search recommends Entra-based auth and role-based access because it gives centralized identity, conditional access, and stronger audit trails. (Microsoft Learn)

2. Chat orchestrator
This is the most important layer. It handles conversation memory, prompt templates, rate limiting, policy checks, tool access, and handoff to a human agent. Microsoft’s baseline Azure chat reference architecture puts this orchestration layer between the UI and Azure OpenAI rather than letting clients call the model directly. (Microsoft Learn)

3. Azure OpenAI
Use one deployment for the chat model and another for embeddings. The chat model generates answers; the embedding model helps retrieve relevant knowledge chunks. Azure documents content filtering and abuse monitoring as built-in safety controls, which is especially important for regulated customer-facing use. (Microsoft Learn)

4. Azure AI Search for grounding
For banking, do not rely on the model’s memory for policies, fees, disclosures, or procedures. Put approved content into Azure AI Search and use hybrid retrieval so the chatbot answers with grounded content and citations. Microsoft’s current guidance explicitly recommends Azure AI Search for RAG and notes support for security trimming and private network isolation. (Microsoft Learn)

5. Banking systems and tools
The chatbot should not directly expose raw core banking systems to the model. Instead, the orchestrator should call tightly scoped internal APIs for approved actions like “show recent transactions,” “freeze card,” or “open a support case.” That way the model suggests the action, but the backend enforces the rules.

Banking-specific design principles

Grounded answers only for policy and product questions
Use RAG for fees, terms, product comparisons, and internal procedures. This reduces hallucinations and supports citations. Microsoft’s RAG guidance for Azure AI Search emphasizes grounding, citations, and security-aware retrieval. (Microsoft Learn)

Document-level access control
If the chatbot is used by employees, access trimming matters a lot. A branch employee should not retrieve internal audit documents just because they ask. Azure AI Search supports document-level access control and security trimming patterns tied to identity. (Microsoft Learn)

Private networking by default
For a bank, expose as little as possible publicly. Microsoft’s baseline Azure chat architecture uses private endpoints and VNet integration, and Azure OpenAI On Your Data guidance also calls out private networking and restricted access paths. (Microsoft Learn)

Human handoff for sensitive cases
For fraud claims, hardship, complaints, suspicious activity, or low-confidence responses, the bot should escalate to a human banker or contact-center agent instead of improvising.

Audit everything
Send logs, prompts, tool calls, retrieval events, and security events to Azure Monitor and Microsoft Sentinel. Sentinel is Microsoft’s cloud-native SIEM for detection, investigation, and response, which fits banking operational monitoring well. (Microsoft Learn)

Recommended banking use cases

Good first-wave use cases:

• product FAQs
• branch and ATM help
• card controls like freeze/unfreeze
• loan application status
• internal employee knowledge assistant
• secure document Q&A for policies and procedures

Use more caution with:

• personalized financial advice
• transaction disputes
• fraud investigations
• credit decisions
• anything that creates legal or regulatory commitments

Suggested deployment pattern

For a bank, I’d recommend this split:

Customer bot

• retail/mobile/web channels
• heavily restricted tools
• strict content policy
• human handoff early

Employee copilot

• internal knowledge access
• stronger retrieval permissions
• workflow tools for CRM/ticketing
• document-level access trimming

This split reduces risk because customer-facing and employee-facing requirements are usually very different.

Minimal Azure stack

• Frontend: Web app, mobile app, or contact-center console
• Orchestrator: Azure App Service or AKS
• LLM: Azure OpenAI
• Retrieval: Azure AI Search
• Identity: Microsoft Entra ID
• Secrets: Azure Key Vault
• Monitoring: Azure Monitor + Application Insights + Log Analytics
• Security ops: Microsoft Sentinel
• Documents: Blob / SharePoint / SQL ingestion pipeline

This aligns closely with Microsoft’s baseline Foundry chat architecture and Azure AI Search RAG guidance. (Microsoft Learn)

Practical request flow

1. User asks: "What is my mortgage payoff amount?"
2. Orchestrator authenticates user and checks entitlements.
3. If answer needs bank data, orchestrator calls approved internal API.
4. If answer needs policy text, orchestrator queries Azure AI Search.
5. Azure OpenAI generates a grounded response using retrieved data.
6. Response includes citation or disclosure.
7. If confidence is low or request is high-risk, escalate to human agent.

What I would avoid

• letting the frontend call Azure OpenAI directly
• storing sensitive long-term memory in prompts
• giving the model direct unrestricted access to core banking systems
• answering regulated policy questions without retrieval/citations
• using only vector search when hybrid search is available
• treating safety filters as the only compliance control

Best one-line summary

For a banking chatbot on Azure, the safest reference architecture is:

customer/app channel → secure orchestrator → Azure OpenAI + Azure AI Search → tightly scoped banking APIs, all behind private networking with identity-aware retrieval, full logging, and human escalation. (Microsoft Learn)

A solid GenAI chatbot on Azure OpenAI usually looks like this:

User
  ↓
Web / Mobile / Teams UI
  ↓
Backend API / Orchestrator
  ├─ Auth (Microsoft Entra ID)
  ├─ Prompt assembly + guardrails
  ├─ Conversation state
  └─ Tool calling / business logic
        ↓
Azure OpenAI
  ├─ Chat model
  └─ Embedding model
        ↓
Azure AI Search
  ├─ Keyword + vector + semantic retrieval
  └─ Citations / grounding docs
        ↓
Enterprise data sources
  ├─ Blob / SharePoint / SQL / Cosmos DB
  └─ Ingestion + chunking pipeline

For Azure, the most common production pattern is RAG: the app retrieves relevant chunks from your data with Azure AI Search, then sends those chunks to Azure OpenAI so answers stay grounded instead of relying only on model memory. Microsoft specifically recommends Azure AI Search as an index store for RAG, and its current docs distinguish classic RAG from newer agentic retrieval patterns. (Microsoft Learn)

Core components

Frontend
A web app, mobile app, or Teams app handles chat UI, file uploads, citations, and feedback.

Backend / orchestrator
This is the “brain” of the app. It manages auth, session history, prompt templates, retrieval calls, tool use, rate limiting, and logging. In Microsoft’s baseline enterprise chat architecture, the app layer sits in front of the model and retrieval services rather than having the client talk to the model directly. (Microsoft Learn)

Azure OpenAI
Use one deployment for chat and usually another for embeddings. The chat model generates the answer; the embedding model converts documents and queries into vectors for retrieval. Azure OpenAI “On Your Data” exists as a simpler way to ground answers in enterprise content, though Microsoft labels that path as “classic.” (Microsoft Learn)

Azure AI Search
This is the retrieval layer. It supports vector search, semantic ranking, hybrid search, enrichment, and newer agentic retrieval features for chatbot scenarios. Microsoft’s current guidance says Azure AI Search is a recommended retrieval/index layer for RAG workloads. (Microsoft Learn)

Data ingestion pipeline
Documents from Blob, SharePoint, SQL, PDFs, and other sources get extracted, chunked, enriched, and indexed. Azure AI Search supports enrichment for content such as PDFs and images that are not searchable in raw form. (Microsoft Learn)

Best-practice architecture

1. Start with RAG, not pure prompting

For an enterprise chatbot, keep company docs outside the prompt until query time. Store them in Azure AI Search, then retrieve only the relevant chunks for each question. Microsoft’s RAG guidance says this improves grounding and supports citations and better relevance. (Microsoft Learn)

2. Use hybrid retrieval

Use vector + keyword + semantic ranking together. Azure AI Search supports this combination, and it is usually stronger than relying on vectors alone for real-world business documents. (Microsoft Learn)

3. Add identity-aware filtering

If different users should see different documents, put Microsoft Entra ID in front of the app and apply Azure AI Search security filters or document-level access trimming. Microsoft documents this specifically for Azure OpenAI On Your Data with Azure AI Search. (Microsoft Learn)

4. Separate conversation memory from knowledge retrieval

Keep short-term chat history in app storage, but keep source-of-truth business content in the search index. This avoids bloated prompts and makes updates to your knowledge base easier. Microsoft’s baseline chat architecture separates the app/orchestration layer from the grounding data layer. (Microsoft Learn)

5. Prefer managed identity where possible

Microsoft’s Azure App Service RAG tutorial uses managed identities for passwordless authentication between services. That is the cleaner production pattern versus storing secrets in code. (Microsoft Learn)

Two good Azure patterns

Pattern A: Simpler RAG app

Use this when you want a straightforward chatbot fast.

App Service / AKS
  ↓
Backend API
  ↓
Azure AI Search
  ↓
Azure OpenAI

This is the easier option and matches Microsoft’s tutorial-style architecture for grounded chat apps. (Microsoft Learn)

Pattern B: Agent-style chatbot

Use this when you need tool use, more complex reasoning, or multi-step workflows.

UI
  ↓
Foundry Agent Service / custom orchestrator
  ├─ retrieval
  ├─ tools
  ├─ memory
  └─ policy checks
      ↓
Azure OpenAI + Azure AI Search + enterprise APIs

Microsoft’s current architecture guidance includes Foundry Agent Service and a baseline Foundry chat reference architecture for enterprise chat applications. (Microsoft Learn)

What I’d recommend

For most teams:

• Frontend: React or Teams app
• Backend: App Service or AKS
• LLM: Azure OpenAI
• Retrieval: Azure AI Search
• Identity: Entra ID
• Secrets: Key Vault
• Telemetry: Application Insights / Azure Monitor
• Documents: Blob + ingestion pipeline

That gives you a practical, scalable architecture without too much complexity. Azure AI Search is the natural retrieval layer, and Azure’s current enterprise chat reference architectures are built around that same idea. (Microsoft Learn)

Common mistakes

• Letting the frontend call the model directly
• Sending entire documents to the model instead of retrieving chunks
• Skipping citations
• Mixing access control into prompt text instead of enforcing it in retrieval
• Using only vector search when hybrid retrieval would work better
• Treating chat history as your knowledge base

Quick starter version

Users
  ↓
Azure App Service
  ↓
Backend API
  ├─ Entra ID auth
  ├─ prompt templates
  ├─ chat history store
  └─ calls Azure AI Search
        ↓
   top-k chunks + citations
        ↓
    Azure OpenAI
        ↓
   answer to user