OpenShift and AI is one of the fastest-growing areas in enterprise IT. Red Hat has built an entire AI platform around OpenShift, making it possible to run Generative AI, Machine Learning, MLOps, and AI inference on the same Kubernetes platform that hosts enterprise applications.
For an OpenShift Architect interview, you should understand not only AI concepts but also how OpenShift integrates with AI services.
OpenShift AI Ecosystem
Users
│
Web / Mobile Apps
│
OpenShift Routes
│
------------------------
│ │
Business Apps AI Applications
│ │
└──────────────┬───────┘
│
OpenShift Platform
│
------------------------------------------------------
│ │ │ │
RHOAI OpenShift GPU Nodes OpenShift GitOps
Pipelines
│
▼
Kubeflow / Jupyter / Model Serving
│
▼
LLM (Llama, Granite, Mistral, GPT, etc.)
│
▼
Vector Database / AI Search
│
▼
Enterprise Data Sources
What is Red Hat OpenShift AI (RHOAI)?
Red Hat OpenShift AI (formerly Red Hat OpenShift Data Science) is Red Hat’s enterprise AI/ML platform built on OpenShift.
It provides:
- Jupyter Notebooks
- Model training
- Model serving
- Pipelines
- Experiment tracking
- GPU scheduling
- AI model deployment
- MLOps
- LLM serving
- Vector database integration
Think of it as:
OpenShift +AI Platform +MLOps
AI Architecture on OpenShift
Developers
│
Git Repository
│
OpenShift GitOps
│
AI Pipeline (Tekton)
│
------------------------
│ │
Train Model Build Container
│ │
------------------------
│
Model Registry
│
Model Serving
│
REST API
│
Applications
Major AI Components
1. Jupyter Notebooks
Used for:
- Data science
- Python
- TensorFlow
- PyTorch
- Hugging Face
- Experiments
Example:
Data Scientist↓Notebook↓Python↓Train Model
2. Model Serving
Once trained:
Model↓Model Server↓REST API↓Application
Supported technologies include:
- KServe
- vLLM (commonly used for LLM inference)
- NVIDIA Triton Inference Server
- Caikit (used in some Red Hat AI scenarios)
3. Pipelines
Uses Kubeflow Pipelines (or Tekton depending on workflow).
Example:
Dataset↓Preprocessing↓Training↓Evaluation↓Deploy↓Production
Everything becomes repeatable.
GPU Support
OpenShift schedules GPUs like any other resource.
Architecture:
GPU Node↓NVIDIA Driver↓NVIDIA Device Plugin↓Pod Requests GPU↓GPU Allocated
Example Pod:
resources: limits: nvidia.com/gpu: 1
AI Model Lifecycle
Collect Data↓Train Model↓Validate↓Containerize↓Deploy↓Monitor↓Retrain
This is called MLOps.
OpenShift AI with LLMs
You can deploy models like:
- IBM Granite
- Llama 3.x
- Mistral
- DeepSeek (where licensing and hardware permit)
- Gemma
- Phi
Architecture:
User↓Chat Application↓OpenShift Route↓Model Server↓LLM↓Response
RAG (Retrieval-Augmented Generation)
This is one of the most common enterprise AI architectures.
User Question↓Embedding Model↓Vector Database↓Relevant Documents↓LLM↓Answer
Instead of relying only on the model’s knowledge, the LLM searches company documents.
Enterprise Banking AI Example
Customer↓Chatbot↓OpenShift Route↓Authentication↓API Gateway↓LLM Gateway↓RAG↓Vector Database↓Bank Policies↓Azure AI Search /OpenSearch /Milvus /PgVector↓Response
This keeps answers grounded in enterprise data rather than only the model’s pretraining.
AI Security
Security is critical.
Authentication
- OAuth
- OpenID Connect
- LDAP
- SSO
Authorization
RBAC
Data Scientist↓Namespace↓Notebook↓GPU
Network
NetworkPolicies isolate:
- Model servers
- Databases
- Pipelines
- Notebooks
Secrets
Store:
- API Keys
- OpenAI keys
- Hugging Face tokens
- Database credentials
using Kubernetes Secrets or an external secret manager such as HashiCorp Vault or cloud-native secret services.
AI Observability
Monitor:
- GPU utilization
- CPU
- Memory
- Inference latency
- Token generation rate
- Request throughput
- Error rates
Using:
Prometheus↓Grafana↓Alertmanager
Logging:
Vector↓LokiorSplunkorElasticsearch
AI Storage
Training:
- S3
- Ceph
- OpenShift Data Foundation
- NFS
Model storage:
Model Registry↓Object Storage
AI Networking
Inference traffic:
Client↓Route↓Model Service↓LLM
Training traffic:
Notebook↓Object Storage↓GPU Worker
AI Scaling
Model serving uses Kubernetes autoscaling.
Traffic↓HPA↓2 Pods↓5 Pods↓20 Pods
For LLMs, scaling decisions often also consider GPU availability and model loading time.
GitOps for AI
Everything is stored in Git.
Git↓ArgoCD↓Notebook↓Pipeline↓Model↓Serving
AI CI/CD
Git Push↓Tekton↓Train↓Test↓Build↓Deploy
AI Monitoring
Monitor:
GPUInferenceLatencyMemoryModel AccuracyToken UsageFailures
In production, you should also monitor:
- Model drift
- Data drift
- Hallucination rates (where measurable)
- Business KPIs
AI Governance
Enterprise AI requires:
- Model versioning
- Approval workflows
- Audit logging
- Data lineage
- Dataset versioning
- Explainability where required
- Compliance controls
AI Operators
Common Operators include:
- NVIDIA GPU Operator
- Red Hat OpenShift AI Operator
- OpenShift Pipelines Operator
- OpenShift GitOps Operator
- Service Mesh Operator (optional)
- OpenTelemetry Operator
AI + OpenShift Architecture
Users
│
Web / Mobile
│
OpenShift Route
│
API Gateway
│
Authentication (OAuth)
│
-----------------------
│ │
▼ ▼
Business APIs AI Inference API
│
Model Serving
│
------------------------
│ │
LLM (Granite/Llama) Embedding Model
│ │
└──────────┬───────────┘
▼
Vector Database
│
Enterprise Documents
│
S3 / ODF / Database
OpenShift AI vs Azure OpenAI
| OpenShift AI | Azure OpenAI |
|---|---|
| Run models on your infrastructure | Managed AI service |
| Full Kubernetes control | Microsoft-managed |
| GPU management required | No GPU management |
| Supports multiple open models | Microsoft-hosted models |
| Air-gapped deployments possible | Cloud service |
| Better for hybrid/on-prem | Better for Azure-native workloads |
Many enterprises use both:
- Azure OpenAI for managed GPT models.
- OpenShift AI for on-premises inference, data sovereignty, or running open models.
Interview Questions
What is OpenShift AI?
Enterprise AI platform built on OpenShift for model development, training, deployment, and lifecycle management.
Why OpenShift for AI?
- Kubernetes-native
- GPU orchestration
- MLOps
- Security
- Multi-cloud
- GitOps
- Scalability
- Enterprise support
How do you deploy an LLM?
- Deploy GPU Operator.
- Configure GPU nodes.
- Deploy the model server (e.g., vLLM).
- Download or mount the model.
- Expose via a Service and Route.
- Monitor latency and resource usage.
How would you secure AI?
- RBAC
- NetworkPolicies
- OAuth/OIDC
- Secrets management
- Image signing
- Signed models where supported
- Audit logging
- TLS everywhere
Enterprise Banking AI Architecture
Customer
│
Mobile App
│
OpenShift Route
│
API Gateway
│
Authentication (OAuth/OIDC)
│
AI Gateway Service
│
------------------------
│ │
▼ ▼
LLM Inference Embedding Model
│ │
└──────────┬───────────┘
▼
Vector Database
│
----------------------------
│ │ │
▼ ▼ ▼
Banking Docs Policies Knowledge Base
Interview Answer (2 Minutes)
“OpenShift provides an enterprise platform for building, deploying, and operating AI applications through Red Hat OpenShift AI. It supports the full MLOps lifecycle, including Jupyter notebooks for development, pipelines for model training, model registries, and scalable model serving. For generative AI, I typically design a Retrieval-Augmented Generation architecture where applications authenticate through OpenShift, call an API or model-serving layer, retrieve relevant enterprise documents from a vector database, and then pass that context to an LLM such as IBM Granite or Llama. OpenShift handles GPU scheduling through the NVIDIA GPU Operator, while GitOps, RBAC, NetworkPolicies, Secrets, and the monitoring stack provide secure and repeatable operations. For regulated industries like banking, I recommend combining RAG, centralized audit logging, strong identity controls, and model governance so that AI responses are based on approved enterprise knowledge and meet compliance requirements.”