OpenShift 4

Architecture Deep Dive

Ayush Garg | PTSE (Products AI and OpenShift)

Session Overview

01

Platform Overview

OpenShift 4 foundation, design principles, and key differentiators from vanilla Kubernetes

02

Control Plane Architecture

Control plane components, etcd cluster, and Machine Config Operator

03

Worker Node Architecture

Kubelet, CRI-O container runtime, and node components

04

Operator Framework

Operator pattern, OLM, and cluster operators

05

Networking Stack

OVN-Kubernetes CNI, Routes, and Ingress

06

Storage Architecture

CSI drivers, persistent storage, and storage classes

07

Installation & Updates

IPI/UPI deployment and over-the-air updates with CVO

08

Security Framework

SCC, RBAC, and integrated authentication

09

Observability Stack

Prometheus, Grafana, Loki logging, and tracing

10

Future Roadmap

OpenShift 4.20+ innovations and what's next

Architecture Deep Dive

OpenShift 4 Complete Architecture

🌐 External Traffic & DNS

🌍

External DNS

api.cluster.example.com
api-int.cluster.example.com
*.apps.cluster.example.com

💻

Clients

oc/kubectl
Web Console
Applications

⚖️ Load Balancers

🔀

API Load Balancer

Port 6443 (External/Internal)

Targets:

• Master-0:6443

• Master-1:6443

• Master-2:6443

→ kube-apiserver pods

⚙️

Machine Config LB

Port 22623 (Internal)

Targets:

• Master-0:22623

• Master-1:22623

• Master-2:22623

→ machine-config-server

🌐

Ingress Load Balancer

Port 80/443 (External)

Targets:

• Worker-0:80/443

• Worker-1:80/443

• Worker-2+:80/443

→ Router Pods (HAProxy)

🎛️ Control Plane Nodes (Masters)

🎛️ Master-0

kube-apiserver

Port 6443 | REST API

etcd

Port 2379/2380 | Raft

kube-scheduler

Pod scheduling

kube-controller-manager

Controllers

machine-config-server

Port 22623 | Ignition

openshift-apiserver

OCP API extensions

🎛️ Master-1

kube-apiserver

Port 6443 | REST API

etcd

Port 2379/2380 | Raft

kube-scheduler

Pod scheduling

kube-controller-manager

Controllers

machine-config-server

Port 22623 | Ignition

openshift-apiserver

OCP API extensions

🎛️ Master-2

kube-apiserver

Port 6443 | REST API

etcd

Port 2379/2380 | Raft

kube-scheduler

Pod scheduling

kube-controller-manager

Controllers

machine-config-server

Port 22623 | Ignition

openshift-apiserver

OCP API extensions

🔒 etcd Raft Quorum: 3-node cluster tolerates 1 failure | Leader election | Log replication

⚙️ Worker Nodes (Compute)

⚙️ Worker-0

kubelet

Port 10250 | Node agent

CRI-O Runtime

Container runtime

OVN-Kubernetes CNI

Networking / OVS

Router Pod (HAProxy)

Port 80/443 | Ingress

Application Pods

User workloads

⚙️ Worker-1

kubelet

Port 10250 | Node agent

CRI-O Runtime

Container runtime

OVN-Kubernetes CNI

Networking / OVS

Router Pod (HAProxy)

Port 80/443 | Ingress

Application Pods

User workloads

⚙️ Worker-2+

kubelet

Port 10250 | Node agent

CRI-O Runtime

Container runtime

OVN-Kubernetes CNI

Networking / OVS

Router Pod (HAProxy)

Port 80/443 | Ingress

Application Pods

User workloads

🔧 Internal Infrastructure Services

🔌

Internal DNS

CoreDNS
Service discovery

📦

Image Registry

Internal registry
Image streams

📊

Monitoring

Prometheus
Grafana

📝

Logging

Loki
Vector

📡 Traffic Flows:

─── API Traffic (6443): Clients → API LB → Masters (kube-apiserver)

─── Machine Config (22623): Nodes → MCS LB → Masters (MCS)

─── Application Traffic (80/443): Users → Ingress LB → Workers (Router Pods)

─── etcd Replication (2379/2380): Master ↔ Master (Raft consensus)

─── Internal DNS: Pods → CoreDNS → Service resolution

─── CNI Traffic: Pod ↔ Pod (OVN-Kubernetes overlay)

Architecture Overview

High Availability Architecture

🛰️

Control Plane Nodes (Masters)

Master-0

kube-apiserver • etcd • scheduler • controller-manager

Master-1

kube-apiserver • etcd • scheduler • controller-manager

Master-2

kube-apiserver • etcd • scheduler • controller-manager

3 Master Nodes: etcd Raft quorum for HA

⚙️

Worker Nodes (Compute)

Worker-0

kubelet • CRI-O • Router Pod • App Pods

Worker-1

kubelet • CRI-O • Router Pod • App Pods

Worker-2+

kubelet • CRI-O • Router Pod • App Pods

2+ Workers: Pod distribution and workload execution

Architecture Overview

Port Reference & Load Balancers

6443

TCP

Kubernetes API Server

kubectl, oc, cluster communications

22623

TCP

Machine Config Server

RHCOS Ignition configurations

443

TCP

HTTPS Ingress / Console

Routes, Web Console access

80

TCP

HTTP Ingress

Routes (redirects to 443)

10250

TCP

Kubelet

Node metrics, exec operations

2379-2380

TCP

etcd Client/Peer

Control plane internal comms

High Availability Setup

• API LB: Active/Passive or Active/Active configuration

• Ingress LB: HAProxy with Router pods distribution

• MCS LB: Machine Config Server for node bootstrapping

Chapter 01

Platform Overview

Enterprise Kubernetes Platform

OpenShift Container Platform 4 is a comprehensive enterprise Kubernetes platform built on Kubernetes with Red Hat's enterprise enhancements. It provides a complete application platform for developing, deploying, and managing containerized applications at scale.

🎯

Control Plane

Manages cluster state & workloads with automated operations

⚙️

Worker Nodes

Execute container workloads with optimized performance

🔧

RHCOS

Purpose-built OS with immutable infrastructure and automatic updates

                Key Insight: OpenShift 4 uses an Operator-based management model where all cluster operations are automated through Kubernetes-native controllers, enabling declarative cluster management and seamless updates.
            

Chapter 01

RHCOS: Purpose-Built OS

Red Hat Enterprise Linux CoreOS

A container-optimized operating system designed specifically for OpenShift 4.

Immutable Infrastructure: Read-only root filesystem
Ignition: First-boot configuration system
CRI-O: Kubernetes-native container runtime
Automatic Updates: Managed by Machine Config Operator

Architecture Stack

Applications & Workloads
Pods, Deployments, Services, Routes

OpenShift Platform Layer
Web Console, CLI, Image Registry, Build Pipelines

Operator Framework
OLM, Cluster Operators, Custom Resources

Kubernetes Core & Runtime
API Server, Scheduler, CRI-O, RHCOS, Kubelet

Chapter 01

OpenShift vs Kubernetes

🔒

Security Context Constraints

Fine-grained security policies controlling pod capabilities, more granular than Kubernetes Pod Security Standards

🌐

OpenShift Routes

Simpler, more powerful way to expose services with built-in HAProxy, path-based routing, and TLS termination

🔄

Integrated CI/CD

BuildConfigs, ImageStreams, and Source-to-Image (S2I) for building containers directly from source code

🎨

Web Console

Rich web console with visual topology views, integrated monitoring, and one-click deployments

🤖

Operator Framework

Native OLM automates deployment, updates, and lifecycle management of applications

📦

Integrated Registry

Built-in container image registry with automatic image stream tracking and integrated authentication

                Latest Release
                OpenShift 4.20 introduces the 2+1 Arbiter control plane topology for edge deployments, Hosted Control Planes for multi-cluster management, and enhanced AI/ML workload support with NVIDIA GPU operator integration.
            

Chapter 02

Control Plane Components

kube-apiserver

Central management point exposing Kubernetes API. All cluster components communicate through it, validating and configuring data for API objects.

etcd Cluster

Distributed key-value store storing all cluster state. Uses Raft consensus for HA, providing single source of truth.

kube-scheduler

Watches for new pods and assigns them to nodes based on resource availability, affinity rules, and constraints.

kube-controller-manager

Runs controllers regulating cluster state: node, replication, endpoints, and service account controllers.

openshift-apiserver

Extends K8s API with OpenShift resources (Routes, Projects, ImageStreams).

openshift-controller-manager

Handles BuildConfigs, ImageStreams, and OpenShift-specific controllers.

High Availability

Control plane components run as static pods on all master nodes, managed by kubelet. API servers are load-balanced, etcd uses Raft consensus.

Chapter 02

etcd: The Cluster's Source of Truth

What is etcd?

A distributed, reliable key-value store that persistently stores all Kubernetes cluster data.

Key-Value Store: Simple, fast data model
Distributed: Multi-node replication
Consistent: Strong consistency (CP)
Watch Support: Real-time change notifications

Raft Consensus Algorithm

Leader Election: One leader handles writes
Log Replication: Leader replicates to followers
Quorum: Majority needed for consensus

etcd Cluster Topology

etcd-0 (Leader)
LEADER - Handles all write operations, replicates to followers

etcd-1
Follower - Read operations

etcd-2
Follower - Read operations

                        Quorum Requirement: 3-node cluster tolerates 1 failure (2 nodes needed for quorum)
                    

💾

Backup & Restore

Regular snapshots for disaster recovery

⚡

Performance

SSD storage recommended for low latency

🔐

Security

TLS encryption for peer & client communication

Chapter 02

NEW in 4.20: 2+1 Arbiter Topology

GA in 4.20

Cost-Optimized HA Control Plane for Edge Deployments

2

Full Masters

Complete control plane functionality

+1

Arbiter Node

Lightweight etcd voting member

3

etcd Members

Maintains quorum with reduced resources

                Ideal for Edge: The Arbiter topology reduces infrastructure costs while maintaining high availability for remote/edge deployments where full 3-node control planes may be resource-prohibitive.
            

Chapter 02

Machine Config Operator & Node Management

Machine Config Operator (MCO)

Manages OS and config updates for RHCOS nodes, ensuring consistency and enabling automated updates.

Machine Configs: Define node configuration
Machine Config Pools: Group nodes by role
Machine Config Daemon: Applies changes to nodes

Machine Config Pools (MCPs)

MCPs group nodes based on their role:

● master - Control plane nodes
● worker - Workload nodes

MCO Update Flow

1

Create MachineConfig (systemd, files, kernel)

2

MCO Detects Change - identifies affected pools

3

MCD Applies Config - daemon applies changes

4

Node Cordoned & Drained - pods drained safely

5

Reboot & Verify - node reboots, MCD verifies

                Configuration Drift Detection: The MCD continuously monitors nodes. If a node's configuration doesn't match the MachineConfig, it marks the node as Degraded, ensuring configuration consistency.
            

Chapter 03

Worker Node Architecture

Kubelet

Primary node agent ensuring containers run in pods. Registers node with API server, pulls pod specs, monitors container health.

CRI-O Runtime

Lightweight OCI-compliant container runtime optimized for Kubernetes. Implements CRI, runs OCI images, minimal attack surface.

OVN-Kubernetes

Manages network rules for pod-to-pod and external communication, maintaining OVS flows for service routing.

crun / runc

Low-level OCI runtime (default: crun) that creates and runs containers per CRI-O's instructions.

Node Communication

Worker nodes communicate with the control plane via the API server, reporting status and receiving instructions.

↑

Upward Communication

Node status reports
Pod status updates
Resource metrics

↓

Downward Communication

Pod specifications
Configuration updates
Control commands

Chapter 03

CRI-O: Kubernetes-Native Runtime

What is CRI-O?

An implementation of the Kubernetes Container Runtime Interface (CRI) enabling OCI-compatible runtimes.

OCI Compliant: Runs any OCI image
CRI Native: Built specifically for Kubernetes
Minimal Footprint: Smaller than Docker
Secure by Default: Reduced attack surface

CRI-O vs Docker

No Daemon: Reduces complexity
Version Sync: Matches Kubernetes versions
Simpler Architecture: Just runtime, no build/swarm
Better Security: Smaller codebase

CRI-O Architecture

Kubelet (CRI Client)
Calls CRI gRPC API

↓

CRI-O (CRI Implementation)
Manages images & containers

Image Service: Pull, list, remove
Runtime Service: Create, start, stop

↓

OCI Runtime (crun/runc)
Low-level container execution

↓

Linux Kernel
cgroups, namespaces, SELinux

Chapter 04

Operator Framework Architecture

What is an Operator?

A method of packaging, deploying, and managing a Kubernetes application. It encodes operational knowledge into software, automating complex tasks like deployment, scaling, backup, and updates.

📋

Custom Resources

Extend Kubernetes API with domain-specific resources

🔄

Custom Controllers

Watch and reconcile actual state with desired state

🧠

Domain Knowledge

Encode operational best practices into software

Operator Lifecycle Manager (OLM)

Manages Operators throughout their lifecycle: installation, updates, and removal.

ClusterServiceVersion: Operator metadata & requirements
InstallPlans: Planned installation/updates
Subscriptions: Automatic update channels
CatalogSources: Operator repositories

Operator SDK

Framework for building Operators in multiple languages:

Go
Ansible
Helm

Provides tools to scaffold, test, and package Operators for distribution through OLM.

Chapter 04

Core Cluster Operators

Cluster Version Operator (CVO)

Manages cluster updates by coordinating the update process, validating versions, and ensuring all components are updated consistently.

Machine Config Operator (MCO)

Manages RHCOS nodes by applying MachineConfigs, handling OS updates, and ensuring node configuration consistency.

Network Operator

Deploys and manages the cluster network (OVN-Kubernetes), handling CNI plugins, SDN configuration, and network policies.

Storage Operator

Manages storage classes and CSI drivers, providing persistent storage for stateful applications across the cluster.

Ingress Operator

Manages HAProxy ingress controller and router configuration.

Image Registry

Internal container registry management and configuration.

📊

Monitoring Operator

Prometheus, Grafana, Alertmanager

📝

Logging Operator

Loki, Vector log collection

🔐

Authentication Operator

OAuth server, identity providers

                Operator Hub: Browse and install community and certified Operators from the integrated Operator Hub in the OpenShift console.
            

Chapter 05

OVN-Kubernetes: Default CNI Plugin

What is OVN-Kubernetes?

The default CNI plugin in OpenShift 4.12+, replacing OpenShift SDN. Uses Open Virtual Network (OVN) to provide a robust, scalable software-defined networking solution.

🧬

Geneve Encapsulation

More efficient than VXLAN for overlay networking

🛡️

Native Network Policies

Built-in support for Kubernetes NetworkPolicy

⚡

No kube-proxy

Direct OVN integration for better performance

🔒

IPsec Encryption

Pod-to-pod encryption support

🌍

Egress IP

Assign static IPs for egress traffic

🔥

Egress Firewall

Control outbound traffic from pods

OVN Architecture Components

Northbound DB: Logical network topology (OVN_Northbound)
Southbound DB: Physical runtime state (OVN_Southbound)
ovnkube-node: Runs on each node, translates to OpenFlow
ovnkube-control-plane: Runs on master nodes, pod Subnet Allocation

Additional Features

Multicast: One-to-many communication support
Hybrid Networking: Windows/Linux mixed clusters

Chapter 05

OpenShift Routes & Ingress

OpenShift Routes

Routes provide a simpler, more powerful way to expose services externally. Predating Kubernetes Ingress, Routes offer enhanced features and tighter OpenShift integration.

Path-Based Routing: Route traffic based on URL paths
TLS Termination: Edge, passthrough, or re-encrypt
Wildcard Routes: Support for wildcard certificates
Sticky Sessions: Cookie-based session affinity

Ingress Controller

Uses HAProxy as the default ingress controller, deployed as a DaemonSet on infrastructure nodes.

Automatic Scaling with router shards
Router sharding support for custom routes

Route Example

apiVersion: route.openshift.io/v1

kind: Route

metadata:

  name: myapp

  namespace: default

spec:

  host: myapp.example.com

  to:

    kind: Service

    name: myapp-service

  tls:

    termination: edge

                        Ingress Support: OpenShift fully supports standard Kubernetes Ingress resources, automatically converting them to Routes for consistent management.
                    

Chapter 05

Route TLS Termination Types

Edge Termination

Client → HTTPS → Router → HTTP → Pod

TLS terminated at router, plain HTTP to pod. Default configuration.

Default

Passthrough Termination

Client → HTTPS → Router → HTTPS → Pod

Encrypted traffic passed directly to pod without decryption at router.

Secure

Re-encrypt Termination

Client → HTTPS → Router → HTTPS → Pod

TLS terminated at router, then re-encrypted to pod. Best practice.

Best Practice

Choosing the Right Termination

Edge Termination

Use when: Router manages certificates, pod doesn't need TLS, performance is priority

Passthrough

Use when: Pod needs direct TLS termination, end-to-end encryption required, router shouldn't decrypt

Re-encrypt (Recommended)

Use when: Maximum security required, defense in depth, separate certificates for router and pod

Chapter 06

Storage Architecture & CSI

Container Storage Interface (CSI)

The standard for exposing block and file storage to containerized workloads. OpenShift uses CSI drivers for all storage operations.

Out-of-Tree: Drivers developed independently
Standardized API: Consistent interface
Dynamic Provisioning: On-demand PV creation

Cluster Storage Operator

Manages storage classes and CSI drivers, setting up default storage based on cloud provider.

Supported Storage Types

☁️

Cloud Provider Storage

AWS EBS, Azure Disk, GCP PD (Block, RWO)

🖥️

vSphere Volume

VMware vSphere VMDK (Block, RWO)

💾

Local Volume

Direct-attached local storage (Block, RWO)

🌐

Network Storage

NFS, iSCSI, Fibre Channel (File/Block, RWO/RWX)

📦

Raw Block

Direct block device access for databases

📈

Volume Expansion

Online PVC resizing support

📸

Volume Snapshots

Point-in-time volume copies

Chapter 07

Installation: IPI vs UPI

Installer-Provisioned Infrastructure (IPI)

Fully automated installation where the installer provisions all infrastructure components.

Fully Automated - Minimal user intervention
Quick Deployment - Cluster ready in ~30-45 min
Best Practice - Follows Red Hat recommendations

Recommended for: New deployments, cloud-native environments, rapid prototyping

User-Provisioned Infrastructure (UPI)

Full control over infrastructure provisioning. You create VMs, networks, and load balancers.

Full Control - Custom infrastructure setup
Enterprise Ready - Existing infrastructure integration
Air-Gapped - Disconnected installation support

                        Recommended for: Existing data centers, custom networking, high-security environments
                    

Supported Platforms

🌩️

AWS

IPI & UPI

🌥️

Azure

IPI & UPI

🚀

GCP

IPI & UPI

🖥️

vSphere/Bare Metal

IPI & UPI

Chapter 07

Over-the-Air Updates with CVO

Cluster Version Operator (CVO)

Manages cluster updates by coordinating component updates, ensuring consistency, and maintaining cluster stability.

Version Management: Tracks current and target versions
Update Coordination: Manages update order
Cluster Operators: Updates all Operators sequentially

OpenShift Update Service (OSUS)

Provides update recommendations via hosted service maintaining graph of valid update paths.

Update Channels

stable-4.20
Production-ready releases

fast-4.20
Early access to new features

candidate-4.20
Pre-release builds

eus-4.20
Extended Update Support

1

Select Channel

2

Check Updates

3

Apply Update

4

Node Reboots

5

Complete

                Canary Updates: OpenShift supports canary deployments, allowing you to update a subset of worker nodes first to validate workload compatibility before full rollout.
            

Chapter 08

Security Context Constraints (SCC)

What are SCCs?

Security Context Constraints control the actions pods can perform and resources they can access. They define conditions a pod must run with to be accepted.

restricted (Default)

Denies privileged access, most restrictive. Default for all pods.

Default

restricted-v2

Aligns with Kubernetes Pod Security Standards. Updated security defaults.

New

anyuid

Allows running as any UID. Legacy applications requiring specific users.

Legacy

privileged

Full access, bypasses all checks. Admin-only infrastructure components.

Admin

SCC Key Capabilities

Privileged containers
Host namespaces
Volume types
SELinux contexts
User/Group IDs
Seccomp profiles

SCC Selection

Pods are validated against SCCs in priority order. The first matching SCC is used.

                        oc adm policy add-scc-to-user

                        <scc-name> <user-name>

Chapter 08

Integrated Authentication & Authorization

OAuth Server & Identity Providers

OpenShift includes built-in OAuth server supporting multiple identity providers:

🏢

Active Directory / LDAP

🔍

Google OAuth

🐙

GitHub

🔐

OpenID Connect

📜

X.509 Client Certs

👤

HTPasswd (local)

Role-Based Access Control (RBAC)

Kubernetes RBAC with OpenShift-specific roles for fine-grained access control.

Cluster Roles: Cluster-scoped permissions (cluster-admin, admin)
Roles: Namespace-scoped permissions
Bindings: Link roles to users/groups

OpenShift Enhancements

Projects: Enhanced namespaces with self-service provisioning
Groups: Organize users for easier RBAC management
Service Accounts: Automated pod authentication

1

User Login

Access console or CLI

2

OAuth Auth

Validate with IdP

3

Token Issued

OAuth token returned

4

API Access

Token authentication

5

RBAC Check

Permissions verified

Chapter 09

Monitoring & Logging Architecture

Monitoring Stack

Prometheus

Core metrics collection and storage. Time-series DB with pull model and PromQL query language.

Grafana

Visualization dashboards with pre-built OpenShift dashboards.

Alertmanager

Alert routing, grouping, and notification (PagerDuty, Slack, email).

User Workload Monitoring support
Custom metrics via /metrics endpoint
Custom alerts with PromQL

Logging Stack

Vector

High-performance log collector (replaces Fluentd).

Loki

Log aggregation system inspired by Prometheus.

Log Storage

LokiStack for log persistence.

Log Forwarding

Forward logs to external systems:

Elasticsearch, Splunk, Kafka, CloudWatch

📝

Application Logs

Container stdout/stderr from application pods

🔧

Infrastructure Logs

Node system logs (journald, services)

🔒

Audit Logs

Kubernetes API audit events for security

Chapter 10

Future Roadmap: OpenShift 4.20+

🏗️

2+1 Arbiter Topology

Cost-optimized HA control plane with 2 full masters + 1 arbiter node for etcd quorum. GA in 4.20

☁️

Hosted Control Planes (HCP)

Multi-cluster management with control planes hosted on a management cluster. Tech Preview

💪

ARM64 Support Expansion

Native ARM64 architecture support for power-efficient deployments on AWS/Azure and Bare Metal.

🤖

AI/ML Workload Optimizations

Enhanced GPU support with NVIDIA GPU Operator, MIG support, and optimized AI/ML pipelines.

💻

Developer Experience

Enhanced dev spaces, inner loop development, and tighter IDE integration with OpenShift Dev Spaces.

🛡️

Security Enhancements

Advanced cluster security with improved compliance reporting, vulnerability scanning, and ACS integration.

🌐

Service Mesh 3.0

Istio-based service mesh with enhanced observability and traffic management.

🖥️

OpenShift Virtualization

Run VMs alongside containers with unified management and networking.

💾

OpenShift Data Foundation

Software-defined storage with enhanced data services and disaster recovery.

🔄

GitOps

ArgoCD integration for declarative continuous delivery and configuration management.

Summary

Key Takeaways

🏗️

Architecture

3-master HA control plane with etcd Raft quorum, 2+ worker nodes for workloads

🔧

Management

Operator-based management with CVO for seamless over-the-air updates

🛡️

Security

SCCs for pod security, integrated OAuth, RBAC, and enterprise authentication

🌐

Networking

OVN-Kubernetes CNI with Routes for external access and built-in ingress

📊

Observability

Prometheus monitoring, Loki logging, and comprehensive cluster metrics

🚀

Future

Edge-optimized topologies, hosted control planes, and AI/ML enhancements

OpenShift 4: Enterprise Kubernetes Platform

Complete application platform for developing, deploying, and managing containerized applications at scale with enterprise-grade security and automation.

Thank You

Questions & Discussion

Ayush Garg
PTSE (Products AI and OpenShift)

Red Hat is the world's leading provider of enterprise open source software solutions