In a microservices architecture, service-to-service communication complexity grows exponentially with scale. The application code should not own these concerns.

At its core, a Service Mesh is a dedicated infrastructure layer for handling service-to-service communication. It operates via two primary planes:

The Traffic Interception Model

Pod
┌─────────────────────────────────────────┐
│  ┌─────────────┐    ┌───────────────┐  │
│  │   App       │◄──►│  Envoy Proxy  │  │  ← Sidecar (istio-proxy)
│  │ Container   │    │  (istio-proxy)│  │    intercepts via iptables
│  │ Port: 8080  │    │  Port: 15001  │  │    or Istio CNI plugin
│  └─────────────┘    └───────────────┘  │
│         ▲                    ▲          │
│         └────────────────────┘          │
│              localhost:8080              │
└─────────────────────────────────────────┘
  All traffic flows through Envoy before leaving/entering the pod

OpenShift Service Mesh 3 is a major architectural shift from OSSM 2. It moves from the Maistra custom control plane to the upstream Istio Sail Operator model.

Component Interaction Flow

┌─────────────┐     watch      ┌─────────────┐     xDS (gRPC)     ┌─────────────┐
│   K8s API   │───────────────►│   istiod    │──────────────────►│ Envoy Proxy │
│  Server     │   (Services,   │  (Pilot +   │  LDS/RDS/CDS/EDS   │ (Sidecar)   │
│             │   Endpoints,   │  Citadel)    │  SDS (certs)       │             │
│             │   CRDs)        │              │                    │             │
└─────────────┘                └─────────────┘                    └─────────────┘
                                      ▲
                                      │ Mutating Webhook
                                      │ (sidecar injection)
                               ┌─────────────┐
                               │  kube-api   │
                               │  (Admission)│
                               └─────────────┘

OSSM 3 is installed via the Operator Lifecycle Manager (OLM). The operator runs cluster-wide and watches for Istio and IstioCNI resources.

Prerequisites

• OpenShift Container Platform 4.16 or later
• Cluster-admin privileges
• No conflicting OSSM 2 ServiceMeshControlPlane in the same cluster (or configured isolation)

Step-by-Step: Web Console

1. Navigate to Operators → OperatorHub
2. Search for "Red Hat OpenShift Service Mesh 3"
3. Click Install
4. Select All namespaces on the cluster (default) — installs in openshift-operators
5. Approval Strategy: Automatic (recommended for production)
6. Update Channel: stable or stable-3.x for pinned versions

The Istio resource is the primary configuration object for the mesh. It defines the version, namespace, update strategy, and Helm values for the control plane.

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
  namespace: istio-system
spec:
  version: v1.24.4
  namespace: istio-system
  updateStrategy:
    type: InPlace                    # InPlace | RevisionBased
    inactiveRevisionDeletionGracePeriodSeconds: 30
  values:
    meshConfig:
      enableTracing: true
      defaultConfig:
        holdApplicationUntilProxyStarts: true   # Ensures envoy starts before app
      extensionProviders:
        - name: otel-tracing
          opentelemetry:
            service: otel-collector.istio-system.svc.cluster.local
            port: 4317
    pilot:
      autoscaleEnabled: true
      autoscaleMin: 2
      autoscaleMax: 5
      cpu:
        targetAverageUtilization: 80
      resources:
        requests:
          cpu: 500m
          memory: 2Gi
    global:
      proxy:
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

The IstioCNI resource deploys the Istio CNI plugin as a DaemonSet. This is mandatory for OSSM 3 on OpenShift 4.14+ and replaces the legacy init-container approach.

Why CNI over Init Container?

• Security: No NET_ADMIN or privileged: true required in application pods
• Compliance: Satisfies OpenShift SCC restrictions (restricted-v2 by default)
• Reliability: Network setup happens before pod sandbox creation, not after
• Performance: Eliminates init container startup latency

Minimal IstioCNI Manifest

apiVersion: sailoperator.io/v1
kind: IstioCNI
metadata:
  name: default
  namespace: istio-cni
spec:
  version: v1.24.4
  namespace: istio-cni
  values:
    cni:
      chained: true              # Append to existing CNI chain (required for OCP)
      cniBinDir: /var/lib/cni/bin
      cniConfDir: /etc/cni/multus/net.d
      cniConfFileName: istio-cni.conf
      logLevel: info
      privileged: true           # CNI DaemonSet needs privileged to modify node networking

When you create an Istio CR, the operator creates an IstioRevision resource. This represents one revision of the control plane and is the actual object that drives the istiod Deployment.

Revision-Based Updates (Canary Upgrade)

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  version: v1.24.4
  updateStrategy:
    type: RevisionBased
  values:
    pilot:
      env:
        PILOT_ENABLE_CROSS_CLUSTER_WORKLOAD_ENTRY: "true"

# The operator creates:
# IstioRevision/default-v1-24-4
# Deployment/istiod-v1-24-4 in namespace istio-system

With RevisionBased, a new istiod Deployment is created alongside the old one. You migrate namespaces by changing the injection label:

# Old revision
oc label namespace bookinfo istio.io/rev=default-v1-24-3 --overwrite

# New revision (canary)
oc label namespace bookinfo-canary istio.io/rev=default-v1-24-4

# Verify proxies connect to correct control plane
istioctl proxy-status | grep istio.io/rev

InPlace Updates

The default strategy. The operator performs a rolling update of the existing istiod Deployment. All proxies reconnect to the new control plane. Faster but riskier for large meshes.

Sidecar injection is the process of automatically adding the Envoy proxy container to application pods. OSSM 3 supports automatic injection via mutating webhook and manual injection via istioctl.

Automatic Injection: Namespace Label

# Enable automatic injection for a namespace
oc label namespace bookinfo istio-injection=enabled

# For revision-based control planes, use the revision label instead
oc label namespace bookinfo istio.io/rev=default

# Verify the label
oc get namespace bookinfo -o jsonpath='{.metadata.labels}'

How Injection Works (Admission Webhook)

1. Pod creation request sent to kube-apiserver
2. MutatingAdmissionWebhook (istiod-sidecar-injector) intercepts the request
3. Webhook checks namespace labels (istio-injection=enabled or istio.io/rev=)
4. If matched, istiod patches the PodSpec:
   - Adds "istio-proxy" container (image: auto)
   - Adds istio-init container (if not using CNI)
   - Adds volume mounts for certs, config, envoy bootstrap
   - Adds annotations for traffic capture
5. Pod scheduled with sidecar injected

Manual Injection (istioctl)

# Generate injected YAML without applying
istioctl kube-inject -f deployment.yaml > deployment-injected.yaml

# Inject and apply directly
istioctl kube-inject -f deployment.yaml | oc apply -f -

A Gateway is a standalone Envoy proxy deployed at the edge of the mesh. It is not part of the control plane — it is a data plane workload that receives traffic from outside the mesh (ingress) or sends traffic outside (egress).

Gateway Deployment Methods in OSSM 3

This is the traditional Istio method where you create a standard Kubernetes Deployment and Service, but annotate it so that istiod injects the proxy configured as a gateway rather than a sidecar.

RBAC for Gateway (Secret Reading)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secret-reader
  namespace: gateway-ns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: gateway-ns
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-reader
  namespace: gateway-ns
subjects:
- kind: ServiceAccount
  name: secret-reader
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Gateway Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-ingressgateway
  namespace: gateway-ns
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  template:
    metadata:
      annotations:
        inject.istio.io/templates: gateway          # ← Use gateway template, not sidecar
        proxy.istio.io/config: '{"gatewayTopology" : {"numTrustedProxies": 2}}'
      labels:
        istio: ingressgateway
        sidecar.istio.io/inject: "true"
    spec:
      serviceAccountName: secret-reader
      containers:
      - name: istio-proxy
        image: auto                                   # ← "auto" resolves to proxyv2 image
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          readOnlyRootFilesystem: true
          capabilities:
            drop: [ALL]
        ports:
        - containerPort: 8080
        - containerPort: 8443
        - containerPort: 15090   # Envoy prometheus metrics

The Kubernetes Gateway API (v1) is the modern, cloud-native standard for ingress traffic management. OSSM 3 supports it alongside the legacy Istio Gateway API. In OCP 4.19+, Gateway API CRDs are automatically installed.

Gateway Resource

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
  annotations:
    networking.istio.io/service-type: ClusterIP   # Or LoadBalancer/NodePort
spec:
  gatewayClassName: istio                           # ← OSSM 3 provides this class
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Same
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: bookinfo-cert

HTTPRoute Resource

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookinfo-route
  namespace: bookinfo
spec:
  parentRefs:
  - name: bookinfo-gateway
  hostnames:
  - "bookinfo.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /productpage
    backendRefs:
    - name: productpage
      port: 9080
  - matches:
    - path:
        type: PathPrefix
        value: /api/v1/products
    backendRefs:
    - name: productpage
      port: 9080
      weight: 90
    - name: productpage-v2
      port: 9080
      weight: 10

The VirtualService defines traffic routing rules applied at the sidecar or gateway. It operates at L7 (HTTP/gRPC) and L4 (TCP) and is the primary tool for advanced traffic management.

Core Capabilities

Advanced VirtualService Example

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: productpage
  namespace: bookinfo
spec:
  hosts:
  - productpage
  gateways:
  - bookinfo-gateway        # Apply to ingress gateway
  - mesh                    # Apply to all sidecars in mesh
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: productpage
        subset: v2
      weight: 100
    fault:
      delay:
        percentage:
          value: 10.0
        fixedDelay: 7s
  - match:
    - uri:
        prefix: /api/v1
    route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v2
      weight: 20
    timeout: 5s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream
  - route:
    - destination:
        host: productpage
        subset: v1
      weight: 100

Security in OSSM 3 is built on identity, not network location. Every workload receives an X.509 certificate with a SPIFFE identity (spiffe://cluster.local/ns/<ns>/sa/<sa>).

PeerAuthentication (mTLS Mode)

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: bookinfo
spec:
  mtls:
    mode: STRICT          # STRICT | PERMISSIVE | DISABLE

# STRICT: Reject plaintext traffic
# PERMISSIVE: Accept both mTLS and plaintext (migration mode)
# DISABLE: Only plaintext

AuthorizationPolicy (L4/L7 Access Control)

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: productpage-policy
  namespace: bookinfo
spec:
  selector:
    matchLabels:
      app: productpage
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/bookinfo/sa/bookinfo-productpage"]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/productpage", "/api/v1/products/*"]
    when:
    - key: request.headers[x-user-group]
      values: ["admin", "user"]

OSSM 3 generates rich telemetry without application changes. The Envoy proxy emits metrics, traces, and access logs for every request.

Telemetry API Configuration

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
  accessLogging:
  - providers:
    - name: envoy
    filter:
      expression: "response.code >= 400"
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 10.0

istioctl is the CLI for managing, debugging, and diagnosing Istio. OSSM 3 supports a subset of upstream commands optimized for OpenShift.

Installation Methods

# Download the istioctl binary from the OpenShift console
# Navigate to: Operators → Installed Operators → Service Mesh
# The "istioctl" tab provides the binary download URL

# Or extract from the istiod container image
oc cp istio-system/$(oc get pod -l app=istiod -n istio-system -o jsonpath='{.items[0].metadata.name}'):/usr/local/bin/istioctl ./istioctl
chmod +x ./istioctl
sudo mv ./istioctl /usr/local/bin/

# Always match istioctl to control plane version
oc get istio -o jsonpath="{range .items[*]}{.spec.version}{'\n'}{end}" | sed 's/^v//'

# Output: 1.24.4
# Download istioctl 1.24.4 from:
# https://github.com/istio/istio/releases/tag/1.24.4

# Verify compatibility
istioctl version --remote

# Client version: 1.24.4
# Control plane version: 1.24.4
# Data plane version: 1.24.4 (10 proxies)

Supported Commands in OSSM 3

Pattern 1: Sidecar Not Injected

# Check namespace label
oc get namespace  -o jsonpath='{.metadata.labels}'
# Must have: "istio-injection":"enabled" OR "istio.io/rev":""

# Check mutating webhook
oc get mutatingwebhookconfiguration istio-sidecar-injector -o yaml
# Verify caBundle is populated and namespaceSelector matches

# Check istiod logs
oc logs -n istio-system -l app=istiod | grep -i "inject\|admission"

Pattern 2: 503 Errors / No Healthy Upstream

# Check if endpoints exist in Envoy
istioctl proxy-config endpoint  -n  | grep 

# Check DestinationRule subsets match Pod labels
oc get pods -l version=v1 -n bookinfo
# Verify VirtualService subset "v1" exists in DestinationRule

# Check mTLS conflict (PERMISSIVE vs STRICT)
istioctl authn tls-check  -n 

Pattern 3: High Control Plane Memory / CPU

# Check number of watched namespaces
oc get istio default -n istio-system -o yaml | grep -A 10 discoverySelectors

# Check proxy count
oc get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].name}{"\n"}{end}' | grep istio-proxy | wc -l

# Scale istiod if needed (HPA or static replica count)
oc patch istio default -n istio-system --type merge -p '
spec:
  values:
    pilot:
      autoscaleMin: 3
      autoscaleMax: 7'