Ganesh – NGINX & Cloud Infrastructure Engineer

Blog

  • NGINX Plus Ingress Controller with App Protect WAF and Let’s Encrypt SSL on AKS – Complete Setup Guide

    This guide walks through deploying NGINX Plus Ingress Controller 5.4.1 on Azure Kubernetes Service (AKS), exposing a workload over HTTPS using a Let’s Encrypt certificate via cert-manager, and protecting it with F5 NGINX App Protect WAF. Every step is in sequence so you can reproduce the full stack from scratch.

    Architecture Overview

    The final stack consists of:

    • AKS cluster — Kubernetes worker nodes on Azure
    • NGINX Plus Ingress Controller 5.4.1 — with NAP (App Protect) module enabled
    • cert-manager — automates Let’s Encrypt certificate issuance and renewal
    • NGINX App Protect WAF — OWASP signature-based blocking policy
    • WordPress + MariaDB — sample backend workload

    Prerequisites

    • AKS cluster up and running (kubectl configured)
    • NGINX Plus license JWT token
    • Access to private-registry.nginx.com (F5/NGINX entitlement)
    • A DNS A record pointing your domain to the LoadBalancer public IP
    • Helm 3 installed locally

    Step 1 — Create the Namespace

    kubectl create namespace nic-perf

    Step 2 — Create the Image Pull Secret

    NGINX Plus images are hosted on a private registry. Create a Kubernetes secret with your registry credentials:

    kubectl create secret docker-registry regcred \
  --docker-server=private-registry.nginx.com \
  --docker-username=<YOUR_NGINX_REGISTRY_USER> \
  --docker-password=<YOUR_NGINX_REGISTRY_PASSWORD> \
  -n nic-perf

    Step 3 — Create the NGINX Plus License Secret

    Store your NGINX Plus JWT license as a Kubernetes secret:

    kubectl create secret generic nplus-license \
  --from-literal=license.jwt="<YOUR_JWT_TOKEN>" \
  -n nic-perf

    Step 4 — Install NGINX Plus Ingress Controller via Helm

    Add the NGINX Helm repository and install NIC 5.4.1 with the NAP-enabled image:

    helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update

helm install nginx-plus-nic nginx-stable/nginx-ingress \
  --namespace nic-perf \
  --version 2.5.1 \
  --set controller.image.repository=private-registry.nginx.com/nginx-ic-nap/nginx-plus-ingress \
  --set controller.image.tag=5.4.1 \
  --set controller.nginxplus=true \
  --set controller.appprotect.enable=true \
  --set controller.ingressClass.name=nginx-perf \
  --set controller.ingressClass.create=true \
  --set controller.ingressClass.setAsDefaultIngress=false \
  --set controller.service.type=LoadBalancer \
  --set controller.serviceAccount.imagePullSecretName=regcred \
  --set controller.mgmt.licenseTokenSecretName=nplus-license

    Wait for the controller to be ready:

    kubectl rollout status deployment/nginx-plus-nic-nginx-ingress-controller -n nic-perf

    Get the public IP assigned by Azure:

    kubectl get svc -n nic-perf nginx-plus-nic-nginx-ingress-controller

    Map this IP to your domain in your DNS provider before proceeding.

    Step 5 — Deploy WordPress and MariaDB

    Create persistent storage and deploy the backend workload.

    MariaDB PVC and Deployment

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mariadb-pvc
  namespace: nic-perf
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: managed-csi
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
  namespace: nic-perf
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: mariadb:11.4
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "<YOUR_ROOT_PASSWORD>"
        - name: MYSQL_DATABASE
          value: wordpress
        - name: MYSQL_USER
          value: wordpress
        - name: MYSQL_PASSWORD
          value: "<YOUR_DB_PASSWORD>"
        ports:
        - containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: mariadb-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: mariadb
  namespace: nic-perf
spec:
  selector:
    app: mariadb
  ports:
  - port: 3306

    WordPress PVC and Deployment

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wordpress-pvc
  namespace: nic-perf
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: managed-csi
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  namespace: nic-perf
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
      - name: wordpress
        image: wordpress:6.7-apache
        env:
        - name: WORDPRESS_DB_HOST
          value: mariadb
        - name: WORDPRESS_DB_USER
          value: wordpress
        - name: WORDPRESS_DB_PASSWORD
          value: "<YOUR_DB_PASSWORD>"
        - name: WORDPRESS_DB_NAME
          value: wordpress
        ports:
        - containerPort: 80
        volumeMounts:
        - name: data
          mountPath: /var/www/html
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: wordpress-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  namespace: nic-perf
spec:
  selector:
    app: wordpress
  ports:
  - port: 80

    Apply both files:

    kubectl apply -f mariadb.yaml -n nic-perf
kubectl apply -f wordpress.yaml -n nic-perf

    Step 6 — Create the Ingress (HTTP only first)

    Start with HTTP only. TLS will be added after the certificate is issued.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress-ingress
  namespace: nic-perf
  annotations:
    nginx.org/proxy-connect-timeout: "60s"
    nginx.org/proxy-read-timeout: "60s"
    nginx.org/client-max-body-size: "64m"
    nginx.org/proxy-buffering: "false"
spec:
  ingressClassName: nginx-perf
  rules:
  - host: your-domain.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wordpress
            port:
              number: 80

    Verify the site is reachable over HTTP before proceeding.

    Step 7 — Install cert-manager

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml

kubectl rollout status deployment/cert-manager -n cert-manager
kubectl rollout status deployment/cert-manager-webhook -n cert-manager
kubectl rollout status deployment/cert-manager-cainjector -n cert-manager

    Step 8 — Create Let’s Encrypt ClusterIssuer

    The key setting here is ingress.name: wordpress-ingress. NGINX Plus NIC enforces exclusive host ownership per ingress — if cert-manager creates a separate ingress for the ACME challenge it will be rejected with “All hosts are taken by other resources”. Referencing the existing ingress by name tells cert-manager to inject the challenge path into it instead.

    apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx-perf
          name: wordpress-ingress

    Step 9 — Add TLS to the Ingress

    Update the ingress to request a certificate and enable HTTPS. Do not add an HTTP-to-HTTPS redirect at this stage — Let’s Encrypt must be able to reach the HTTP-01 challenge path over plain HTTP during initial issuance.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress-ingress
  namespace: nic-perf
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.org/proxy-connect-timeout: "60s"
    nginx.org/proxy-read-timeout: "60s"
    nginx.org/client-max-body-size: "64m"
    nginx.org/proxy-buffering: "false"
spec:
  ingressClassName: nginx-perf
  tls:
  - hosts:
    - your-domain.example.com
    secretName: your-domain-tls
  rules:
  - host: your-domain.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wordpress
            port:
              number: 80

    Watch the certificate until it is ready:

    kubectl get certificate -n nic-perf -w

    You should see READY: True within 60-90 seconds. Once the certificate is issued you can optionally re-enable the HTTP redirect by patching the NIC ConfigMap:

    kubectl patch configmap nginx-plus-nic-nginx-ingress -n nic-perf \
  --type=merge -p '{"data":{"ssl-redirect":"true"}}'

    Step 10 — Create the WAF Log Profile

    apiVersion: appprotect.f5.com/v1beta1
kind: APLogConf
metadata:
  name: waf-logconf
  namespace: nic-perf
spec:
  content:
    format: default
    max_message_size: 64k
    max_request_size: any
  filter:
    request_type: illegal

    Step 11 — Create the WAF Policy

    This policy uses the NGINX base template with blocking mode enabled, all attack signatures active, and bot defense turned on.

    apiVersion: appprotect.f5.com/v1beta1
kind: APPolicy
metadata:
  name: waf-policy
  namespace: nic-perf
spec:
  policy:
    name: waf-policy
    description: "OWASP Top 10 protection - blocking mode"
    template:
      name: POLICY_TEMPLATE_NGINX_BASE
    applicationLanguage: utf-8
    enforcementMode: blocking
    blocking-settings:
      violations:
        - name: VIOL_BOT_CLIENT
          alarm: true
          block: true
        - name: VIOL_EVASION
          alarm: true
          block: true
        - name: VIOL_HTTP_PROTOCOL
          alarm: true
          block: true
        - name: VIOL_RATING_THREAT
          alarm: true
          block: true
        - name: VIOL_THREAT_CAMPAIGN
          alarm: true
          block: true
        - name: VIOL_PARAMETER
          alarm: true
          block: false
        - name: VIOL_URL
          alarm: true
          block: false
      evasions:
        - description: "Bad unescape"
          enabled: true
        - description: "Directory traversals"
          enabled: true
        - description: "Bare byte decoding"
          enabled: true
        - description: "Multiple decoding"
          enabled: true
          maxDecodingPasses: 2
    signature-sets:
      - name: All Signatures
        block: true
        signatureSet:
          filter:
            minRevision: "2020-01-01"
    bot-defense:
      settings:
        isEnabled: true
      mitigations:
        classes:
          - name: malicious-bot
            action: block
          - name: untrusted-bot
            action: alarm
          - name: trusted-bot
            action: detect
    response-pages:
      - responsePageType: default
        responseContent: |
          <html><head><title>Request Rejected</title></head>
          <body><h1>Request Rejected</h1>
          <p>Your request has been blocked by the Web Application Firewall.</p>
          <p>Support ID: <%TS.request.ID()%></p></body></html>
        responseHeader: "HTTP/1.1 403 Forbidden\r\nContent-Type: text/html; charset=utf-8"

    Step 12 — Enable WAF on the Ingress

    Add the App Protect annotations to the ingress. The log destination is set to stderr so events appear in kubectl logs without needing a separate syslog server.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress-ingress
  namespace: nic-perf
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.org/proxy-connect-timeout: "60s"
    nginx.org/proxy-read-timeout: "60s"
    nginx.org/client-max-body-size: "64m"
    nginx.org/proxy-buffering: "false"
    appprotect.f5.com/app-protect-enable: "True"
    appprotect.f5.com/app-protect-policy: "nic-perf/waf-policy"
    appprotect.f5.com/app-protect-security-log-enable: "True"
    appprotect.f5.com/app-protect-security-log: "nic-perf/waf-logconf"
    appprotect.f5.com/app-protect-security-log-destination: "stderr"
spec:
  ingressClassName: nginx-perf
  tls:
  - hosts:
    - your-domain.example.com
    secretName: your-domain-tls
  rules:
  - host: your-domain.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wordpress
            port:
              number: 80

    Step 13 — Verify Everything Works

    Test HTTPS

    curl -I https://your-domain.example.com

    Expected: HTTP/2 200

    Test WAF blocking

    # SQL injection - should return 403
curl -sk "https://your-domain.example.com/?id=1'+OR+'1'='1" -o /dev/null -w "%{http_code}"

# XSS - should return 403
curl -sk "https://your-domain.example.com/?q=<script>alert(1)</script>" -o /dev/null -w "%{http_code}"

# Directory traversal - should return 403
curl -sk "https://your-domain.example.com/../../etc/passwd" -o /dev/null -w "%{http_code}"

# Normal request - should return 200
curl -sk "https://your-domain.example.com/" -o /dev/null -w "%{http_code}"

    View WAF logs

    NIC_POD=$(kubectl get pods -n nic-perf -l app=nginx-plus-nic-nginx-ingress \
  -o jsonpath='{.items[0].metadata.name}')

# Stream all security events
kubectl logs -n nic-perf $NIC_POD -f | grep 'request_status'

# Stream only blocked requests
kubectl logs -n nic-perf $NIC_POD -f | grep 'request_status="blocked"'

# Pretty-print key fields
kubectl logs -n nic-perf $NIC_POD \
  | grep 'request_status="blocked"' \
  | grep -oP '(date_time|ip_client|attack_type|violations|support_id|severity)="[^"]*"'

    Key Gotchas

    1. NGINX Plus NIC host ownership — NIC rejects any ingress that tries to claim a hostname already owned by another ingress. This breaks the default cert-manager HTTP-01 solver which creates a new ingress. Fix: set ingress.name in the ClusterIssuer solver to reuse the existing ingress.
    2. SSL redirect blocks ACME challenge — If HTTP-to-HTTPS redirect is enabled when the certificate is first requested, the Let’s Encrypt HTTP-01 challenge will be redirected to HTTPS and fail. Disable the redirect until the certificate is issued.
    3. WAF violation names — The APPolicy CRD validates violation names strictly. VIOL_ATTACK_SIGNATURE is not a valid blocking-settings violation name in current versions. Use VIOL_RATING_THREAT and VIOL_THREAT_CAMPAIGN for signature-based blocking control.
    4. Bot defense actions — The allow action is not valid for bot-defense mitigations. Use detect for trusted bots instead.
    5. cert-manager auto-renewal — cert-manager renews the certificate automatically 30 days before expiry. No manual intervention needed.

    Component Versions Used

    • NGINX Plus Ingress Controller: 5.4.1 (Helm chart nginx-ingress-2.5.1)
    • NGINX Plus: R36-P3
    • NGINX App Protect: 5.12.0
    • cert-manager: v1.16.3
    • AKS: Azure Kubernetes Service (any recent version)
    • WordPress: 6.7-apache
    • MariaDB: 11.4

  • How I Fixed NAP v5 Security Monitoring in NIM on AKS

    Background

    I was tasked with integrating NGINX App Protect (NAP) v5 with NGINX Instance Manager (NIM) 2.21 Security Monitoring on an AKS cluster. The goal: WAF security events from NAP should appear in the NIM Security Monitoring dashboard in real time.

    The Architecture

    The setup involved three main components running as a 3-container pod in Kubernetes:

    • nginx-plus – NGINX Plus R36-P3 with App Protect module
    • waf-enforcer – NAP v5 enforcement engine
    • waf-config-mgr – Policy configuration manager

    NIM 2.21 runs separately with ClickHouse as the backend for security event storage, and nginx-agent v2.46.1 is supposed to forward events from NAP to NIM via a syslog listener on port 5514.

    The Problem

    WAF attacks were being blocked by NAP, but zero events appeared in NIM Security Monitoring. The nginx-agent was running but the syslog listener on port 5514 was never starting.

    Root Cause Investigation

    After deep investigation including analyzing the DPM dqlite database, I found the root cause: the NIM license was a null placeholder with current_report_type=initial and no JWT token. This caused DPM entitlement check to fail, which blocked the nginx-app-protect feature from being granted to nginx-agent, which meant the syslog listener on port 5514 never started.

    The Fix

    Since fixing the license required vendor involvement, I implemented a Python-based NAP event forwarder as a workaround:

    1. Changed NAP security log from syslog to file-based logging
    2. Wrote a Python script nap_forwarder.py that tails the security log file
    3. The script parses NAP key=value log format and maps fields to ClickHouse schema
    4. Events are inserted directly into ClickHouse nms.v4_security_events via HTTP API
    5. Deployed via ConfigMap mounted into the nginx-plus container

    Result

    Within minutes, WAF security events started appearing in NIM Security Monitoring dashboard. The solution bypassed the broken DPM license chain entirely while maintaining full event visibility for security operations.

    Key Takeaways

    • Always verify DPM license entitlements when nginx-agent features are missing
    • NAP v5 syslog forwarding depends on DPM granting the nginx-app-protect feature
    • Direct ClickHouse insertion is a viable workaround for broken agent pipelines
    • Understanding the full data flow is essential for troubleshooting complex multi-component systems