July 10, 2017

Exposing kubernetes pods to the Internet

For those who don’t know what a pod is (in kubernetes terms), a pod can be considered as a logical host with collection of one or more containers, shared storage and network namespace. The aim of this this post is to expose one or more pods to the Internet.

Let’s define the pod first.

apiVersion: v1
kind: Pod
metadata:
  name: foo
  labels:
    application: foo
spec:
  selector:
    matchLabels:
      application: foo
  containers:
    - name: foo
      image: gcr.io/foo
      imagePullPolicy: IfNotPresent
      resources:
        limits:
          cpu: 0.5
          memory: 512Mi

Note: Kubernetes cluster/resource operations are beyond the scope of this article.

The above resource template creates a pod with one container foo with mentioned memory and CPU constraints. By defenition, “Services generally abstract access to pods”. It also defines other configurations like what ports to expose, define the subnets, .. To understand more about services, this document helps. A sample service file looks like the following.

apiVersion: v1
kind: Service
metadata:
  name: foo
  labels:
    application: foo
spec:
  type: NodePort
  selector:
    application: foo
  ports:
    - port: 80
      targetPort: 9026
      protocol: TCP
      name: foo-bar
    - port: 8000
      targetPort: 8000
      protocol: TCP
      name: foo-baz

Let’s break down the above resource template.

  • Label selectors are used to tie service and pod together
  • Type nodePort binds container ports with node/minion ports
  • Application foo has two TCP endpoints, say a HTTP server on 9026 and websocket server on 8000

Now we have a pod and a service which points to the pod. To expose the application to the internet, we should define an ingress rule. Ingress what? By defenition, “An Ingress is a collection of rules that allow inbound connections to reach the cluster services.” To put in simple terms, Pods and services are assigned IP addresses for routing within the cluster network. Ingress helps expose the services(thus the pod/application) to be accessed from outside the cluster. Let’s define ingress resource template for the same. In the following example, we use nginx as our ingress controller. Will explain more about ingress controller in a minute.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: foo
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
  labels:
    application: foo
spec:
  rules:
  - host: foo.com
    http:
      paths:
      - path: /
        backend:
          servicName: foo-bar
          servicePort: 80
      - path: /ws
        backend:
          serviceName: foo-baz
          servicePort: 8000

Similar to service label selectors are used to tie ingress rule and the service together. We expose the HTTP application endpoint on foo.com and ws endpoint on foo.com/ws. To get a clear idea on whatthose nginx configurations are, let’s try to understand what nginx controller is and how it works with ingress rule together to expose application endpoints on respecive paths.

In our case we use nginx as reverse proxy. In nginx, the paths are defined and pointed to the application. A minimal nginx configuration to acheive the above might look like the following. If you’ve written/seen nginx configurations, this configuration might look familiar.

server {
	# .foo.com will match both foo.com and anything.foo.com
	server_name .foo.com;

	# It's always good to set logs
	access_log /var/log/nginx/example.access.log;
	error_log /var/log/nginx/example.error.log;

        location / {
            proxy_pass http://127.0.0.1:9026;
	}

        location /ws {
            proxy_pass http://127.0.0.1:8000;
        }

In simple terms, nginx ingress controller is an nginx instance which watches for ingress rules and dynamically updates the configuration. Let’s define an ingress controller.

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
  annotations:
    domainName: foo.com
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ AWS certificate ARN }}
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
spec:
  type: LoadBalancer
  selector:
    app: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80

---

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-nginx
        k8s-addon: ingress-nginx.addons.k8s.io
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3
        name: ingress-nginx
        imagePullPolicy: Always
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx

Note: The above ingress controller assumes the cloud provider is aws. If you use other cloud providers, refer to nginx ingress controller documentation for annotations to change.

Post creation of the nginx ingress controller resource, changes on/for ingress resources are watched and nginx configuration is updated dynamically.

Any number of applications could be created as pods with its services and ingress rules tied together by labels and exposed out of the cluster. If you have noticed, the above template has spec which says loadBalancer. The above controller creates one Elastic load balancer and can be used to expose n number of applications. It is possible to expose applications as subdomains as well like x.foo.com points to port 8010 and y.foo.com points to port 8020.

Something missing (or) not clear? Tweet @dolftax

Copyleft Jaipradeesh