Kubernetes plugin

The Kubernetes[^1] plugin provides instance deployment and operations using the Kubernetes container orchestration system.

For easiest configuration and reduced failure modes, it is best to run Skopos on the same cluster where the application runs, with the Kubernetes API accessed using the in-cluster service access method. The plugin also supports kubectl-style context in a ~/.kube/config file mapped into the Skopos container (see the Installing Skopos document for command examples). For advanced configurations, please contact our technical support.

CAUTION: The Kubernetes plugin for Skopos is currently in alpha.

Quick jump to Configuration.

[^1]: All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Compatibility

Kubernetes version

The Skopos Kubernetes plugin requires Kubernetes version 1.8 or later (tested with 1.8.3). Please contact our technical support if you need to use an earlier version (e.g., 1.7)

It is possible to use it on a single-node cluster (e.g., minikube) and on a multi-node cluster, either on premise or with a number of Kubernetes cloud service providers (tested with KOPS-installed Kubernetes on AWS but supported in GKE; contact technical support for Azure Kubernetes Service and EC2 Kubernetes Service).

Compatibility with Kubernetes kubectl operations

The Kubernetes plugin for Skopos uses the native Kubernetes API for creating workloads and other Kubernetes objects. Applications and services deployed with the Kubernetes plugin for Skopos can be inspected with the kubectl utility as well as the Kubernetes dashboard and any other API tools. It uses the Deployment and StatefulSet workload objects, as well as other objects (Namespace, Service, etc.).

By default, the Skopos project name corresponds to a Kubernetes namespace. Skopos will automatically create (and delete) the namespace, if needed. Other namespace modes will be supported; please contact technical support for more information.

Importing existing projects into Skopos

Currently Skopos does not support importing Kubernetes objects directly. Projects can be built using the visual editor, directly in YAML, or by importing existing Skopos projects or Docker compose files (v2/v3).

Support for importing Kubernetes objects, helm charts and for discovery is in progress.

Mapping Volumes

Connecting volumes to components is supported with the Kubernetes plugin. Any type of volume defined in the Kubernetes Volume object can be used.

For each volume used in the application, add a corresponding key in plugin_config:

plugin_config:
   k8s:
      volumes:
         webdata:
            (...place any valid k8s Volume structure here...)
         hostpath-example:
            hostPath:
               type: Directory
               path: /var/lib/datavol1

NOTES:

  • Kubernetes Persistent Volumes are not supported. They can be used only with StatefulSet workloads and Skopos uses a 'Deployment', rather than StatefulSet in many cases. This does not mean you cannot use volumes for persistent data that needs to survive component teardowns and upgrades - many of the volume types supported by the 'Volume V1 Core' object are in fact persistent.
  • There is no mechanism to provide a different volume to each instance in a set of replicas (except for the trivial 'emptyDir' volume type). If you have components that have multiple replicas and use a volume, they will all be bound to the same volume (except when using the 'hostPath' type - in this case all replicas that are on the same node will get the same volume).

Plugin-specific component attributes

This plugin supports the following plugin-specific attributes of components in the Skopos model:

components:
  web:
    image: opsani/sample-back

    provides:
      ports:
        - "8080/tcp"
    visual: { x: 350, y: 200 }

    plugin:
      k8s:
        kind: Deployment # Deployment or StatefulSet

        workload:        # raw Deployment or StatefulSet attributes (except containers[])
          annotations:
            skopos.io/test: sample
          spec:
            template:
              spec:
                terminationGracePeriodSeconds: 40,
                volumes:
                  - name: test
                    hostPath:
                      path: '/etc/test'
                      type: File

        container:       # raw attributes for the main container
          resources:
            limits:
              cpu: '250m'
              memory: '128Mi'
            requests:
              cpu: '125m'
              memory: '64Mi'
            volumeMounts:
              - name: test
                mountPath: '/test'
            env:
              - name: SECRET_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: mysecret
                    key: password
              - name: CONFIG_VAR
                valueFrom:
                  configMapKeyRef:
                    name: mycmap
                    key: mycmap_var

        service:         # raw attributes for the service discovery service
          spec:
            sessionAffinity: ClientIP

Details:

  • kind - kind of workload object to create for the component, optional. Supported values are Deployment and StatefulSet. If not specified, Skopos will use heuristics to choose the kind of workload object to create, based on the singleton and stateful component attributes.
  • workload - raw attributes to merge into the workload object (Deployment or StatefulSet). Attributes must be compatible with the workload object
  • container - raw attributes to merge into the container specification (for the main container)
  • service - raw attributes to merge into the Service object created for accessing the component service (type ClusterIP). If the Service attribute is present, even if empty, this will force the creation of a Service object, even if otherwise Skopos would not create one (exposing at least one port will also cause the service object to be created). Note: this attribute should not be used merely to (a) trigger the creation of a service object (specify an exposed port on the component if that's the reason to have the Service object) or (b) expose component's services outside of the cluster/application components (gateways should be used for that, see the next section)

Note: the raw settings may cause the Kubernetes plugin to malfunction due to unexpected behavior caused by specific settings (e.g., changing the MaxSurge in a DeploymentPolicy for a singleton).

Note: See the next section for exposing services outside of the application and for gateway's plugin-specific attributes

Gateways

Both Skopos ingress gateway types, load_balancer and host_port are automatically mapped to Kubernetes Service objects of type LoadBalancer and NodePort, respectively.

Currently, there are no custom load balancer plugins that work with the Kubernetes plugin; creating load balancers is currently done by the Kubernetes cluster itself.

Other gateway types, most prominently including external_service and Ingress object support, are in development.

Plugin-specific attributes for gateways

Gateways support the following Kubernetes-specific parameters:

  • service - raw attribute to be applied to the service object generated by the plugin

Example:

components:
  web:
    provides:
      ports:
        - "8080/tcp"
    # ...
#...
gateways:
  in:
    type: host_port
    target: [ web ]
    exposes:
      - name: http
        port: 30000/tcp  # note k8s default config exposes NodePorts >=30000
        target_port: 8080
    plugin:
      k8s:
        service: # raw attributes for the service object for this gatway, example below
          spec:
            sessionAffinity: ClientIP
            sessionAffinityConfig:
              kubernetes.client_ip:
                timeout_seconds: 3600

Service Discovery

Service discovery is always included with Kubernetes. The Kubernetes plugin for Skopos determines whether a Service object is needed and creates it when needed (typically, when a component exposes at least one port and/or has other components connected to it). There is no need to specify service_discovery_plugin or service_discovery_auto in the environment file. Note that currently, it is not possible to replace or append service discovery steps (see supported injection points below).

Metrics

The Kubernetes plugin for Skopos supports both heapster API and the metrics API introduced in Kubernetes 1.8. See Configuration section for details. When configured, instance metrics become available in the Skopos API as well as in the instance details property sheet in the Skopos web UI.

Known Limitations

The following known limitations exist in the current version of the Kubernetes plugin for Skopos. Please contact our support if you have further questions or are constrained by one or more of these limitations.

  • By default, Kubernetes does not provide network isolation between components that belong to different namespaces (projects/applications in Skopos terminology). Upcoming versions of the Kubernetes plugin will provide automatic ingress filtering. [The default behavior is especially important for those migrating applications from Docker, which does provide network isolation by default.]
  • Only the following step injection points are supported in the model lifecycle with the Kubernetes plugin (all other injection points are silently ignored if present):
  • application level: pre_flight, post_flight and post_teardown
  • component level: pre_component, post_component, quality_gate and on_change (the pre-component, post-component and on-change steps are invoked once; the quality gate steps are invoked once per new instance).
  • It is not possible to change the type of workload object between Deployment and StatefulSet while replicas exist. To change, either teardown the application or set the component's replica count to 0.
  • The following limitations are temporary and expected to be removed in upcoming versions:
  • Automatic rollback is not available for stateful objects that are not singletons, since Kubernetes does not provide history and rollback for StatefulSet workloads. Skopos may add support for rollback in the future.
  • force-replace deploy option is not available in any flavor (target action, per-component, etc.)
  • only in-pod and .kube/config options exist for choosing the target cluster; specifically, this means that the cluster selection is global for the Skopos instance and cannot be overridden in the target environment file (consistent with the Docker and Docker-swarm configuration)
  • preflight checks are not provided; this may delay the recognition of some errors until a deployment is attempted rather than during pre-flight
  • Service objects associated with a component are not rolled back to their prior configuration in case of update failure (these objects change rarely, so in most cases this limitation will not have impact). This applies to Service objects used for (a) inter-component service discovery (b) gateways.
  • Some error messages are not yet propagated, esp. in case of timeout during deployment (waiting for new component version to become ready)
  • only a single container per pod is currently supported (this can be worked around by using the raw workload attributes, essentially overwriting model's attributes)
  • Per-project namespace that was automatically created by Skopos is not automatically deleted after teardown
  • Volumes, secrets and config maps are not supported directly. Raw attributes may be used as a workaround. This limitation will be removed shortly.
  • Model change detection is not implemented yet, which means that Skopos will detect only changes in image spec and replica count (but will not detect changes in arguments or environment vars). This limitation will be removed shortly.
  • The debug logging is enabled by default (contrary to the documented default)

Configuration

All configuration for this plugin is optional and may be omitted.

A detailed configuration may look something like this:

core_plugin: k8s

plugin_config:
  k8s:
    timeout: 300
    log_api: true
    log_objects: true
    log_actions: true
    metrics_server: heapster
    heapster_namespace: kube-system
    heapster_service: heapster2
    volumes:
      hostpath-example:
        hostPath:
          type: Directory
          path: /var/lib/datavol1

The following configuration settings may be provided when desired:

timeout

timeout is the timeout, in seconds, to wait either for the deployed service to become operational or for a change in the service's set of ready instances. As such, this timeout rolls forward during the deployment and will only fail if there is no forward progress and no completion during its period. The default is 300 seconds.

Example:

plugin_config:
  k8s:
    timeout: 300

log_api, log_objects, log_actions

These boolean attributes control what debug information is logged by the plugin. Supported values are true and false (default).

Example:

plugin_config:
  k8s:
    ipv6: true

metrics_server

metrics_server selects which metrics server will be used to obtain instance metrics. Allowed values are heapster (use heapster service), core (use metrics API) and none (disable metrics access). When not specified, the plugin chooses the metrics server automatically.

Note: in the future, auto value may be provided and made the default.

Example:

plugin_config:
  k8s:
    metrics_server: core

heapster_namespace, heapster_service

These attributes allow overriding the default namespace and service name for the heapster service (used to obtain instance metrics). These attributes are used only when the heapster service is used to obtain metrics (see metrics_server). The default values are kube-system and heapster, respectively.

Example:

plugin_config:
  k8s:
    heapster_namespace: heapsterspace
    heapster_service: heapo

volumes

volumes is a configuration a map that defines the location of data and type of volumes for volumes used by the application. See the Mapping Volumes section for an overview of supported volume options.