Deployment Verification

Overview

An essential function of application deployment is verifying that a created or upgraded service is working properly. Each instance of a service may be assessed to verify:

  • It is alive (running/responsive)
  • It is healthy (passes a healthcheck)
  • It performs adequately (passes a performance check)
  • It exposes the expected API or service

Skopos enables these assessments by including quality gates in generated deployment plans. Quality gates gate the success of component deployment, triggering a failure and rollback when they do not succeed. A quality gate may assess an instance internally or externally:

  • Internal assessments execute a check within the process space of the instance. Such checks can verify an instance is alive and healthy from the point-of-view of the instance itself, but cannot verify the instance properly exposes its API or service.
  • External assessments execute a check over the service network of the instance to faithfully reproduce the position of a client consuming that service. Such checks can fully verify the instance provides its expected API or service as it will be consumed by a client of that service.

Quality gates may be added automatically by Skopos or may be user defined actions attached to components. Quality gates are visible as injected steps in the Skopos plan view UI.

Automatic Quality Gates

Skopos automatically adds quality gates to deployment plans for components based on known images. Automatic quality gates use probes to assess a target component on its service network. These autogates typically perform a service_up check on the target component. The service_up check verifies a service is up, but does not verify that service in detail: service_up does not verify access with particular credentials, or verify a particular resource is available. This makes the service_up check suitable for automatic assessment as it does not rely on the particular configuration of target components.

For example, when deploying a component based on the official tomcat Docker Hub image, Skopos adds an autogate which uses the http probe to verify a HTTP GET request on port 8080 for the default resource / returns a 2x, 3x or 4x response code. Such a response verifies the tomcat server is responding on its service network at the expected port, but it does not verify this instance serves a particular resource. This check is visible in the plan view UI for a tomcat component:

tomcat component plan details

Types of Autogates

The Skopos autogates fall into three categories:

Database Checks

The service_up check for database components verifies the database API is exposed on the expected port by performing a simple list datbases or ping request. This request is considered successful even if the connect response is authentication failure or invalid database. Auogates are provided for the following public Docker Hub images:

redis, postgres, mongo, mariadb, mysql, mysql-server

HTTP Checks

The service_up check for components which serve HTTP requests verifies a GET request for / returns a 2x, 3x or 4x response code. Autogates are provided for the following public Docker Hub images:

httpd, nginx, tomcat, tomee, wordpress, phpmyadmin/phpmyadmin, bekt/logspout-logstash, gliderlabs/logspout, imriz/logspout, kibana, docker.elastic.co/kibana/kibana, sebp/elk

TCP Connect Checks

The generic service_up check for components which service over TCP on a particular port verifies a socket open request on that port succeeds. Autogates are provided for the following public Docker Hub images:

consul, progrium/consul, memcached, elasticsearch, docker.elastic.co/elasticsearch/elasticsearch

The list of images for which Skopos provides automatic quality is extended as more suitable use cases are brought forward. If you would like to suggest additional automatic quality gate use cases, please send us a feature request at http://opsani.com/feedback.

Disabling Automatic Quality Gates

Skopos autogates can be disabled for an entire application or for a particular component through any of these means:

Command Line

Autogates can be disabled for an application by providing the --skip-autogates option to skopos on application load or run.

Model Descriptor

Autogates can be disabled for an application by specifying the skip_autogates attribute in the lifecycle section of the application:

doctype: com.datagridsys.doctype/skopos/model
version: 1

lifecycle:
    skip_autogates:  true

Autogates can be disabled for a particular component of an application by specifying the skip_autogates attribute in the lifecycle section of that component:

components:
    mysql:
        ...
        lifecycle:
            skip_autogates:  true

If the Skopos autogates are failing a particular use case, please let us know at http://opsani.com/feedback.

User Defined Quality Gates

For ease of use and flexibility, Skopos supports specifying user defined quality gates in TED files (target environment descriptors). Quality gates are typically factored into their own TED file for re-use. In this manner, the checks which gate the successful deployment of components are abstracted from any particular application and made generally usable by many applications.

Quality gates may also be specified directly in an application model. Both of these use cases are covered below.

Quality Gates in Environment Descriptors

Quality gates specified in TED files associate user defined deployment checks to one or more component images. Each quality gate is named, and its mapping of component images to deployment checks is specified. Deployment checks are specified in the form of steps which will be injected in deployment plans. Here is an example:

quality_gates:

    # check HTTP service is up
    http_default:
        images:
            - httpd                      # exact match, no tag
            - httpd:*                    # wildcard matches all tags
            - nginx*                     # match any image starting w/ "nginx"
            - my_registry/websrv:1.2     # exact match, specific tag
        steps:                           # list of checks as plan steps
            - probe: opsani/probe-http   # abbreviated probe step syntax

    # check access to customer database based on mariadb/mysql image
    customer_db:
        images:
            - my_registry/customer_db:*
        steps:
            - probe:                     # full probe step syntax
                image: opsani/probe-mariadb
                action: check_access
                label: "Probe: check customer db access"
                arguments:
                    user: my_user
                    password: ${db_password}
                    database: my_database

This example specifies two named quality gates:

  • http_default - this quality gate uses the Skopos http probe to perform a simple HTTP healthcheck. This check is attached to components using any official httpd image, any image which starts with nginx (e.g. nginx, nginx:stable-alpine, nginxyz/abc:1) and the private image my_registry/websrv:1.2. When Skopos deploys a component with a matching image, the healthcheck is executed against each new instance. If a check fails, the component is rolled back, and the application deployment fails.
  • customer_db - this quality gate uses the Skopos mariadb probe to verify the component is accessible using the provided user, password and database. Any component matching the my_registry/customer_db:* image is deployed using this quality gate.

While the above example uses probe steps, quality gates may use any of the injected step types:

  • probe - execute a containerized application probe to assess the target component on its service network (the same network used to consume that service). See the Probes document for details, including a current list of Skopos probes.
  • exec - execute a user provided script
  • gate - inject a manual gate requiring user confirmation to continue or abort
  • delay - inject a delay in execution of the component plan
  • call - call a plugin with a specified action

See also Injected Step Types for details of injected step types and their attribution.

Quality gates in practice:

  • A quality gate may specify multiple images to be matched. The image specifier supports a trailing wildcard *. Note that mysql* matches both mysql:8.0 and mysql/mysql-server from different repositories. To strictly match all tags of a particular repository, a fail-safe method is to use two image specifiers, e.g. mysql (matching the bare repository name exactly) and mysql:* (matching all tags of that repository after the colon).
  • A quality gate may specify multiple steps. Steps are injected in the order specified.
  • A given component image may match more than one quality gate. In this case, the injected steps for all matching quality gates are included: ordered first alphabetically by quality gate name, and second by the step order within each quality gate.
  • Quality gates are merged through replacement when merging TED files: if the same-named quality gate exists in multiple TED files on application load, that of the last loaded TED file prevails.
  • The quality_gates section of a TED file supports variable substitution just like the model descriptor. Variables are substituted from those defined in the same TED file as the quality gates, or from the merged (effective) TED where multiple TED files are loaded. See Variable Substitution in the model reference for details.
  • Quality gate steps are injected at the quality_probe injection point of the component after any autogates and before any steps specified in the lifecycle section of the component in the application model. Advanced: see Component Injection Points for details.

Quality Gates in Model Descriptors

Specifying Quality Gates by Reference

Simply loading a TED file is sufficient to put its quality gates into practice: components with matching images will deploy using the specified gates. Sometimes, however, it is convenient to cause a component use a particular quality gate without adding its image to the quality gate specification in the TED file. This can be accomplished by adding a quality_gate attribute to the lifecyle section of the component in the application model. Here is an example:

components:
    api:
        image: my_registry/api_server:2.1
        replicas: 1
        provides:
            ports: [ "80" ]
        lifecycle:
            quality_gate: http_default      # use this named quality gate

In this example, even though the api component image my_registry/api_server:2.1 does not match any of those specified for the http_default quality gate in the TED file, this component will deploy using this quality gate because of the model lifecycle attribution.

Specifying Quality Gates Directly

Advanced: quality gates may also be specified directly in a model descriptor without reference to a TED file. In this case, a quality gate is specified as an injected step at the quality_probe injection point. See Component Injection Points for details. Here is an example:

components:
    api:
        image: my_registry/api_secure:1.0
        replicas: 1
        provides:
            ports: [ "443" ]
        lifecycle:
            quality_probe:
                steps:
                    - probe:
                        image: opsani/probe-http
                        action: get_ok
                        label: "Probe https get_ok port 443 /healthz"
                        arguments: { schema: "https", path: "/healthz" }

This example verifies the deployed service responds successfully to a HTTPS request for a particular resource.

Sample Application with Quality Gates

This Skopos sample application is a scalable variant of the Docker example Pet Voting Application which uses quality gates to validate its deployment. A full tutorial is available at https://github.com/opsani/skopos-sample-gates. Quick-start instructions are provided below.

The sample application deploys to Docker Swarm and is comprised of the following descriptors:

The env-quality-gates.yaml descriptor specifies user defined quality gates for this application:

quality_gates:

    # check HTTP service is up
    http_default:
        images:
            - opsani/sample-result:*
            - opsani/sample-vote:*
        steps:
            - probe: opsani/probe-http

    # check access to redis
    redis:
        images:
            - redis:*
        steps:
            - probe:
                image: opsani/probe-redis
                action: check_access

    # check access to postgres
    postgres:
        images:
            - postgres:*
        steps:
            - probe:
                image: opsani/probe-postgres
                action: check_access

Load the Sample App

Start the Skopos engine on a swarm manager node:

docker run -d -p 8100:8100 --restart=unless-stopped --name skopos \
   -v /var/run/docker.sock:/var/run/docker.sock \
   opsani/skopos:edge

Use skopos to load the sample app, fetching the descriptors directly from github (add --bind hostname:port after skopos if Skopos is not running locally on port 8100):

skopos load --project sample-gates \
--env github://opsani/skopos-sample-gates/env-swarm.yaml \
--env github://opsani/skopos-sample-gates/env-quality-gates.yaml \
github://opsani/skopos-sample-gates/model.yaml

Once this application is loaded, open the Plan view in the Skopos UI to examine the deployment plan. Click on any of the db, redis, result or vote components to zoom in to the component plan details. Here you can see the injected quality gate step(s) for that component.

For example, the db component includes two quality gates:

  • automatic quality gate: Skopos includes an automatic quality gate for the db component because it is based on the official postgres image. This gate verifies the postgres service is up on port 5432. The service_up check will succeed if the postgres service is up, even if the connect response is authentication failure or invalid database.
  • user defined quality gate: this quality gate comes from the env-quality-gates.yaml TED file and uses the Skopos postgres probe to verify the postgres API is accessible on the component's service network using the probe default user, password and database (these are configurable).

skopos sample app

Deploy the Sample App

Deploy the application using the UI controls. If you follow the deployment progress in the Skopos Plan view, you can observe the deployment of each of the db, redis, result and vote components is validated by its associated quality gate.

The application can also be deployed using the Skopos CLI (add --bind hostname:port after skopos if Skopos is not running locally on port 8100):

skopos start --project sample-gates

After the deploy completes, the web interfaces exposed by the sample application are available at the gateway ports specified in the env.swarm.yaml environment descriptor:

  • Vote: http://my-ip-or-host:8880/
  • Result: http://my-ip-or-host:8881/