Continuous Deployment with Skopos™


Continuous Deployment is the autonomous deployment of new application artifacts (code, data) to live systems.

Skopos provides several different workflows for continuous deployment:

  • standalone, using the Skopos™ Auto-Pilot, based directly on Skopos detecting changes in deployment artifacts (e.g., containers and model descriptors)
  • integrated into a CI/CD pipeline, triggered by code commits
  • combination of the above

For these workflows, Skopos provides deployments both for running tests and for updating live production systems.

Note: If you know all about CI/CD and just want to add Skopos to your existing Jenkins pipelines, you can skip ahead to Jenkins with Skopos. Consider the advantages of using mutable and immutable tags, as well as static and dynamic deployments. Finally, check out the Skopos Auto-Pilot for non-pipeline autonomous deployment and for automatically tracking updated components (yours or 3rd party).

Static and Dynamic Deployments

Depending on the type of project, you may have one or more static live, 24x7 environments (e.g., production), as well as the need to deploy temporary, on-demand environments (e.g., to perform tests).

Static Deployments

Static deployments are fixed places where your live applications operate. They are specific and named, for example: production, prod-americas, prod-emea, staging, qa, daily, etc.

The static environments may encompass:

  • different locations (e.g., North America, Europe, Asia), and
  • different purposes (production, staging, stress-test).

These enviroments usually run continuously and when new versions are needed, they are deployed live (e.g., using rolling upgrades, A/B deploys, canary deploys, etc.). It is essential that new versions' operation is verified as part of the deployment and rollbacks are initiated in case of failures.

These are also environments which may be used with the Skopos Auto-Pilot to detect availability of updated deployment artifacts, and either notify DevOps or autonomously deploye these newer versions.

For static deployments, it is best to have a dedicated instance of the Skopos engine configured for each static environment and monitoring it for changes. The Pet Voting App Tutorial illustrates this use.

Dynamic Deployments

Dynamic deployments are on-demand instances of your application, that are created when needed for a specific purpose and destroyed (torn down) immediately after they serve that purpose. Unlike the static environments, the dynamic environments usually are not named and can vary in number as needed.

Examples of dynamic deployments include:

  • transient integration test environments (one or more) created by a CI/CD system as part of the CI pipeline in order to perform specific test(s)
  • development environments created for writing and testing code
  • ad-hoc test environments, e.g., created by support to reproduce/verify a specific reported problem (incl. using older versions of your services)
  • staging and upgrade verification environments, which are deployed to match the current production version and then upgraded to the new version in order to verify the upgrade process

It is possible to launch a separate Skopos engine for each of these environments and take it down once done; however, in most cases, it is preferable to execute a transient deployment, in which case the Skopos engine runs as a task which terminates when the deploy process completes.

Mutable and Immutable Container Tags

Docker container tags support two significantly different workflows. These differ only by convention - there is no enforcements by the container registries - but they are significantly different in the way they can be used to drive continuous deployment (as well as traceability and ability to restore a prior version):

  • immutable tags: these are tags applied to a particular version of a container and will never be reused to tag a different version. For example, nginx:1.11.12 refers to a specific version of nginx. Using this tag (1.11.12) results in always deploying that version of nginx, even if a newer version is available.
  • mutable tags: these are tags that have a certain meaning (e.g., latest or stable) and are being applied usually to the latest version of a project that fits that meaning. For example, nginx:latest refers to the latest version of nginx, while nginx:1.11 refers to the latest release within the 1.11 family.

Note: mutable tags are also known as dynamic tags.

Skopos works well with workflows based on both types of tags, including a mix of the two.

For immutable tags, change either the image attribute in the model or use variable substitution in environment files in order to specify the exact desired version. Reload the model for the changes to take effect (this is usually done automatically at the end of a CI/CD pipeline). The advantage of immutable tags is that an exact version is being deployed.

For mutable tags, simply enable the Skopos auto-pilot (see the next section) and let Skopos detect when a the mutable tag has been moved to a newer version and initiate an update. The advantage of the mutable tags is that the newest fitting version can be deployed automatically, simplifying both development workflow and updates with security fixes.

In your applications, it is possible to use a mix of the two approaches, for example:

  • using immutable tags for your services (e.g., provided by a CI/CD system) and mutable tags for automatically upgrading third party services to always have the latest security fixes (e.g., nginx, redis, etc.)
  • using mutable tags for your services (so Skopos can automatically deploy your code as soon as the new version shows up in the repo/is tagged) and immutable tag to freeze external dependecies (e.g. nginx, mysql).

Auto-Pilot For Deployments

Starting with version 0.10, Skopos can automatically and continuously detect differences between your desired state (as described in the model file) and what is currently running. Skopos can also automatically initiate a deploy in order to bring your application to its desired state.

Note that Skopos Auto-Pilot can initiate deployments based on changes in your own application services, as well as based on changes in third party components (e.g., nginx or redis from Docker Hub)

To detect a difference, Skopos monitors the following:

  • local and remote Docker registries in order to see if the image digest for a given component has changed. This enables software delivery workflows where a component is described with a "dynamic" tag (i.e. :latest or :staging) in the model file with the intention to deploy every time a new version with that tag is released.
  • (new in Skopos 0.14) changes to model and/or environment descriptors that are loaded as a reference to a remote location, i.e. or github://opsani/skopos-sample-app/env.yaml?branch=dev. See CLI reference for details on supported schemes. This allows you to automatically initiate a deploy when you commit a change to your application architecture (as described in the model descriptor) or you environment configuration (as described in the environment descriptor).

Currently, the following levels of autonomy are supported:

  • detect (default) - Skopos monitors the state of the application and generates a new plan whenever the current state is different from the desired state. After a deploy is executed, Skopos will no longer detect changes until the state is manually reloaded (either by issuing a reset command from CLI/GUI or by issuing a new load command). This allows for the status of the last deploy to stay visible.
  • deploy - Skopos detects changes and initiates a deploy whenever the current state is different from the desired state. Skopos will continue to detect (and deploy) changes even after a deploy is done, allowing for a fully autonomous operation. If a failure occurs during a deploy, Skopos will continue to monitor the current state, but will only try a new deploy if something in the current application state changes (i.e. a new version of an image becomes available or the number of replicas for a component changes). This prevents infinite deploy loops when one or more components have a permanent problem that prevents them from deploying successfully.
  • off - Skopos does not monitor state after initial load; the user can to manually reload in order to detect any changes in the current state of the application. (This is a compatibility mode for Skopos pre-0.10 and for using manually created deployment plans.)

The autonomy mode can be specified on application load (via the --autonomy CLI parameter) and changed at any point after that via the GUI or by issuing an API request. The interval at which Skopos will poll for changes can be set on application load via the --poll-interval CLI parameter, default: 30 (seconds).

Note: you cannot use autonomy if you load a manually created deployment plan - when running in autonomous mode Skopos auto-generates deployment plans that get the application in its desired state.

In certain cases (i.e. local component development), it may be desirable to use autonomous mode, but only detect changes in the local Docker registry and ignore any differences between what is running and what is in a remote registry. To do so, set the local_images_only configuration parameter of the docker plugin to true - this will prevent using remote Docker registries when detecting changes and creating new instances.

Example CLI command that loads an application and enables continuous monitoring (every 60s) and auto-deploy:

skopos --bind :8100 load --autonomy deploy --poll-interval 60 \
    --project HelloWorld --env prod.yaml model.yaml

Same, but loading the model and environment directly from a Github repository. Any change in the state of the application as well as the model or environment descriptors will trigger a deploy. This example specified an additional environment (specified as a local file) that can be used to overwrite part (or all) of the environment descriptor that comes from the github repository. Also, the model and environment come from different git branches:

skopos --bind :8100 load --autonomy deploy --poll-interval 60 \
    --project HelloWorld \
    --env github://opsani/skopos-sample-app/env.yaml \
    --env my-local-env.yaml \

Using Skopos with Jenkins Pipelines

Continuous Deployment works great as part of a CI/CD pipeline. Continuous Integration tools, such as Jenkins and CircleCI are great at detecting code commits, building the code (deployment artifacts) and running various tests.

Skopos provides three different capabilities when integrated with CI/CD tools:

  • deploying new code to target enviroments (e.g., development, staging, production)
  • creating and tearing down test environments in which the CI/CD system can run automated tests
  • triggering deployments when external artifacts change (not driven by code changes in your app but based on new versions of library artifacts, e.g., nginx or mongodb)

For a step-by-step tutorial, see and fork CI/CD with Jenkins and Skopos

Next Steps

You can continue with one of the following steps:

  • modify the model and/or environment to deploy your own services
  • add ChatOps in order to keep track of the deployment changes and status
  • try out the various options (mutable vs. immutable tags; static and dynamic deployments; autopilot) to understand their use cases
  • try setting up continuous deployment for the Pet Voting Application