Skopos™ Model Reference

The Skopos Model file is a YAML file that defines application services (components) and their relationships.

Section shortcuts:

Introduction to Skopos models

Unlike deployment pipelines (e.g., Jenkins), Skopos models do not describe a sequence in which components need to be deployed. Instead, they define what applications should look like when they are deployed and operational. In addition, the model files define dependencies between components; these dependencies are used to generate a deployment plan that can bring the application from whatever current state it is in into the desired target state.

Further, Skopos models can define lifecycle-related attributes of components, including additional actions or verification steps that need to be executed when deploying instances of the component (e.g., adding/removing the component from a service discovery service, checking the instance's correctness of operation, uploading files, etc.)

Skopos accepts several different file formats for the model. This document defines the native Skopos model format; all other formats supported by Skopos are automatically converted to this native format.

Here is a simple application model, shown visually:

model-visuals

Here is what the model file for this application looks in YAML:

# Sample two-tier application model

doctype: com.datagridsys.doctype/skopos/model
version: 1

components:
  back:
    image: myregistry/back:1.0
    provides:
        ports:
            - "8080"
    visual:
        x: 720
        y: 280
    replicas: 2

  front:
    image: myregistry/front:1.1
    provides:
        ports:
            - "8000"
    uses:
        back: {}    # default implies strict start order, back must be running
    visual:
        x: 400
        y: 280
    replicas: 2

gateways:
  elb:
    type: load_balancer
    exposes:
        name: http          # name is optional
        port: "80"          # required; may be in the form 80/tcp, 53/udp; if no protocol, tcp
        target_port: "8000"
    target: [ front ]
    visual:
        x: 80
        y: 280

  consul:
    type: external_service  # primarily for documentation purposes
    visual:
        x: 560
        y:  80

The minimum required elements of a model are:

  • a header, identifying the document format
  • a components section with at least one application components and target image to deploy

Optionally, the model may also contain:

  • additional attributes about each component, including target number of replicas, configuration and dependencies on other components, as well as visual layout
  • a gateways section containing network gateways, load balancers and other service components that are not part of the application and are used by it by reference (i.e., they exist outside of the application's deployment and their lifecycle is outside of this application's lifecycle)
  • a lifecycle section that defines custom steps to be performed before and/or after deployment

Note that the model defines the logical elements of an application that are expected to be the same across multiple instantiations of the application (e.g., staging vs. production environment). The Target Environment Descriptor binds these elements to particular infrastructure resources in each instantiation of the model.

Skopos models also can contain references to substitution variables in order to parameterize the models for different target environments/deployments. See Variable Substitution below.

Header section

The header section of the model should contain the following two elements:

doctype: com.datagridsys.doctype/skopos/model
version: 1

This header identifies the YAML file as a Skopos native model descriptor and provides forward compatibility with later versions of Skopos.

components section

The components section contains zero or more component definitions. The order of components is not important, although it is practical to arrange the components in a meaningful order (e.g., following the visual layout, going from left to right or from right to left).

Each component in this section represents a particular role in the application. For example, a worker component may have multiple replicas of the same image but is logically responsible for a single role. In another example, an application may use two different roles with the same image (e.g., one MySQL server for inventory and another for invoices); even though they use the same image, they will be defined as separate components as their logical roles differ (and so will, likely, their dependencies and other attributes).

Component names must be alphanumeric and are case sensitive; they may contain hyphens or underscores, although keep in mind that some execution environments may not support both. The component names are associated with each instance that is created or accessed by Skopos (usually via labels for Docker and AWS EC2 but the exact mechanism depends on the selected execution environment).

The section looks like this:

components:
    componentA:
        # component A attributes
    componentB:
        # component B attributes
    ...

Each component has the following attributes:

image

(Required) The image attribute defines the component image to be used when creating new component instances during the deploy process. It may also be used to distinguish between the new (target version) running instances and old (other than target version) running instances when preparing the deployment plan.

This is considered the target version of the image. The value of this image attribute may be provided verbatim to environment plugins (e.g., to do docker pull or docker create)

components:
  mysql:
    ...
    image: mysql:5.7.17

It is recommended that the image attribute identifies a unique, immutable image version. If the image attribute represents a mutable image, such as myimage:latests consider using force_replace set to true in the lifecycle section. (In an upcoming version of Skopos this may not be necessary.)

visual

(introduced in Skopos 0.8)

Defines the visual attributes of the component. Currently supports only x and y coordinates of the component position. For example:

components:
  mysql:
    ...
    visual:
      x: 200
      y: 140

X and Y coordinates for visual placement when the model is displayed in the Skopos user interface. When defining coordinates manually, it is reasonable to assume a placement grid of approximately 200 x 200 for compact and 300 x 300 for looser placement.

The visual section supersedes the now deprecated pos_x and pos_y attributes.

pos_x, pos_y (deprecated)

A deprecated form of the x and y position now within the visual attribute. Kept for backward compatibility.

replicas

Target number of instances (aka replicas) to be created of this component. If this attribute is omitted, the default is 1. The explicit value of 0 is supported (starting in Skopos 0.8)

components:
  web:
    ...
    replicas: 8

Automatically generated plans will always provide steps to bring the number of instances of the component to the specified number of replicas.

singleton

A boolean attribute that designates the component as a singleton, meaning that at most one instance of it can exist at any given time. This represents a limitation imposed by the type of component or by its role in the application, which, in most cases, is due to its use of an exclusive resource (e.g., a host port, read-write disk volume). In other cases, the limitation may be also due to application's functionality, e.g., a processor service that must ensure that all incoming requests are processed in strict order (and having two of them may break this requirement).

components:
  mysql:
    ...
    singleton: true

Possible values are true and false. If the attribute is not provided, false is assumed; there is no need to specify the false value explicitly.

Skopos uses the singleton attribute to choose an appropriate deployment strategy for the component.

Note that singleton components cannot have their replicas attribute set to a value above 1.

stateful

A boolean attribute that designates the component as stateful. Stateful components need special considerations when being deployed in order to ensure their binding to the correct state. A component may be stateful due to having persistent state, e.g., by having a storage volume connected to it; another reason to designate a component as stateful is to provide a unique identity (e.g., member ID within a cluster).

components:
  mysql:
    ...
    stateful: true

Possible values are true and false. If the attribute is not provided, false is assumed; there is no need to specify the false value explicitly.

Skopos uses the stateful attribute to choose an appropriate deployment strategy for the component. In addition, the visual user interface shows a disk icon in components designated as stateful.

Unlike the singleton attribute, the stateful attribute does not limit the number of replicas the component may have.

The current version of Skopos has limited support for stateful components with multiple replicas. Either also set singleton: true or define an explicit deploy strategy to ensure that deployments can be planned properly for these components.

command

The command attribute specifies the command to execute on container start as a list of strings. It may be used to over-ride the default entrypoint specified in an image manifest.

components:
  busybox:
    ...
    command: ["sleep", "1", "2", "3"]

A command specified in the plugin specific configuration for a component takes precedence over this attribute.

args

The args attribute specifies additional arguments to pass to the command which is executed on container start.

components:
  busybox:
    ...
    command: ["sleep", "1", "2", "3"]
    args: ["4", "5", "6"]

Arguments specified in the plugin specific configuration for a component take precedence over this attribute.

env

This section defines environment variables to be set on all instances of the component. Values can be string constants or variables (e.g., provided through the Target Environment Descriptor).

components:
  mysql:
    ...
    env:
      RACK_ENV: development
      SHOW: 'true'
      TIMEOUT: '30'
      USER_SECRET: '{{.secret1}}'
      FRUIT_URL: 'http://{{.host}}:{{.port}}/tangerine'

Note that any boolean or integer values need to be enclosed in quotes in order to prevent the YAML parser from converting them (values must remain strings).

Variables defined in the TED file can be referred to by using the Go language template syntax. It is possible to combine multiple variables to produce the desired environment value (e.g., the FRUIT_URL in the example above).

Note: this section may be ignored by plugins that don't interact with the container API directly. For example, it works with the docker core plugin but not with the ec2-asg plugin (which let's the AMI/userdata choose how and whether a container will be created); such plugins can be extended to use this section.

volumes

This section defines volumes to be mapped into instances of the component.

components:
  mysql:
    ...
    volumes:
      data:     # logical volume name, used to map to physical in the TED file
        path: '/var/lib/mysql'
      init:
        path: '/docker-entrypoint-initdb.d'
        readonly: true

For each volume, a logical volume name is used as a key under volumes and the following two attributes:

  • path (required) filesystem path inside the instance filesystem where the volume should be mounted
  • readonly (optional, true or false, default false) whether the volume should be mapped as read-only (by default, volumes are mapped read-write)

The logical volume name is a string (alphanumeric, with the same charset rules as component names) and should represent the role of the volume within the component. The name may be used by Skopos plugins to determine what volume resource to provide to the instance (see the configuration section in the Docker plugin for an example); if no mapping is provided, the plugin may use the logical volume name to derive the resource (e.g., data or prefix it with the project name)

Note: this section may be ignored by plugins that don't interact with the container API directly. For example, it works with the docker core plugin but not with the ec2-asg plugin (which let's the AMI/userdata choose how and whether a container will be created); such plugins can be extended to use this section.

provides

This section defines the resources and capabilities that the component provides for use by others. This section is complemented by the uses section which defines what resources and capabilities this componenet uses from other components.

Currently, there is only one type of resource that can be provided, ports, which defines what network ports the component makes available for use.

The ports sub-section is a list of strings, containing port number and protocol; if the protocol is omitted, tcp is implied:

components:
  mysql:
    ...
    provides:
      ports:
        - "3306/tcp"
        - "4567/udp"
        - "8080"      # tcp is implied

uses

This section defines what the component uses from other components in the application (and, optionally, from external services). This section complements the provides section which defines what resources and capabilities this componenet provides to other components.

components:
  web:
    ...
    uses:
      mysql:        # name of component being used
        start_order: tolerant  # strict(default), tolerant or independent
        ports:
          - "3306/tcp"

The uses section is organized as a map by component name, with a sub-section for each component that is used by this component (in the example above, the mysql component is being used by the web component).

The following attributes are supported in the uses section:

  • start_order defines what is the start order dependency, if any. By default, if the start_order attribute is omitted, strict dependency is assumed. The avilable values are strict (mysql must start before web), tolerant (they can start at the same time or slightly out of order), or independent (web can work regardless of whether mysql is running).
  • ports defines what network ports this component uses from the specified component; it uses the same syntax as the ports sub-sectin in the provides section: a list of strings, with a port number and optional protocol name: tcp (default) or udp

The attributes in this section, among other things, affect the sequence in which components are deployed, upgraded or torn down.

labels

This section defines user-specified metadata to be attached to all instances of the component.

The labels section is organized as a map of key-value pairs:

components:
  web:
    ...
    labels:
      env: prod
      region: us-west1
      role: web
      empty: ''

Note that some execution environments may restrict the character set and/or the length of keys and values. In general, valid DNS names are OK as keys (e.g., env, com.example.meta1) and free-form strings of at least 1024 characters as values (including non-ASCII) should work

lifecycle

This section contains configuration and steps that can be used to customize the deployment and other operation aspects of the component.

See the Component Lifecycle document for details.

plugin

This section can provide additional instance specification data to be passed to specific plugins. This allows defining parameters and instructions that have no equivalent within (the current version of) the Skopos model.

components:
  mysql:
    ...
    plugin:
      docker:
        HostConfig:
          RestartPolicy: always

In this section, sub-sections can be defined for one or more plugins. The data for each plugin is provided raw (just YAML-parsed) to the respective plugin, only when that plugin is being used. For example, any data for the docker plugin will be passed to it only when it is the core plugin; and will be ignored if the docker plugin is not used (e.g., if the core plugin is emulator)

Note that if there is a conflict or duplication between an attribute defined in the model (e.g., image) and the raw configuration, the corresponding plugin may fail operations in order to avoid ambiguity.

depends_on

This section is planned to be deprecated in a future version of Skopos; (consider using the uses section instead)

Defines the dependencies of this component on other components (and gateways/services), as well as how it depends on them.

A component may depend on zero or more components; a component may depend on another component in one or more ways.

The dependencies are defined in the following format:

depends_on:
    other_componentA:  # name of the component we depend on
        type: start        # how we depend on it
    other_componentB:
      type: start

The following dependency type values are supported:

  • start - the other component must be started before this component is created.
  • reconfig - the other component must be recongifured when this component's instances are changed (e.g., a load balancer that needs to be kept up to date with the list of instances to direct traffic to)

If a component does not depend on other components, the depends_on attribute for the component should be omitted. When a component uses another component (as declared in its uses section), Skopos will automatically assume dependencies derived from the attributes in the uses section. Any explicit dependencies declared in the depends_on section take precedence over the implied dependencies.

class (deprecated)

Component "class" name - a short version of the image name (e.g., mysql or worker). If provided, it may be used for display instead of the image.

components:
  mysql:
    ...
    image: mysql:latest
    class: MySQL

Note that this attribute may not be supported in future versions as it can cause ambiguity and confusion (e.g., if the image attribute is updated and the class attribute is not)

ver (deprecated)

Component target version. A short, display version of the image's target version. If provided, it may be used for display instead of parsing it from the image.

components:
  redis:
    ...
    image: redis:3.2.6-alpine
    ver: 3.2.6

Note that this attribute may not be supported in future versions as it can cause ambiguity and confusion (e.g., if the image attribute is updated and the version attribute is not)

gateways section

The gateways section defines additional components (gateways or services) that this application uses but are not provisioned together with the application.

They can be included in the model for two important reasons:

  • reconfiguration when instances change (e.g., load balancers, host ports, consul)
  • documenting the relationship (e.g., a database service)

The syntax of the gateways section is similar to the syntax of the components section; the names of gateways must not duplicate names of existing components in the model since both gateways and components can be used to document dependencies.

gateways:
  gatewayA:
        # gateway A attributes
    gatewayB:
        # gateway B attributes

Gateway names must be alphanumeric and are case sensitive; they may contain hyphens or underscores, although keep in mind that some execution environments may not support both.

The gateway names in the model are "logical" names (roles). They are also used as keys in target environment descriptor files to provide the gateway's name or ID in the target environment/instance (e.g., the gateway name of a AWS ELB load balancer for the staging deployment).

Note that it is not necessary for Skopos to be aware of all gateways, only the ones that you want to be configured (and/or documented in the model diagram).

type

Type of the external gateway or service.

gateways:
  myinput:
    type: load_balancer
    ...

Starting with Skopos 0.8, the following values are supported for the type attribute:

  • load_balancer - a load balancer gateway allowing multiple instances to be connected to a single service point
  • host_port - a single-instance gateway, exposing a network port of a single component
  • external_service - an external service (currently used only for documentation purpose)

Prior to Skopos 0.8, the type value was free form (use any alphanumeric string); it is case sensitive. Starting with version 0.8, the type must be one of the supported values.

exposes

This section defines a list of ports that a gateway exposes to the outside world. Optionally, for each exposed port, you can define a name and a target port number (if different from the exposed port).

gateways:
  myinput:
    ...
    exposes:
      - name: monitoring  # optional name, just for documentation
        port: 8000
        target_port: 80   # omit this attribute if same as 'port'

For a host port gateway, the port list can have only one element. For load balancer gateways, any number is supported. This section is not used for external services but it can be present for documentation purposes (except for the target_port attribute)

List of ports to serve; for each item, the following attributes are defined:

  • name - optional name for the port's service (e.g., http or invoices). Not used.
  • port - port number (and protocol) to be exposed, e.g., 80/tcp (tcp and udp are supported, tcp if a protocol is not provided)
  • target_port - port number to which to send traffic to connected component instances (if different from port)

target

List of components to which this gateway should forward traffic.

gateways:
  myinput:
    ...
    target: [ websrv ]

Typically, there is only a single target component. If the component has multiple replicas, then the gateway should be of type load_balancer; all deployed instances of the component will be connected to the load balancer.

The target section implies a reconfig dependency between the gateway and the target component(s), meaning that when the set of instances changes, the gateway will be reconfigured. This implied dependency obviates the need for explicit depends_on section in the gateways.

The list form of target is rarely used. It allows requests to be directed to two or more different components (or versions of components) via a load balancer:

components:
  websrv-v1:
    image: myapp/websrv:1
    replicas: 50    # stable version, gets most of the traffic
    ...

  websrv-v2:
    image: myapp/websrv:2
    replicas: 5     # canary version, gets just 10% of traffic
    ...

gateways:
  myinput:
    type: load_balancer
    ...
    target:
      - websrv-v1
      - websrv-v2

depends_on

This section is planned to be deprecated in a future version of Skopos; (consider using the target section instead)

This section expresses dependencies of gateways on other components in the model. It is used primarily for the reconfig dependency type, so that a load balancer (or another gateway- or service discovery-type service) can be configured with the instance IDs (or IP addresses) of the deployed instances.

gateways:
  myinput:
    ...
    depends_on:
      websrv:             # name of the component we depend on
        type: reconfig    # how we depend on it (reconfigure on change)

Only the reconfig value is supported for the type attribute.

If a gateway/service does not have a dependency on model's components, this depends_on attribute should be omitted (e.g., for external_service gateways).

visual

(introduced in Skopos 0.8)

Defines the visual attributes of the gateway. Currently supports only x and y coordinates of the component position. For example:

gateways:
  myapp-staging-lb:
    ...
    visual:
      x: 20
      y: 140

X and Y coordinates for visual placement when the model is displayed in the Skopos user interface. When defining coordinates manually, it is reasonable to assume a placement grid of approximately 200 x 200 for compact and 300 x 300 for looser placement.

The visual section supersedes the now deprecated pos_x and pos_y attributes.

pos_x, pos_y (deprecated)

A deprecated form of the x and y position now within the visual attribute. Kept for backward compatibility.

plugin

This section can provide additional gateway specification data to be passed to specific plugins. This allows defining parameters and instructions that have no equivalent within (the current version of) the Skopos model.

components:
  web:
    provides:
      ports:
        - "8080/tcp"
    # ...
#...
gateways:
  in:
    type: host_port
    target: [ web ]
    exposes:
      - name: http
        port: 8081/tcp
        target_port: 8080
    plugin:
      docker-swarm:
        publish_mode: host
        raw:
          PublishedPort: 8000   # overrides port '8081/tcp' with '8000'

In this section, sub-sections can be defined for one or more plugins. The data for each plugin is provided raw (just YAML-parsed) to the respective plugin, only when that plugin is being used. For example, any data for the docker-swarm plugin will be passed to it only when it is the core plugin; and will be ignored if that plugin is not used (e.g., if the core plugin is emulator or docker)

Note that if there is a conflict or duplication between an attribute defined in the model (e.g., port) and the raw configuration, the corresponding plugin may fail operations in order to avoid ambiguity or malfunction. See the plugin-specific settings sections in the references for the plugins you use for more specific information.

lifecycle section

The lifecycle section defines additional steps and configuration that affects the application as a whole, including pre-flight, post-deployment and post-teardown steps.

Typical pre-flight steps may include:

  • making sure required resources exist (e.g., create docker network or EC2 load balancer)
  • verifying image availability and access
  • running external actions, such as logging in into docker registries, putting up maintenance pages, posting deployment begin/end messages to chatops, etc.
lifecycle:
  pre_flight:
    steps:
      - call:
          plugin   : docker
          action   : network_create
      - call:
          plugin: docker
          action: image_check
          label: "Verify redis image exists"
          arguments: { arg: "{{.model.Components.redis.Image}}" } # from model

See the Application Lifecycle reference for details.

Variable Substitution

Skopos models can contain references to variables defined in the Target Environment Descriptor. This is useful when deploying multiple copies of an application to different target environments.

Variable substitutions use a Linux shell-style syntax: ${VARIABLE}. Here is an example of using variable substitution:

A sample environment file defining 3 variables (make sure the values are all string values by surrounding them with quotes):

#...
vars:
  env_type: "development"
  front_ver: "latest"
  port: "8080"

A sample model file using variable substitution:

doctype: com.datagridsys.doctype/skopos/model
version: 1

components:
  front:
    image: myregistry/front:${front_ver:-1.1}  # use version 1.1 if none defined
    replicas: 2
    env:
      PROD_ENV_TYPE: "${env_type:-production}"  # use 'production' if not defined
    provides:
      ports:
        - "8000"
    visual:
      x: 720
      y: 280

#...

gateways:
  elb:
    type: load_balancer
    exposes:
        name: http
        port: "${port:-80}"   # use 80 if port is not defined
        target_port: "8000"
    target: [ front ]
    visual:
        x: 80
        y: 280

In addition to simple value substituion, Skopos supports default values using the conventional shell syntax:

  • ${VARIABLE:-default} evaluates to default if VARIABLE is not set or empty.
  • ${VARIABLE-default} evaluates to default only if VARIABLE is not set.

Other shell-style variable substitution forms, like $VARIABLE or ${VARIABLE/foo/bar}, are not supported.

Referencing an undefined variable results in an error (unless using the :- default format); Skopos does not assume the empty value as it is rarely what is needed.

While any string value in the model can contain references to variables, some are more useful than others. The following model attributes are typically used with substitution variables:

  • image - various parts of the image specification may be replaced:
  • image: newco/myimage:${ver} replaces just the version tag
  • image: ${repo}/newco/myimage:1.0 uses a configurable repo URL (e.g., dev vs. prod or using different repo URLs in different geographies)
  • image: ${image_spec} replaces the full image specification
  • image: ${repo:-123456.dkr.ecr.us-east1.amazonaws.com}/myimage:${myver-staging} replaces both the image repo URL and the tag, providing defaults (ECR URL and a staging tag)

  • env - environment variables used to configure container instances

  • http_proxy: ${http_proxy}

  • gateway port - exposed port for load_balancer and host_port gateways

  • port: "${port:-80}"

  • injection step arguments:

  • `arguments: [ "log", "Starting web server for ${env_type} environment on port ${port}"]
  • note that injections steps also support replacement of special variables using the {{.var}} syntax

Currently, it is not possible to specify the number of replicas through variable substitution. This limitation will be removed in an upcoming version.

Note that by using multiple target environment files, it is possible to construct various configurations with target environment files as building blocks. For example, placing the image version tags variables in one target environment file and the env configuration vars in a different file, allows two or more different deployments to share the same model and version tags but have different configuration parameters (e.g., AWS load balancer name, credientials, repo URLs, etc.).