Skopos™ Model Lifecycle Reference

The Skopos Model file may optionally contain a lifecycle section for the application or for any of the components.

The application lifecycle section describes the transformation of the application at the application scope. For example, it allows for customization of pre-flight, post-deployment or post-teardown steps, or for handling the teardown of orphans (instances found in the application environment but not present in the application model).

A component lifecycle section describes the transformation of a component within its own scope. For example, it allows for the injection of custom steps into the generated componenent deployment plan (e.g., to verify created instances are healthy), for customizing service discovery, or for specifying componenent deployment attributes (e.g., to indicate a component requires always replacing matching instances on upgrade).

Application Lifecycle

Here is an example of an application lifecycle section in a YAML application model:

doctype: com.datagridsys.doctype/skopos/model
version: 1

lifecycle:
    teardown_orphans:  after
    pre_flight:
        steps:                      # additional pre-flight steps
            - exec: [ "maintenance_page.sh", "on" ]
            - gate: "Are you ready?"
    post_flight:
        skip_standard_steps: true   # skip standard post-flight steps
        steps:                      # additional post-deployment steps
            - exec: [ "maintenance_page.sh", "off" ]
    post_teardown:
        steps:                      # additional post-teardown steps
            - exec: [ "external_cleanup.sh" ]

teardown_orphans

The teardown_orphans attribute defines how to teardown orphans during application transformation. An orphan is an instance of a component found in the application environment but not present in the application model. For example, if an application is deployed containing instances of components A, B and C, and then the model is changed to remove any reference to component C, any extant instances of C are orphans. By default, Skopos tears down orphans before performing any other component transformations. Valid values for the teardown_orphans attribute: before | after | none. A value of none indicates orphans will not be torn down.

pre_flight, post_flight and post_teardown

By default Skopos creates the following top-level steps for all generated plans:

  • deployment plan: pre-flight and post-flight
  • teardown plan: pre-flight and post-teardown

Each of these includes standard steps which call the pre-flight, post-flight or post-teardown actions for each plugin in the environment (core, service discovery or load balancer). The application pre_flight, post_flight and post_teardown sections of the model allow these top-level steps to be customized using the following attributes:

skip_standard_steps

The boolean attribute skip_standard_steps indicates whether or not the standard plugin pre-flight, post-flight or post-teardown step(s) should be skipped. Default: false

steps

The steps attribute takes as its value a list of zero or more steps which will be injected just after the standard pre-flight, post-flight or post-teardown step(s), if any. See Injected Step Types for details of injected step types and their attribution.

Given the application lifecycle provided as an example above, the pre-flight step generated when using docker (core) and docker-elb (load balancer) plugins is shown in this image:

pre-flight-step

Component Lifecycle

Here is an example of a lifecycle section for a component named mysql, as it appears in a YAML application model:

components:
    mysql:
        ...
        lifecycle:
            force_replace: true
            quality_probe:
                steps:
                    - exec:
                        label: "Verify new mysql operation"
                        arguments: [ "verify-mysql-ok", "{{.id}}" ]

force_replace

If true, the force_replace attribute causes an existing instance which matches the desired target image to be force-replaced during an upgrade. Default: false

quality_probe

The quality_probe is a named injection point for a component. An injection point is a logical location in a component deployment plan where custom steps may be injected. Logically, the quality_probe is injected after attaching any newly created instance to any service discovery or ingress. This injection point is commonly used to verify the operation or performance of newly created instances. In the example here, an exec, or executable, step is injected which executes the script verify-mysql-ok passing to it as an argument the ID of the newly created instance.

The quality_probe is one of several standard component injection points. See Component Injection Points for a description of all such injection points. See also Injected Step Types for details of injected step types and their attribution.

Given the mysql component lifecycle provided as an example above, the deployment plan for this component generated when using docker as the core and service discovery plugin is shown in this image:

mysql-comp-deploy

Component Injection Points

An injection point is a logical location in a component deployment plan where custom steps may be injected.

Standard Injection Points

Standard injection points allow for the injection of one or more steps of any of the supported step types: exec (script), call (plugin), gate (manual gate) or delay. If more than one step is specified for a given injection point, these steps are expressed in the component deployment plan in the order they are specified in the model. Standard injection points include:

  • pre_component: steps added at the beginning of the component plan. Example: manual gate
  • post_component: steps added at the end of the component plan. Example: manual gate.
  • pre_instance_create: steps added to the component plan just before creating any new instance. Example: custom image fetch mechanism (e.g, requiring auth/login), or even an image build script for use in development environments.
  • post_instance_create: steps added to the component plan just after creating any new instance. Example: verify instance operation before live traffic.
  • post_instance_attach: steps added to the component plan just after attaching any instance to ingress/service discovery. These steps are included on attaching a new instance or re-attaching an old instance.
  • quality_probe: steps added to the component plan after post_instance_attach of a newly created instance only (not after re-attaching an old instance on cleanup). Example: verify instance operation (live traffic).
  • post_instance_detach: steps added to the component plan just after detaching any instance from ingress/service discovery. These steps are included on detaching old or detaching failed new. Example: drain instance of outstanding connections.
  • pre_instance_destroy: steps added to the component plan just before destroying any instance (old or failed new). Example: copy data (e.g., from instance or volume).
  • post_instance_destroy: steps added to the component plan just after destroying any instance (old or failed new). Example: custom cleanup (e.g., volumes).
  • on_change: steps added immediately after any operation that can create or destroy instances, including operations in the cleanup/recovery path of a generated deployment plan. Note that for some deployment plans (e.g., upgrades that include separate 'create new' and 'destroy old' steps), the steps for this injection may be repeated in more than one place in the component's deployment plan.

In practice, the post_instance_create and quality_probe injection points are most commonly used. Because injected steps are expressed logically according to the generated component plan, their use is most easily understood through examining visual examples of generated component plans. Below are several examples for a stateless component named vote whose component lifecycle includes injected steps as follows:

components:
    vote:
        image: vote:2.0
        replicas: 3
        lifecycle:
            post_instance_create:
                steps:
                    - exec:
                        label: "Verify new vote operation"
                        arguments: [ "verify-vote-ok", "{{.id}}" ]
            quality_probe:
                steps:
                    - exec:
                        label: "Verify new vote response time"
                        arguments: [ "verify-vote-perf", "{{.id}}" ]

vote component: create 3 instances

vote-deploy

vote component: upgrade 3 instances

vote-upgrade

Below are equivalent examples for a singleton component named mysql

mysql component: create new

mysql-deploy

mysql component: upgrade

mysql-upgrade

Service Discovery Injection Points

Service discovery injection points allow for the injection of steps after the standard service discovery attach/detach steps, if any, and before the standard ingress (load balancer) attach/detach steps, if any. These injection points also support a boolean attribute skip_standard_steps which if true causes the standard service discovery attach/detach steps to be skipped (Default: false). Service discovery injection points only support the call (plugin) step type. If more than one step is specified for a given injection point, these are injected in the order specified in forward steps, and in reverse order in cleanup steps.

  • service_discovery_attach: steps added after standard service discovery attach step(s), if any.
  • service_discovery_detach: steps added after standard service discovery detach step(s), if any.

Below are examples for a stateless component named vote whose component lifecycle skips standard service discovery and includes injected custom service discovery steps as follows:

components:
    vote:
        image: vote:2.0
        replicas: 3
        lifecycle:
            service_discovery_attach:
                skip_standard_steps: true
                steps:
                    - call:
                        plugin: my_custom_plugin
                        action: sd_attach
                        label: "custom service attach"
                        arguments:
                            selector: "{{.id}}"
            service_discovery_detach:
                skip_standard_steps: true
                steps:
                    - call:
                        plugin: my_custom_plugin
                        action: sd_detach
                        label: "custom service detach"
                        arguments:
                            selector: "{{.id}}"

vote component: create 3 instances

vote-deploy

vote component: upgrade 3 instances

vote-upgrade

Injected Step Types and Attribution

Each injected step supports the following optional attributes:

  • label: step label - a replacement string for the auto-generated step label
  • on_fail: a list of strings indicating actions to take if the step fails. Valid values include:
    • next: replace the on_fail goto value in the compononent plan with next - causes the step to procede on failure
    • warn: on failure log a warning message
    • info: on failure log an informational message
    • pause: on failure, pause the plan before continuing

Skopos supports the following injected step types.

Exec (script) step

The exec step executes the script specified as the first of the arguments, passing to it the remaining arguments, if any. This step supports the following additional attributes:

  • arguments: (required) a list of string values. At least the first argument (script name) must be specified.

The exec step supports abbreviated and full syntax as demonstrated in these examples:

- exec: script.sh                   # abbreviated syntax
- exec: [script.sh, arg1, arg2]     # abbreviated syntax
- exec:                             # full syntax
    arguments: [script.sh, arg1, arg2]
    label: "my custom label"
    on_fail: [warn, next]

Call (plugin) step

The call step calls a plugin with a specified action. This step supports the following additional attributes:

  • action: (required) the plugin action
  • plugin: the plugin to call (defaults to the core plugin if not specified)
  • arguments: a list of key-value pairs to be passed as arguments to the plugin

The call step supports full syntax as demonstrated in this example:

- call:                             # full syntax
    plugin: docker
    action: service-signal
    arguments: {service: worker}
    on_fail: [warn, next]

Probe step

The probe step calls a containerized application probe with a specified image. A probe packages a Skopos plugin as a container image where the plugin is configured to be invoked as the container's entrypoint. See Calling Plugins as Application Components for details. This step supports the following additional attributes:

  • image: (required) the probe image
  • action: the probe action (defaults to service_up)
  • arguments: a list of key-value pairs to be passed as arguments to the probe
  • timeout: maximum time to wait for the probe call to complete, in seconds. Default is 300 seconds (5 minutes)
  • network: name of the application network to which the probe will connect (default by default; advanced use)
  • logfile: (for some systems) configure the probe's error output to be sent to a file in the Skopos container. This is useful as a last resort when diagnosing probe action failures in environments that cannot obtain the log (e.g., docker service logs is an experimental feature in Docker Swarm 17.03, disabled by default). Note that having this argument redirects the plugin error output, hiding it from the Skopos engine's error processing.

The probe step supports abbreviated and full syntax as demonstrated in these examples::

- probe:  my_registry/probe-customer-db    # abbreviated syntax
- probe:                                   # full syntax
    image: my_registry/probe-customer-db
    action: verify_customer_db
    arguments: {port: 6379}
    label: "Verify customer db completeness and integrity"
    timeout: 180
    network: my_network
    logfile: "/tmp/my_probe.log"
    on_fail: [warn, next]

Gate (manual) step

The gate step is a simple manual gate requiring user confirmation to continue or abort. This step supports the following additional attributes:

  • message: (required) the message to display

The gate step supports abbreviated and full syntax as demonstrated in these examples:

- gate: "Are you ready?"            # abbreviated syntax
- gate:                             # full syntax
    message: "Are you ready"
    label: "Confirm before proceding"

Delay step

The delay step introduces a delay in execution of the component plan. This step supports the following additional attributes:

  • duration: (required) span of delay in seconds

The delay step supports abbreviated and full syntax as demonstrated in these examples:

- delay: 10                         # abbreviated syntax
- delay:                            # full syntax
    duration: 10
    label: "Delay 10 seconds"

The delay duration must be a positive number, and can be a fraction in any floating-point number format supported by the YAML specification.

Special Variables Substitution

When defining injected steps, Skopos defines several variables that refer to deployment state of your application. These variables use a Go template syntax that looks like this: {{.component}}. These special variable and syntax are available only in arguments of injection steps (script exec, plugin call and manual gate messages).

Note that the regular variable substitution also applies and is performed prior to the special variable substitution.

The following special variables are supported only in lifecycle injected steps for components:

  • .project - project name, as provided to Skopos when loading the application model (e.g., myproj in skopos load --project myproj model.yaml)
  • .component - the name of the current component in the model
  • .image - the target image of the current component (from the image attribute in the model)
  • .id - the instance ID of the component instance for which the steps are invoked (core plugin-specific identifier, e.g., container ID in Docker, instance ID in EC2, etc.)
  • .ipaddr - the IP address of the component instance for which the steps are invoked (this variable may not be supported by all deply templates yet - plase contact our support if this variable does not resolve in your project)
  • .index - index of the component instance (when executing steps for multiple replicas); 0-based

The following special variables are supported in lifecycle injected step both for components and for the application as a whole (e.g., pre_flight)

  • .Model.Components.<component>.Image - target image to deploy for a <component>. It provides the value of the image attribute of a component from the model. It can be used, for example, with the docker plugin action image_check or image_pull to ensure that the target image is available prior to starting the destructive steps of the plan. Unlike .image, this variable can be used outside of component steps, e.g., in a pre-flight group step. The same syntax can be used to access any attribute of the model
  • (deprecated) .vars.<varname> - user-defined variable from the vars section in the model or target environment descriptor. Using the special variable syntax for user variables is deprecated, use ${VARIABLE} instead.

Calling Plugins As Application Components

By default, Skopos runs all plugins in the context of the Skopos engine itself. In most environments, this provides sufficient isolation and access to application components (services).

In many advanced environments, application components can be or are restricted to application-specific networks, providing for better isolation between applications running on the same cluster. This, however, may create problems for Skopos plugins that need to interact with the application components over their network (e.g., REST call for health check or API requests).

For these cases, Skopos provides a mechanism for calling plugins as instances of the application so that they have the same access to application's internal services as any other application component.

This mechanism is facilitated by the core plugin used for a given application.

Calling Skopos built-in plugins

Since Skopos is already packaged as a container that can be executed multiple times, when calling Skopos standard plugins, it is possible to use the existing Skopos container image to invoke the plugins. This is the recommended approach to calling plugins that are built-ins (e.g., http).

The form of the injection step is shown below; note that the arguments of the plugin-to-be-called are passed as usual and, in addition, a few special parameters define which plugin to call and how to call it:

- call:
    action: call_builtin_plugin     # calls the core plugin to start another plugin as part of the application
    arguments:
        plugin_name: http               # name of the plugin to call
        plugin_action: get_wait         # action of the plugin to call (see plugin's reference)
        plugin_timeout: 300             # how long to wait, in seconds, for the operation to complete
        selector: {{.id}}
        port: 8080
        path: '/healthz'
        timeout: 300

The following parameters control the plugin invocation:

  • plugin_name: (required) the name of the plugin to invoke, e.g., http
  • plugin_action: (required) the action to invoke on that plugin
  • plugin_timeout: maximum time to wait for the plugin call to complete, in seconds. Default is 300 seconds (5 minutes)
  • plugin_image: specify a Skopos container image/version to use, overriding the default (advanced use). The default image is the same version as Skopos (opsani/skopos:VER, where VER is the Skopos semver, e.g., 0.11.13); you may want to use this setting if you re-package the Skopos container or place it in your own private registry
  • plugin_network: name of the application network to connect to the plugin (default by default; advanced use)
  • plugin_logfile: (for some systems) configure the plugin's error output to be sent to a file in the Skopos container. This is useful as a last resort when diagnosing plugin action failures in environments that cannot obtain the log (e.g., docker service logs is an experimental feature in Docker Swarm 17.03, disabled by default). Note that having this argument redirects the plugin error output, hiding it from the Skopos engine's error processing

Calling containerized plugins (user plugins)

This mechanism allows for invoking plugins packaged as containers and running them as part of the application that is deployed by Skopos. It is intented for user-defined plugins, or for any plugin which may benefit from invocation as an application component. For built-in plugins, see the section above.

The containers to be used with this mechanism should have the desired plugin installed and configured to be invoked as the container's entrypoint. Skopos invokes this entrypoint with the following arguments:

  • name of the action to perform (plugin action, e.g., check_access)
  • optional redirection settings (--stdout <file> and --stderr <file>), used when plugin_logfile is specified
  • a single string with a JSON object containing action's arguments and additional engine state

Currently, the output of the plugin is ignored; its standard error output is captured and passed to Skopos as error text in case the container terminates with an exit code other than zero.

To call a containerized plugin, use the probe step syntax described above.

The following syntax is deprecated, and is retained as a reference.

The deprecated form of the containerized plugin injection step is shown below; note that the arguments of the plugin-to-be-called are passed as usual and, in addition, a few special parameters define which plugin to call and how to call it:

- call:
    action: call_plugin     # calls the core plugin to start another plugin as part of the application
    arguments:
        plugin_image: opsani/http:0.11.13     # image of the plugin to call
        plugin_action: get_wait         # action of the plugin to call (see plugin's reference)
        plugin_timeout: 300             # how long to wait, in seconds, for the operation to complete
        selector: {{.id}}
        port: 8080
        path: '/healthz'
        timeout: 300

The following parameters control the plugin invocation:

  • plugin_image: (required) the container image to run (in the same form as component image specification; OK to use (substitution vars)[MODEL-REF.md#variable-substitution]
  • plugin_action: (required) the action to invoke on that plugin
  • plugin_timeout: maximum time to wait for the plugin call to complete, in seconds. Default is 300 seconds (5 minutes)
  • plugin_network: name of the application network to connect to the plugin (default by default; advanced use)
  • plugin_logfile: (for some systems) configure the plugin's error output to be sent to a file in the Skopos container. This is useful as a last resort when diagnosing plugin action failures in environments that cannot obtain the log (e.g., docker service logs is an experimental feature in Docker Swarm 17.03, disabled by default). Note that having this argument redirects the plugin error output, hiding it from the Skopos engine's error processing