ec2-asg plugin

The ec2-asg plugin uses EC2 auto-scaling groups (ASG) to deploy components as EC2 virtual machines, with 1:1 mapping of instance to virtual machine. It is intended to be used with docker images deployed by each VM, although the ec2-asg plugin does not enforce this.

The plugin also supports the EC2 classic load balancer (ELB), allowing component instances to be automatically added/removed to/from the load balancer. It can also wait until load balancer recognizes the instance as healthy (or fail with timeout) in order to ensure that only successfully working instances are considered deployed.


The ec2-asg plugin creates auto-scaling groups (ASG) for each component in the application and creates matching launch configurations for each ASG. The image ID specified for each component in the application model will be inserted into the launch configuration's user_data text, allowing the "cloud init" (or "cloud config") startup script to load and run the specified image.

Details and cautions

NOTE: a new launch configuration is created only if one of the following conditions are met:

  • a launch configuration for this component did not already exist
  • there are no running instances of this component that have the same image_id (in this case, even if an LC already existed, it will be replaced by a new one, to ensure that the current configuration will be reflected in it)

If a launch configuration exists and appears to be used by running instances and the associated ASG is tagged as having the same image_id, the existing LC will be re-used. This means that if Skopos is used to increase the number of instances of an already-active component, the new instances will have the same configuration as the ones that are running, ignoring any configuration provided in the environment file.

NOTE: this also means that ec2-asg cannot be used to re-install an application with a new configuration by "upgrading" it to one with the same component versions (i.e., having the same image_ids as the running ones), even if running Skopos with --replace-all mode or having force-replace: true).

Limited support for model attributes

Starting with Skopos 0.8, the application model allows specifying attributes that apply mostly to container-based components. The ec2-asg plugin maps those to equivalent EC2 API attributes; however, not all attributes are supported (e.g., env)

The following model attributes are not mapped: env, volumes, provides, uses, labels.


Here is the format of the plugin configuration record for the ec2-asg plugin:

Note: only the "ec2-asg" item is shown here, for the full environment description format, see the Target Environment Configuration document.



    naming_template: "{prj}.{comp}_{side}"
    vpc: my-vpc
    subnets: [ "id-or-name", "..." ]

      template: another_launch_config
      ami: ami-4d795c5a
      sg: [ "sg-12345678" ]
      type : t2.nano
      user_data : "#!/bin/sh\n docker run --restart=unless-stopped {{image_id}}"

    image_id_subst: "{{image_id}}"

    # per-component launch config overrides (optional)
          type: t2.large

    # load balancer bindings
      app: "myapp-staging-lb"

Required settings

vpc - A name or ID of an EC2 VPC in which the VM instances will be created and all load balancers belong (if using the classic ELB with this plugin). This setting is optional if home_vpc is set (see below).

launch_config - a map with data needed to create an instance. The map can contain one or more of the following attributes:

  • template - name of an existing launch configuration to use as a template. If provided, most or all of the other settings can be omitted - the settings from the template config will be used as defaults.
  • ami - (string) AMI ID
  • sg - (list of strings) names or IDs of security groups that define network permissions
  • type (string) VM type, e.g., t2.micro
  • profile (string, optional) - instance profile; note the AWS GUI displays this setting as “role”
  • user_data - one of:
    • "string" - literal value to place in user_data
    • { "s3" : "bucket/key" } - retrieve value from S3. NOTE the data retrieval will be performed only when a new launch config is being created. The read be done in the context of the process in which the ec2-asg plugin runs (same credentials as those that would be used to configure EC2 objects and run instances)
  • ebs_optimized = true | false. Defaults to false.
  • public_ip = true | false. If not set, the subnet’s default will be used.
  • key - ssh key name (the name, if given, should be one of the registered ssh key names in AWS). This may be left undefined, the instances will be created without a key provided by AWS (you will need your own ssh key setup in this case).

Optional settings

region - if this setting is present, it defines the AWS API region to connect to. If not set, ec2-asg will use the AWS CLI configuration to find the region: either an environment variable (AWS_DEFAULT_REGION) or a setting in the ~/.aws/config file (youcan provide either of those when starting the Skopos container - by using the -e option to set an env variable or the -v option to map a file into the container's name space). If no setting for the region found, ec2-asg will connect to the AWS default 'us-east-1' region.

naming_template - if this setting is present, it replaces the default format used for creating EC2 object names (Auto-scaling Groups, Launch Configurations). It should be a string containing placeholder patterns for the application name, component name and 'slot' (single-character slot IDs are used to allow creating multiple ASGs and launch configs for the same component, when instances of that component exist in multiple versions concurrently). The placeholder patterns for these are {prj}, {comp} and {slot}, respectively. {prj} must be present and {comp} should be present except when working only with applications that have only one component in them. The following restrictions apply:

  • the placeholder patterns should not appear more than once in the string (e.g. '{prj}.{comp}.{prj}' is not valid);
  • other than the placeholders mentioned here, there should be no substrings enclosed in {}, nor any mismatched { and } characters in the string.
  • there must be at least one character between any two placeholders;
  • if {slot} is not present, the plugin will append '_{slot}' to the value set in naming_template;
  • {comp} may be omitted, but this makes it impossible to create more than one component in the application (the plugin needs a distinct ASG name for each component) and in this case the plugin will refuse to create any component except one named 'main'.

If naming_template is not set, the default is {prj}.{name}_{slot}.

home_vpc - If set to true, this setting enables retrieving the VPC id of the EC2 host on which skopos is running and checking that it is the same as what the vpc setting refers to. If the vpcs do not match, all attempts to use the plugin, including initial application state load will result in a failure. If home_vpc is set and the configuration does not have a vpc setting, the host's VPC is used - making the vpc setting optional. The default is false. It should not be set to true unless Skopos is running on an EC2 VM.

subnets - List of subnet names or IDs. If names are used, they will be looked up in the specified vpc. If IDs are given, they must be in the named vpc (plugin will fail if they’re not). There should be no more than one subnet per availability zone listed here. The subnets value can also be given as a string, and will be interpreted as a shell-style wildcard pattern to match against the subnets in the VPC. If not provided at all, subnets="*" is assumed (use all subnets in the VPC - note this will work only if the VPC doesn’t have multiple subnets in the same availability zone).

tags - Object, containing “key” : “string” pairs, which are set as additional tags on the ASG. Do not use tag names that start with “skopos:” here.

components - Object, containing one key per application component. Each value is an object that can contain launch_config and/or tags and/or subnets, in the same format as the global values above. If an application has a component with a matching name in the ‘components’ object, the settings found here will be merged with those in the global launch_config and tags settings and the merged configuration used when creating the launch config and ASGs for that component (with the per-component settings always taking precedence). A subnets setting for a component will override the global subnets setting.

image_id_subst - a regular expression to replace with the image ID in the user-data text. This can be any regular expression in the syntax supported by Python, as long as it is found in the 'user_data' text only where it needs to be replaced. If you are preparing a 'user_data' string specifically for use with a Skopos application, any fixed string that cannot appear in the data except at the place designated for the image name could be used. If you are replacing a fixed string that contains regular-expression metacharacters (e.g., [].?*), these should be escaped appropriately so that they match the literal character that you have in the 'user_data' template, e.g., if you have '' in the data, set image_id_subst to put\.image\.id\.\here NOTE the \ itself will need to be escaped, as it is a meta-character for the YAML/JSON text representation of the environment descriptor file. The input 'user_data' is treated as multi-line text: '^' at the beginning of the regular expression matches at the start of any line and '.' matches anything except a newline.

The ec2-asg plugin will look for regular expression matches in the user_data text and replace every occurrence with the value of the image key in the deployment model file. For example, this could be interpreted as the name of a Docker image to pull and run, e.g.,

the model file might have:

    class: backend
    pos_x: 720
    pos_y: 280
    replicas: 2
    version: '1.0'

In this example, if the specified 'user_data' string contains something like

docker run --restart=unless-stopped {{image_id}}

the launch configuration for the back component will have this in its user_data:

docker run --restart=unless-stopped

image_id_path - a string, in the form key1.key2….keyN. This is mutually exclusive with img_id_subst and can be used only if the user data text is in yaml form. The yaml document is modified like this:

doc[key1][key2]...[keyN] = the_image_id

NOTE if neither image_id_subst nor image_id_path is given, the user_data string will be left alone. In this case it is necessary to have per-component settings in the ‘components’ object, otherwise all components will be created with identical launch config.

image_check_type - string, defines how the plugin should validate and compare image IDs. If set to 'docker' (the default value), image IDs are assumed to be Docker image names that exist in a Docker registry and are accessible in the context of the Skopos engine with the equivalent of 'docker pull'. The plugin will validate any image ID received and read the image metadata from the Docker Registry to get the image digest and use it for comparing image versions - therefore an image that has changed will be seen as being of a different version, even if it has the same exact tag as an older image. Image checking is done on creating new instances and on initial state load. In the latter case, the plugin compares the digest associated with any running instances to that found in the Docker Registry for the same image ID and if they are different, it replaces the 'tag' portion of the reported image ID (normally used as a version string) with 'unknown', making it different from any other version. As a result, if an application model specifies an image ID that's the same as what was used on a previous deploy/upgrade, it will still be seen as different and cause component upgrade to be planned. This is useful when image tags such as the Docker's default ":latest" are used. image_check_type: docker should not be used whenever the image ID is not an actual accessible Docker image or when the Skopos engine has no access permissions to your image registry.

If image_check_type is set to the string none, no validation of the image IDs is done and no image digest is stored with running instances. In this case, it is expected that a given image ID always corresponds to the same exact image and replacing the image while keeping the same ID will not be detected as a change and no component upgrade will be planned (unless forced explicitly with -replace-all).

lb - Object, contains "name":"id" or "name" : {"target_group":"name-or-arn", "port":N} pairs (one for each load balancer in the target environment). The key (name) is what the LB is referred to in the application model/plan. The lb object is optional, can be omitted if no LBs are used (or if they’re not being attached/detached with the ec2-asg plugin). If the string "id" value is used, the value is the name of an EC2 TCP ('classic') load balancer. If the object value is used, the object must contain both the "target_group" and "port" values and the entry refers to an EC2 application load balancer target group. In this case, "port" is the port number on which the EC2 instances that would be targets for the LB are listening. IMPORTANT: the target group must have exactly the same availability zones as those covered by the VPC subnets listed in the subnets setting (the subnets define the set of availability zones where instances could run); if an instance is created in a zone that isn't covered by the target group, AWS will not refuse to connect it to the target group, but will mark its health status as 'unused' and never route traffic to it. Setting the value to null (e.g. "lb" { "aux": null }) is permitted and is the same as not having the named LB listed at all (this is useful when the same configuration file is reused in multiple environments and some of them do not have the LB at all; NOTE that leaving an LB mapping definition in an environment where the LB is not present in the target EC2 region/VPC will cause the plugin to fail on its pre-flight check).

join_timeout - amount of time in seconds that lb_wait_join should wait for an instance to become 'healthy' in the load balancer state. The default is 360.

inst_running_timeout - amount of time (seconds) that lb_wait_join should wait for a (newly-created) instance to have a 'running' state reported by EC2, before it attempts to check for load-balancer status. This value is independent of "join_timeout" and is applied before the "join_timeout" time, whenever lb_wait_join is executed on an instance that has just been created and its EC2 state is 'pending'. The default is 50.

Delegated Destroy

The plugin can be configured to handle instance destruction differently for one or more components. Instead of terminating them, the instances are removed from the ASG and handed off to an external API to be terminated. This can be used in cases where instance termination can take a very long time, but the component is designed for being placed in a 'terminating' state by some custom mechanism.

To enable delegated destroy, set the following in plugin_config/ec2-asg (changing the 'uri' and 'components' settings as needed):


      uri: "http://my_destroy_handler:port/delayed-termination"
      components: [ "backend", "etc.", ... ]
      # user_data is optional
      user_data: { (any type of data) }

If 'components' is not present or empty, the delegated destroy function is disabled and any other settings in 'delegated_destroy' are ignored. If it is non-empty, the 'uri' setting is required.

The server specified by 'uri' is up to the application's designer/operator to set up. The given URI should respond to GET and POST. On GET, it must return a success status, one of the valid 2xx status codes (the actual response and payload is ignored). On POST, ec2-asg will send the following data:

      "ec2_region": (the region used by the plugin itself, as configured either in the environment or in an AWS config file,
      "ec2_instance_id": (the ec2 instance ID),
      "ip_address": (the ec2 instance PRIVATE address),
      "project": (application name),
      "component": (component name from the model),
      "user_data": (the value from the configuration or 'null', if not set)

IMPORTANT: the POST request is expected to complete within 5sec. It is entirely up to the implementor what the request would actually do. The ec2-asg plugin loses track of the instance once it is handed off - it is removed from the Auto-scaling group and is no longer visible to it as part of the application. It is expected that the API handler will place the instance into some special mode of operation (maintenance/cleanup/terminating) and schedule its termination whenever it has finished cleaning up. NOTE the instance becomes a regular EC2 instance and should be terminated with the EC2 API, not the ASG API. For example, if using the boto3 binding for Python:

s = boto3.session.Session(region_name=data["ec2_region"])
ec2 = s.client('ec2')

Limited permissions mode (deprecated)

NOTE: this mode of operation is kept for compatibility with the old ec2-asg plugin, it should be used only in cases when it is necessary to have Skopos run very limited access permissions to AWS. In this mode Skopos does not update ASG/LC configuration, nor it provides image IDs as user_data.

For this mode, the required scaling groups for each component class mentioned in the plan are expected to be pre-created and configured to run instances of that component.

To use ec2-asg with pre-created scaling groups, create a configuration that contains only the lb section (or an empty configuration, if load balancers are not used), e.g.:


      app: "myapp-staging-lb"

Set up for deployment

Create a pair of scaling groups in your EC2 account for each of the components mentioned in the deployment plan. The groups must have names in a fixed format, like this:



  • {prj} is the project name (as given to skopos load --project={prj})
  • {name} is the component (class) name, as used in the model/plan, e.g. 'worker'
  • {side} is one of ('A','B'). Both the _A and the _B groups must be created, even if not used immediately.

Each of the pre-made ASGs named {prj}.{name}_{side} should be set up with a Launch Configuration and all other parameters as needed for starting the component of type {name}. E.g., if there is a group named staging.worker_A, it is expected to be launching instances that behave as “worker”. In addition the following tags MUST be set in the ASG (and configured as "propagated to launched instances"):

  • skopos:image_id - set to the identifier of the software image that will run, in the form {prefix}:{version}. When configuring a new empty group to deploy new components (or start upgraded versions for upgrade), this should match the image name in the model descriptor file. For groups that already have running instances, the {version} part will be displayed as the “current” version in the GUI.
  • skopos:comp_name - must be set exactly to {name} (same as in the ASG’s name itself)

Set up for upgrade

If a previoius deployment was done using the pre-created scaling groups, and you want to run an upgrade plan: edit the group that isn't running anything (e.g., if the _A group was used for the initial deploy, edit _B and vice-versa) and set it up to run the new version. Also, edit its skopos:image_id tag so it matches the image name in the new model file. It should be different from the one currently running.