NAV
Switch version:

Elastic Agents

Extension Information

Availability GoCD version 19.3.0 onwards
Extension Name elastic-agent
Extension API Version 5.0

Background

In the current GoCD setup, all agents are fully provisioned and always available waiting for a job. When there are no jobs to be executed in that environment or for that specific resource that the agents have, the agents stay in an idle state waiting for jobs to be assigned to it, which is a waste of infrastructure resources. It could also lead to high costs if agents are running on cloud providers.

Elastic agents are on-demand agents which are created and provisioned by an elastic-agent plugin when there are jobs to be executed, and terminated when the agents are running idle. These agents can be in a data center or in the cloud or both, and may be physical or virtual.

Developers can start building their own elastic-agent plugins by forking the skeleton plugin and looking at a sample docker plugin as an example reference implementation.

How will it help you?

A feature like this can allow for more efficient use of agent machines, can allow flexible scaling and in many cases, can reduce the cost of running agents. Imagine an automated performance test which runs occasionally and needs a lot of machines. These machines can be started at the beginning of the performance test, possibly using some cloud service, and then brought down when not needed. This feature should enable a more flexible and dynamic build grid.

Current process of handling agents

Agents are currently started manually or automatically by scripts written by users. They are registered to the GoCD server manually, on the agents page or automatically, using auto-registration.

Idle agents, after registration with the server, continuously poll the server, asking for jobs to run. When a stage is triggered, the jobs that need to be started are identified by the server. Suitable jobs are assigned to agents (depending on resources and environments). The agents pick up the jobs and run them. Once finished, they start polling the server for the next job to run.

  1. When a job is scheduled, it is considered for assignment to an agent, and it waits in the queue for an idle agent to ping the server.
  2. When an idle agent pings the server for work, assuming that resources and environments match, the agent is assigned the scheduled and waiting job.
  3. Once the agent finishes running that job, it goes back into the idle agent pool, waiting for another, suitable job to be scheduled and assigned to it.

Watch the video linked below to help make this a little more clear.

Job assignment with normal agents

Process of handling elastic agents

Here is an example cluster profile, elastic profile configuration, and a Job configured to use the profile for the docker plugin

<server>
...
</server>
<elastic>
  <clusterProfiles>
    <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
      <property>
        <key>DockerURI</key>
        <value>https://docker-uri/</value>
      </property>
    </clusterProfile>
  </clusterProfiles>
  <agentProfiles>
    <agentProfile  id="docker.unit-test" clusterProfileId="docker-local">
      <property>
        <!--
          The plugin currently only supports the `Image` property,
          which allows you to select the docker image that the build should run with
        -->
        <key>Image</key>
        <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
      </property>
    </agentProfile>
  </agentProfiles>
</elastic>
...
<job name="defaultJob" elasticProfileId="docker.unit-test">
  <tasks>
    <exec command="mvn" args="clean test" />
  </tasks>
  <!-- Specify the id of the plugin that should manage elastic agents for this job -->
</job>

Cluster profile represents the cluster where a user wants to start an elastic agent. An elastic agent profile represents the configuration to be used to create an elastic-agent instance. Depending on the type of the plugin and the capabilities it provides, a profile configuration could contain the AMI ID, or docker image name, size of the machine, volumes to be mounted etc.

The plugin decides whether to assign an agent to a job, or to start another agent, when consulted by Go.

  1. When a job is scheduled, if its job configuration contains a elasticProfileId, the job is not considered for normal assignment as described in the previous section.
  2. The plugin corresponding for that job elasticProfileId is contacted (to create an agent), along with the corresponding profile configuration for which this assignment needs to be made.
  3. At this point the plugin may, at its discretion, create an agent or not.
    1. If it sees that there are no idle agents that satisfy the profile, it may choose to spin a new agent.
    2. If it sees an existing idle agent that satisfies the profile, the plugin may choose to not spin a new agent.
    3. If there is no more capacity available to create new agents, the plugin may choose to not spin a new agent. (This is same as point above).
  4. In case the job is not assigned after a specified interval, the server waits some time, and then comes back to the plugin, asking it to choose, again.
  5. When an elastic-agent pings the server for work, and if there is a job that requires an elastic-agent, then the corresponding plugin for that job is contacted (to check if the server should assign work to the agent), along with the profile configuration for which this assignment needs to be made.
Lax matching of agents to jobs
Job Configuration Agent Configuration Should assign work?
<profile id="aws.small" pluginId="aws">
  <property>
    <key>instance-type</key>
    <value>m1.small</value>
  </property>
</profile>
<job name="unit-tests"
     elasticProfileId="aws.small">
</job>
<profile id="aws.large" pluginId="aws">
  <property>
    <key>instance-type</key>
    <value>m1.xlarge</value>
  </property>
</profile>

<!-- Agent for profile "aws.large" -->
<agent elasticPluginId="aws"/>

Yes

Since the job requires a smaller agent, and a larger agent is available.
<profile id="aws.small" pluginId="aws">
  <property>
    <key>instance-type</key>
    <value>m1.small</value>
  </property>
</profile>
...
<job name="unit-tests"
     elasticProfileId="aws.small">
</job>
<profile id="aws.small-us-east" pluginId="aws">
  <property>
    <key>aws-region</key>
    <value>us-east-1</value>
  </property>
  <property>
    <key>instance-type</key>
    <value>m1.small</value>
  </property>
</profile>
<!-- Agent for profile "aws.small-us-east" -->
<agent elasticPluginId="aws"/>

Yes

Since the job does not care which region it runs in.

See the video linked below to help make this a little more clear.

Job assignment with elastic agents

Requests from the GoCD server

In order to implement an elastic agent extension point the following messages must be implemented by the plugin.

These are general purpose messages that a plugin must implement to allow users to configure the plugin through the browser.

These are messages that a plugin must implement in order to allow users to configure elastic profiles through the browser.

If a plugin supports supports status reports, apart from the above, the plugin must implement the following messages. The plugin should use the Capabilities to expose this feature.

Get Plugin Icon

This call is expected to return the icon for the plugin, so as to make it easy for users to identify the plugin.

Request name

cd.go.elastic-agent.get-icon

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example plugin response body:

{
  "content_type": "image/svg+xml",
  "data": "PHN2ZyB2ZXJzaW9u..."
}

The plugin is expected to return an image object.

Get Plugin Capabilities

This message is a request to the plugin to provide plugin capabilities. Based on these capabilities GoCD would enable or disable the plugin features for a user.

Request name

cd.go.elastic-agent.get-capabilities

Request body

Server sends request with Empty request body.

Response Body

An example response body:

{
  "supports_plugin_status_report": true,
  "supports_agent_status_report": true,
  "supports_cluster_status_report": true
}

The response body will contain the following JSON elements:

Key Type Description
supports_plugin_status_report boolean Whether plugin supports Plugin status report depends on this boolean value
supports_agent_status_report boolean Whether plugin supports Agent status report depends on this boolean value
supports_cluster_status_report boolean Whether plugin supports Cluster status report depends on this boolean value

The plugin is expected to return status 200 if it can understand the request.

Migrate config

This message is a request to the plugin perform migration of the existing config on load of the plugin. This allows a plugin to perform the migration on the existing config in order to support the newer version of the plugin.

Request name

cd.go.elastic-agent.migrate-config

Request body

The plugin will receive the following JSON body —

{
  "cluster_profiles": [
    {
      "id": "cluster_profile_id",
      "plugin_id": "plugin_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    }
  ],
  "elastic_agent_profiles": [
    {
      "id": "profile_id",
      "plugin_id": "plugin_id",
      "cluster_profile_id": "cluster_profile_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    }
  ],
  "plugin_settings": {
    "url": "https://some-url",
    "password": "my-password"
  }
}

Key Type Description
cluster_profiles Array List of cluster profiles configured for the plugin.
elastic_agent_profiles Array List of elastic agent profiles configured for the plugin.
plugin_settings Array Plugin settings of the plugin.

Response Body

An example response body:

{
  "cluster_profiles": [
    {
      "id": "cluster_profile_id",
      "plugin_id": "plugin_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    },
    {
      "id": "migrated_from_plugin_settings",
      "plugin_id": "plugin_id",
      "properties": {
        "url": "https://some-url",
        "password": "my-password"
      }
    }
  ],
  "elastic_agent_profiles": [
    {
      "id": "profile_id",
      "plugin_id": "plugin_id",
      "cluster_profile_id": "cluster_profile_id",
      "properties": {
        "some_key": "some_value",
        "some_key2": "some_value2"
      }
    }
  ]
}

The response body will contain the following JSON elements:

Key Type Description
cluster_profiles Array List of cluster profiles configured for the plugin.
elastic_agent_profiles Array List of elastic agent profiles configured for the plugin.

The plugin is expected to return status 200 if it can understand the request otherwise the plugin will be marked as invalid.

Create Agent

This message is a request to the plugin to create an agent for a job that has been scheduled.

Request name

cd.go.elastic-agent.create-agent

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6" />
  <elastic>
    <clusterProfiles>
      <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
        <property>
          <key>DockerURI</key>
          <value>https://docker-uri/</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
    <agentProfiles>
      <agentProfile  id="docker.unit-test" clusterProfileId="docker-local">
        <property>
          <!--
            The plugin currently only supports the `Image` property,
            which allows you to select the docker image that the build should run with
          -->
          <key>Image</key>
          <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
        </property>
      </agentProfile>
    </agentProfiles>
  </elastic>
  <pipelines group="first">
    <pipeline name="build">
      ...
      <job name="test-job" elasticProfileId="docker.unit-test">
        ...
      </job>
      ...
    </pipeline>
  </pipelines>
  ...
</cruise>

The plugin will receive the following JSON body —

{
  "auto_register_key": "1e0e05fc-eb45-11e5-bc83-93882adfccf6",
  "elastic_agent_profile_properties": {
    "Image": "gocd/gocd-agent-alpine-3.5:v18.1.0",
    "MaxMemory": "https://docker-uri/"
  },
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  },
  "environment": "prod",
  "job_identifier": {
    "job_id": 100,
    "job_name": "test-job",
    "pipeline_counter": 1,
    "pipeline_label": "build",
    "pipeline_name": "build",
    "stage_counter": "1",
    "stage_name": "test-stage"
  }
}

The request body will contain the following JSON elements:

Key Type Description
auto_register_key String The key that an agent should use, if it should be auto-registered with the server. The plugin is expected to use the key to create an appropriate autoregister.properties file on the agent instance, before it starts the agent process. See the auto-register documentation for more information.
environment String The environment that this job belongs to. Agents are expected to auto-register using this environment so that they can be assigned to the correct job. See the environments section to know more about environments.
elastic_agent_profile_properties Object Elastic agent profile associated with the job. It represents the elastic agent configuration for the job in form of key value pairs.
cluster_profile_properties Object The field represents the cluster profile associated with elastic profile.
job_identifier Object Job identifier of the job for which this call is being made.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

Can be left blank, the server does not parse any response body returned.

Should Assign Work

When there are multiple agents available to run a job, the server will ask the plugin if jobs should be assigned to a particular agent. The request will contain information about the agent, the job configuration and the environment that the agent belongs to. This allows plugin to decide if proposed agent is suitable to schedule a job on it. For example, plugin can check if flavor or region of VM is suitable.

Request name

cd.go.elastic-agent.should-assign-work

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6"/>
  <elastic>
    <clusterProfiles>
      <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
        <property>
          <key>DockerURI</key>
          <value>https://docker-uri/</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
    <agentProfiles>
      <agentProfile id="ec2.small-us-east" clusterProfileId="docker-local">
        <property>
          <key>ami-id</key>
          <value>ami-6ac7408f</value>
        </property>
        <property>
          <key>region</key>
          <value>us-east-1</value>
        </property>
      </agentProfile>
    </agentProfiles>
  </elastic>
  <pipelines>
    <pipeline name='build'>
      <job name="run-upgrade" runOnAllAgents="true" timeout='30' elasticProfileId="ec2.small-us-east">
        <tasks>
          <ant target="upgrade"/>
        </tasks>
      </job>
    </pipeline>
  </pipelines>
  <environments>
    <environment name="staging">
      <pipelines>
        <pipeline name="build"/>
      </pipelines>
    </environment>
  </environments>
  <agents>
    <agent elasticAgentId='i-283432d4' elasticPluginId='com.example.go.testplugin'/>
  </agents>
</cruise>

The plugin will receive the following JSON body —

{
  "agent": {
    "agent_id": "i-283432d4",
    "agent_state": "Idle",
    "build_state": "Idle",
    "config_state": "Enabled"
  },
  "environment": "staging",
  "job_identifier": {
    "job_id": 100,
    "job_name": "run-upgrade",
    "pipeline_counter": 1,
    "pipeline_label": "build",
    "pipeline_name": "build",
    "stage_counter": "1",
    "stage_name": "test-stage"
  },
  "elastic_agent_profile_properties": {
      "Image": "gocd/gocd-agent-alpine-3.5:v18.1.0",
      "MaxMemory": "https://docker-uri/"
  },
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  }
}

The request body will contain the following JSON elements:

Key Type Description
agent Object An object describing the elastic agent.
environment String The environment that this job belongs to. Agents are expected to auto-register using this environment so that they can be assigned to the correct job. See the environments section to know more about environments.
elastic_agent_profile_properties Object Elastic agent profile associated with the job. It represents the elastic agent configuration for the job in form of key value pairs.
cluster_profile_properties Object The field represents the cluster profile associated with elastic profile.
job_identifier Object Job identifier of the job for which this call is being made.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The server must return a JSON string response with a boolean — true or false to indicate whether the agent should be assigned work.

Server Ping

Each elastic agent plugin will receive a periodic signal at regular intervals for it to perform any cleanup operations. Plugins may use this message to disable and/or terminate agents at their discretion.

Request name

cd.go.elastic-agent.server-ping

Request body

Given the following config XML snippet —

<elastic>
  <clusterProfiles>
    <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
      <property>
        <key>DockerURI</key>
        <value>https://docker-uri/</value>
      </property>
    </clusterProfile>
    <clusterProfile id="ecs" pluginId="cd.go.contrib.elastic-agent.ecs">
      <property>
        <key>AWS_ACCESS_KEY</key>
        <value>AMSDKFSDOFSFSI</value>
      </property>
      <property>
        <key>AWS_SECRET_KEY</key>
        <value>yshfksdfasd,fmsldgjdflgjgdflgjdlfgjdfl</value>
      </property>
      <property>
        <key>CLUSTER_NAME</key>
        <value>Dev</value>
      </property>
    </clusterProfile>
  </clusterProfiles>
</elastic>

The plugin will receive the following JSON body —

{
  "all_cluster_profile_properties": [
    {
      "DockerURI": "https://docker-uri/"
    },
    {
      "AWS_ACCESS_KEY": "AMSDKFSDOFSFSI",
      "AWS_SECRET_KEY": "yshfksdfasd,fmsldgjdflgjgdflgjdlfgjdfl",
      "CLUSTER_NAME"  : "Dev"
    }
  ]
}
Key Type Description
all_cluster_profile_properties Array The field represents the list of cluster profiles for the plugin.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

Can be left blank, the server does not parse any response body returned.

Get Cluster Profile View

This is a message that the plugin should implement, to allow users to configure cluster profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-cluster-profile-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Get Cluster Profile Metadata

This is a message that the plugin should implement, to allow users to configure cluster profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-cluster-profile-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body for a docker plugin

[
  {
    "key": "DockerURI",
    "metadata": {
      "required": true,
      "secure": false
    }
  },
  {
    "key": "MaxDockerContainersAllowed",
    "metadata": {
      "required": false,
      "secure": false
    }
  }
]

A JSON cluster profile metadata object.

Validate Cluster Profile

This call is expected to validate the user inputs that form a part of the cluster profile.

Request name

cd.go.elastic-agent.validate-cluster-profile

Request body

The request body will contain a JSON object with the keys and values that form part of the cluster profile.

An example validation request body for the docker elastic agent plugin

{
  "DockerURI": "https://docker-uri",
  "MaxDockerContainersAllowed": "foo"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "MaxMemory",
   "message": "'foo' is not a valid value for `MaxDockerContainersAllowed`."
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the cluster profile is valid, the plugin should return an empty JSON array.

Validate Elastic Agent Profile

This call is expected to validate the user inputs that form a part of the elastic agent profile.

Request name

cd.go.elastic-agent.validate-elastic-agent-profile

Request body

The request body will contain a JSON object with the keys and values that form part of the elastic agent profile.

An example validation request body for the docker elastic agent plugin

{
  "Image": "alpine:latest",
  "MaxMemory": "foo"
}

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

The plugin should respond with JSON array response for each configuration key that has a validation error

[
  {
   "key": "MaxMemory",
   "message": "'foo' is not a valid value for `MaxMemory`."
  }
]

If any of the input keys have a validation error on them, the plugin is expected to return a list of validation error objects. If the profile is valid, the plugin should return an empty JSON array.

Get Elastic Agent Profile View

This is a message that the plugin should implement, to allow users to configure elastic agent profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-elastic-agent-profile-view

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

A JSON settings view object

An example response body:

{
  "template": "<div>some html</div>"
}

Get Elastic Agent Profile Metadata

This is a message that the plugin should implement, to allow users to configure elastic agent profiles from the Elastic Profiles View in GoCD.

Request name

cd.go.elastic-agent.get-elastic-agent-profile-metadata

Request body

The server will not provide a request body.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

An example response body for a docker plugin

[
  {
    "key": "Image",
    "metadata": {
      "required": true,
      "secure": false
    }
  }
]

A JSON elastic agent profile metadata object.

Job Completion

The intent on this message is to notify the plugin on completion of the job. The plugin may choose to terminate the elastic agent or keep it running in case the same agent can be used for another job configuration.

Request name

cd.go.elastic-agent.job-completion

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6" />
  <elastic>
    <clusterProfiles>
      <clusterProfile id="docker-local" pluginId="cd.go.contrib.elastic-agent.docker">
        <property>
          <key>DockerURI</key>
          <value>https://docker-uri/</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
    <agentProfiles>
      <agentProfile  id="docker.unit-test" clusterProfileId="docker-local">
        <property>
          <!--
            The plugin currently only supports the `Image` property,
            which allows you to select the docker image that the build should run with
          -->
          <key>Image</key>
          <value>gocdcontrib/ubuntu-docker-elastic-agent</value>
        </property>
      </agentProfile>
    </agentProfiles>
  </elastic>
  <pipelines group="first">
    <pipeline name="build">
      ...
      <job name="test-job" elasticProfileId="docker.unit-test">
        ...
      </job>
      ...
    </pipeline>
  </pipelines>
  ...
  <agents>
    <agent hostname="1697e6b164f7" ipaddress="172.17.0.6" uuid="a45e3ca1-4419-4bd5-b18b-882f75ffd4c2" elasticAgentId="GoCD18efbeef995e40f688cd92dc22a4d332" 
      elasticPluginId="cd.example.elastic-agent" />
  </agents>
</cruise>

The plugin will receive the following JSON body —

{
  "elastic_agent_id": "GoCD18efbeef995e40f688cd92dc22a4d332",
  "elastic_agent_profile_properties": {
    "Image": "gocd/gocd-agent-alpine-3.5:v18.1.0",
    "MaxMemory": "https://docker-uri/"
  },
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  },
  "job_identifier": {
    "job_id": 100,
    "job_name": "test-job",
    "pipeline_counter": 1,
    "pipeline_label": "build",
    "pipeline_name": "build",
    "stage_counter": "1",
    "stage_name": "test-stage"
  }
}

The request body will contain the following JSON elements:

Key Type Description
elastic_agent_id String The Elastic agent ID of the agent on which the job has completed.
elastic_agent_profile_properties Object Elastic agent profile associated with the job. It represents the elastic agent configuration for the job in form of key value pairs.
cluster_profile_properties Object The field represents the cluster profile associated with elastic profile.
job_identifier Object Job identifier of the job for which this call is being made.

Response code

The plugin is expected to return status 200 if it can understand the request.

Response Body

Can be left blank, the server does not parse any response body returned.

Get agent status report

If plugin supports status report, this message must be implemented to report the status of a particular elastic agent brought up by the plugin to run a job. The purpose of this call is to provide specific information about the current state of the elastic agent.

Request name

cd.go.elastic-agent.agent-status-report

Request body

An example request body for a docker plugin

{
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  },
  "elastic_agent_id": "46d7aa499c9f44958e118ffb7f975c52",
  "job_identifier": {
    "pipeline_name": "test-pipeline",
    "pipeline_label": "Test Pipeline",
    "pipeline_counter": 1,
    "stage_name": "test-stage",
    "stage_counter": "1",
    "job_name": "test-job",
    "job_id": 100
  }
}

The request body will contain the following JSON elements:

Key Type Description
job_identifier Object The job identifier for which the agent is created.
elastic_agent_id String This key contains the elastic agent id, this is available only when agent is registered with GoCD server.
cluster_profile_properties Object The key represents the cluster profile for the given elastic_agent

Response code

The plugin is expected to return status 200 JSON object of view and in following format.

Response Body

An example response body:

{
  "view": "agent-status-report-html-view"
}

The request body will contain the following JSON elements:

Key Type Description
view String Agent status report view html. GoCD server will render this view as a agent status report.

Get cluster status report

If plugin supports cluster status report, this message must be implemented to provide the overall status of the cluster.

Request name

cd.go.elastic-agent.cluster-status-report

Request body

An example request body for a docker plugin

{
  "cluster_profile_properties": {
    "Image": "DockerURI",
    "MaxMemory": "500Mb"
  }
}
Key Type Description
cluster_profile_properties Object This key cluster profile for which the status report request is made.

Response code

The plugin is expected to return status 200 JSON object of view and in following format.

Response Body

An example response body:

{
  "view": "plugin-status-report-html-view"
}

The request body will contain the following JSON elements:

Key Type Description
view String Plugin should return html view containing information about elastic agents.

Get plugin status report

If plugin supports the plugin status report, this message must be implemented to provide the overall status of the environment.

Request name

cd.go.elastic-agent.plugin-status-report

Request body

Given the following config XML snippet —

<cruise>
  <server agentAutoRegisterKey="1e0e05fc-eb45-11e5-bc83-93882adfccf6" />
  <elastic>
    <clusterProfiles>
      <clusterProfile id="Dev" pluginId="cd.go.contrib.elastic-agent.ecs">
        <property>
          <key>AWS_ACCESS_KEY</key>
          <value>developer-access-key</value>
        </property>
        <property>
          <key>AWS_SECRET_KEY</key>
          <value>developer-secret-key</value>
        </property>
        <property>
          <key>CLUSTER_NAME</key>
          <value>Dev</value>
        </property>
      </clusterProfile>
      <clusterProfile id="Prod" pluginId="cd.go.contrib.elastic-agent.ecs">
        <property>
          <key>AWS_ACCESS_KEY</key>
          <value>prod-access-key</value>
        </property>
        <property>
          <key>AWS_SECRET_KEY</key>
          <value>prod-secret-key</value>
        </property>
        <property>
          <key>CLUSTER_NAME</key>
          <value>Production</value>
        </property>
      </clusterProfile>
    </clusterProfiles>
  </elastic>
</cruise>

The plugin will receive the following JSON body —

{
  "all_cluster_profiles_properties": [
    {
      "AWS_ACCESS_KEY": "developer-access-key",
      "AWS_SECRET_KEY": "developer-secret-key",
      "CLUSTER_NAME": "Dev"
    },
    {
      "AWS_ACCESS_KEY": "prod-access-key",
      "AWS_SECRET_KEY": "prod-secret-key",
      "CLUSTER_NAME": "Production"
    }
  ]
}
Key Type Description
all_cluster_profiles_properties Array The field represents the list of cluster profiles for the plugin.

Response code

The plugin is expected to return status 200 JSON object of view and in following format.

Response Body

An example response body:

{
  "view": "plugin-status-report-html-view"
}

The request body will contain the following JSON elements:

Key Type Description
view String Plugin should return html view containing information about elastic agents.

Requests to the GoCD server

The plugin may make the following requests to the server using GoApplicationAccessor#submit(GoApiRequest)

List Agents

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private List listAgents() {
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.elastic-agents.list-agents",
      "1.0",
      pluginIdentifier()
    );

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }

    // parse the response, using a json parser of your choice
    List agents = new Gson().fromJson(response.responseBody(), ArrayList.class);
    return agents;
  }
}

This messages allows a plugin to query the server for a list of elastic agents that belong to a particular plugin.

Request name

go.processor.elastic-agents.list-agents

Request version

The request version must be set to 1.0.

Request body

Can be left blank, the server does not parse the request body.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The server will send a list of agents.

Disable Agents

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private disableAgents(List agents) {
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.elastic-agents.disable-agents",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    request.setRequestBody(new Gson().toJson(agents));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }
  }
}

Before a plugin terminates an agent instance, it must first disable it in order to prevent jobs from being assigned to it. Agents in any state can be disabled, this will prevent jobs from being assigned to it. Any jobs the agent is running will eventually complete.

Request name

go.processor.elastic-agents.disable-agents

Request version

The request version must be set to 1.0.

Request body

The body must contain a list of agents.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The server will not send a response body.

Delete Agents

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private deleteAgents(List agents) {
    // first terminate the instance from AWS, or a cloud provider of your choice
    aws.terminateInstances(agents);
    // ensure that they are really terminated,
    // to prevent stray agents from re-registering
    aws.waitForTermination(agents);

    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.elastic-agents.disable-agents",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    request.setRequestBody(new Gson().toJson(agents));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }
  }
}

Before the delete-agent message is sent to the server, the plugin MUST ensure that the agent is terminated.

Request name

go.processor.elastic-agents.delete-agents

Request version

The request version must be set to 1.0.

Request body

The body must contain a list of agents that should be removed from the config XML.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The server will not send a response body.

Get Plugin Settings

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerElasticAgentPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerElasticAgentPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"))
  }

  private PluginSettings getSettings() {
    Gson gson = new Gson();
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.plugin-settings.get",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    Map<String, String> map = new HashMap<>();
    map.put("plugin-id", "com.example.rocket.launcher");
    request.setRequestBody(gson.toJson(map));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }

    // parse the response, using a json parser of your choice
    return gson.fromJson(response.responseBody(), PluginSettings.class);
  }
}

This messages allows a plugin to query the server to get the user configured settings for this plugin.

Request name

go.processor.plugin-settings.get

Request version

The request version must be set to 5.0.

Request body

An example request body:

{
  "plugin-id": "sample-plugin-id"
}

Must be a JSON object with a key plugin-id with the value being the ID of your plugin.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

An example response body:

{
  "server_url": "https://build.go.cd",
  "username": "view",
  "password": "password"
}

The server will send a map of settings.

Get Server Info

This messages allows a plugin to query the server to get some metadata about the server.

Available since v17.9.0.

Request name

go.processor.server-info.get

Request version

The request version must be set to 1.0.

Request body

The plugin should not provide a request body.

Response code

The server is expected to return status 200 if it could process the request.

Response Body

The plugin will provide a server info object.

An example response body:

{
  "server_id": "df0cb9be-2696-4689-8d46-1ef3c4e4447c",
  "site_url": "http://example.com:8153/go",
  "secure_site_url": "https://example.com:8154/go"
}

Add Server Health Messages

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerElasticAgentPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerElasticAgentPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"));
  }

  private void addErrorsAndWarnings() {
    Gson gson = new Gson();
    // create a request
    DefaultGoApiRequest request = new DefaultGoApiRequest(
      "go.processor.server-health.add-messages",
      "1.0",
      pluginIdentifier()
    );

    // set the request body
    List<Map<String, String>> messages = new ArrayList<>();

    Map<String, String> message1 = new HashMap<>();
    message1.put("type", "warning");
    message1.put("message", "A warning message from the plugin.");

    Map<String, String> message2 = new HashMap<>();
    message2.put("type", "error");
    message2.put("message", "An error message from the plugin.");

    messages.add(message1);
    messages.add(message2);

    request.setRequestBody(gson.toJson(messages));

    // submit the request
    GoApiResponse response = accessor.submit(request);

    // check status
    if (response.responseCode() != 200) {
      LOG.error("The server sent an unexpected status code " + response.responseCode() + " with the response body " + response.responseBody());
    }
  }
}

This message allows a plugin to add error and warning messages to be shown in GoCD. Any previous messages sent by the plugin will be cleared and replaced with the newly specified messages (or cleared if the body is an empty list).

Available since v18.3.0.

Request name

go.processor.server-health.add-messages

Request version

The request version must be set to 1.0.

Request body

An example request body:

[
  {
    "type": "warning",
    "message": "A warning message from the plugin."
  },
  {
    "type": "error",
    "message": "An error message from the plugin."
  }
]

Must be a JSON array made up of JSON objects as described below:

Key Type Description
type String Should be either warning or error, corresponding to the type of message to be shown.
message String A message to be shown in the “Errors and Warnings” box.

Response code

The server is expected to return status 200 if it could process the request. It is expected to return status 500 if it failed to process the request.

Response Body

An example response body for a failure:

{
  "message": "An error occurred ..."
}

The server will respond with a single JSON object with an error message with the key message, if it is unable to process the request. If successful, the response body will be empty.

Log Messages to Job Console

import com.thoughtworks.go.plugin.api.*;
import com.thoughtworks.go.plugin.api.annotation.Extension;
import com.thoughtworks.go.plugin.api.logging.Logger;
import com.thoughtworks.go.plugin.api.request.*;
import com.thoughtworks.go.plugin.api.response.*;
import com.google.gson.Gson;
import java.util.*;

@Extension
public class DockerElasticAgentPlugin implements GoPlugin {
  private GoApplicationAccessor accessor;
  public static final Logger LOG = Logger.getLoggerFor(DockerElasticAgentPlugin.class);

  public void initializeGoApplicationAccessor(GoApplicationAccessor accessor) {
    this.accessor = accessor;
  }

  public GoPluginIdentifier pluginIdentifier() {
    return new GoPluginIdentifier("elastic-agent", Arrays.asList("5.0"));
  }

  private void appendToConsoleLog(String text) throws ServerRequestFailedException {
    Map<String, String> requestMap = new HashMap<>();
    requestMap.put("pipeline_name", "my_pipeline_1");
    requestMap.put("pipeline_counter", "123");
    requestMap.put("stage_name", "stage1");
    requestMap.put("stage_counter", "1");
    requestMap.put("job_name", "job1");
    requestMap.put("text", text);

    DefaultGoApiRequest request = new DefaultGoApiRequest("go.processor.console-log.append", "2.0", pluginIdentifier());
    request.setRequestBody(new GsonBuilder().create().toJson(requestMap));

    GoApiResponse response = accessor.submit(request);

    if (response.responseCode() != 200) {
      LOG.error("Failed to append console log for " + jobIdentifier.represent() + " with text: " + text);
    }
  }
}

This message allows a plugin to add messages to be shown in the GoCD job console.

Available since v19.9.0.

Request name

go.processor.console-log.append

Request version

The request version must be set to 2.0.

Request body

An example request body:

{
  "pipeline_name": "my_pipeline_1",
  "pipeline_counter": "123",
  "stage_name": "stage1",
  "stage_counter": "1",
  "job_name": "job1",
  "text": "Message to show in console log."
}

Must be a JSON object made up of JSON elements as described below:

Key Type Description
pipeline_name String Name of the pipeline in which the job is.
pipeline_counter String The pipeline counter (run counter of the pipeline).
stage_name String Name of the stage the job is in.
stage_counter String The run counter of the stage.
job_name String The name of the job.
text String The message to be shows in the console log of the job.

Response code

The server is expected to return status 200 if it could process the request. It is expected to return status 500 if it failed to process the request.

Response Body

An example response body for a failure:

{
  "message": "An error occurred ..."
}

The server will respond with a single JSON object with an error message with the key message, if it is unable to process the request. If successful, the response body will be empty.

Request/Response JSON Objects

The Elastic Agent Object

Here’s an example of the elastic agent object:

{
  "agent_id": "i-283432d4",
  "agent_state": "Idle",
  "build_state": "Idle",
  "config_state": "Enabled"
}

Attribute Type Description
agent_id String The elastic agent ID. This is the value of the elasticAgentId attribute of the <agent/> element in the config XML.
agent_state String The state that an agent is in. Can be one of Idle, Building, LostContact, Missing, Unknown.
build_state String The state the build is in. Can be one of Idle, Building, Cancelled, Unknown.
config_state String The state of the agent in the config file. This is the value of the isDisabled attribute of the <agent/> element in the config XML. Can be one of Pending, Enabled, Disabled.

The Settings View Object

Here’s an example of the settings view object:

{
  "template": "<div class=\"form_item_block\">...</div>"
}

Attribute Type Description
template String A string containing an HTML AngularJS based view.

This template is an AngularJS based template.

GoCD uses Angular JS as its template engine for the plugin UI. This allows plugin authors to use a limited set of AngularJS features to specify how the UI of their plugins looks.

Getting started with AngularJS based templates

Given a configuration:

<configuration>
  <property>
    <key>username</key>
    <value>alice</username>
  </property>
</configuration>

This gets converted into the following JSON representation in the browser:

{
  "username": "alice"
}

The AngularJS template is expected to bind to the JSON object shown above:

<div class="form_item_block">
  <label>Username:<span class='asterix'>*</span></label>
  <input ng-model="username" />
</div>

When an Angular template is used in a Go plugin, to define the configuration UI, the configuration key which is stored in the configuration XML is used everywhere and is expected to be consistent. Since Angular works off of JSON, GoCD will make sure that the key in the JSON provided to the Angular template is the same as the key in the configuration XML.

Suppose the key of the configuration property stored in the XML is “username”, with value, “alice”, then Go will make sure that the value is available to the template as “username” when used in an Angular-specific HTML attribute like “ng-model”.

Plugin Angular Architecture

So, the name “foobar” needs to be the same across the configuration XML, the Angular template as well as in any code that the plugin has.

Showing validation errors in the UI

We use some simple string replacement
to substitute GOINPUTNAME with a unique identifier
for your plugin in order to render
any server side errors

<div class="form_item_block">
  <label>Username:<span class='asterix'>*</span></label>
  <input ng-model="username" />
  <span class="form_error" ng-show="GOINPUTNAME[username].$error.server">
    {{ GOINPUTNAME[username].$error.server}}
  </span>
</div>

In case of validation errors returned by go.plugin-settings.validate-configuration, the error messages needs to be populated on the UI, use the snippet here to show the validation errors.

The Plugin Settings Configuration Object

Here’s an example of the plugin settings configuration object:

{
  "server_url": {
    "display-name": "Server URL",
    "display-order": "0"
  },
  "username": {
    "required": false,
    "display-name": "Username",
    "display-order": "1"
  },
  "password": {
    "secure": true,
    "required": false,
    "display-name": "Password",
    "display-order": "2"
  }
}

Attribute Type Description
display-name String The name of the property.
default-value String The default value of the property.
display-order String A string containing a numerical value.
required Boolean If the field is mandatory.
secure Boolean If the data in the field should be stored encrypted.

The Validation Error Object

Here’s an example of the validation error object:

[
  {
    "key": "email_address",
    "message": "Email address is invalid"
  },
  {
    "key": "password",
    "message": "Password must be provided"
  }
]

Attribute Type Description
key String The name of configuration key that has an error.
message String The error message associated with that key.

The Image Object

Here’s an example of the image object:

{
  "content_type": "image/svg+xml",
  "data": "...."
}

Attribute Type Description
content_type String A valid content type for the image. Please make sure the content type is supported by most browsers.
data String A base-64 encoded (single-line non-chunking) byte array of the byte-sequence that composes the image.

The Profile Metadata Object

Here’s an example of the profile metadata object:

{
  "content_type": "image/svg+xml",
  "data": "...."
}

Attribute Type Description
key String The name of the configuration property supported by an elastic profile.
metadata Object The metadata associated with the key used in the elastic profile. Valid keys are required and secure.

The server info object

Here’s an example of the server info object:

{
  "server_id": "df0cb9be-2696-4689-8d46-1ef3c4e4447c",
  "site_url": "http://example.com:8153/go",
  "secure_site_url": "https://example.com:8154/go"
}

Attribute Type Description
server_id String This contains a unique identifier for this server.
site_url String This contains the site url configured for this server.
secure_site_url String This contains the secure site url configured for this server.

Glossary