Concourse¶

Concourse is a pipeline-based continuous thing-doer. Pipelines are designed to be self-contained so as to minimize server-wide configuration. Everything in Concourse runs in a container.

You can use Concourse for building your projects, deploying your applications or running cron jobs like backups.

Note

In this tutorial, please replace the following values:

ZONE_NAME with the name of the administrative zone (it starts with ocb-).
TEAM_NAME with the name of the team used for the deployment.

Concourse access¶

The web interface¶

In your administration environment, Concourse service is located at this address:

https://ci.ZONE_NAME.caascad.com

The web interface is used only for viewing teams and pipelines. You can do some actions in this interface, like running a pipeline, however, most of the administration and configuration are done with the CLI.

The CLI¶

The Concourse CLI command is named fly. You can download it on the front page of the Concourse website.

Use the login command to connect to Concourse. You will have to provide these parameters:

-t <target> set a friendly name for your configuration, you can see this as an alias. You can use any name you want. In our examples, we will be using caascad.
--team-name <team> concourse is deployed with team configured. Your teams are named after the name of your zones.
--concourse-url <url> the URL is the same as the web interface.

Example:

$ fly -t caascad login --team-name TEAM_NAME --concourse-url https://ci.ZONE_NAME.caascad.com
logging in to team 'TEAM_NAME'
navigate to the following URL in your browser:
https://ci.ZONE_NAME.caascad.com/login?fly_port=45743
or enter token manually:

Click on the url in the command output and then login using Keycloak by clicking on "Log In".

Now you can use the fly command.

Check out your user info with this command:

$ fly -t caascad userinfo
username   team/role
yourname   TEAM_NAME/member

Introduction to pipelines¶

Resources¶

Concourse use "resources" to represent external inputs and outputs of jobs. Example of resources: git repository, docker images, a S3 bucket...

Each resources has a type and Concourse knows how to deal with a limited set of basic resource types. You can add third party resources types or even develop your own.

Check out this website for a list of official and third party resource types: Resource Types

Jobs¶

A pipeline is compound of one or several jobs. The documentation define jobs as:

Jobs determine the actions of your pipeline. They determine how resources progress through it, and how the pipeline is visualized.

Jobs are described using a plan. The plan explains which resources are needed and what to do with it. You can have a plan that fetch a git repository (input resource), build the Dockerfile located in this repository and finally, push the image into a registry (output resource).

In a pipeline, all jobs are independent. You can link your jobs by passing resources between them.

For example, a pipeline can have the following three jobs:

Test: fetch the repository, execute the go testing suite
Build: fetch the repository, build the Dockerfile
Deploy: fetch the repository, get the name of the docker image, execute a helm command with the name of the new image

As you can see, the resource "repository" is used in all three jobs. If you want to trigger the jobs sequentially, you have to "pass" the repository resource through the three jobs using the keyword "passed" (documentation).

Pipelines¶

Pipelines defines the resources and jobs that will work together. Pipelines are defined in YAML and are set using the command fly set-pipeline. The pipeline belongs to a TEAM.

Variables can be used to configure pipelines. Your pipeline use the variables by substituting the following pattern ((myvariable)). There is no templating language in Concourse, just variable substitution (like your favorite Kubernetes files).

The variables can be defined using two ways:

The variable is defined during the execution of the set-pipeline command and the substitution is done by the fly command. The value of this variables will be visible by anybody that have a read access to the Concourse API.
The variable is not defined when the set-pipeline is executed, the variable will not be substituted by fly but by Concourse itself just before the pipeline is executed by the worker. There is different back-end for storing these secret and we have chose Vault for Caascad.

Limits¶

Two limits are set up on the Concourse instance:

Limit	Value
Default memory limit on tasks	512mb
Limit active tasks per worker	13

The default memory limit can be override using the keyword container_limits. If your task exceed the memory limit, the process might be kill by the kernel.

The limit on the number of tasks is defined per worker. If this limit is reached, the new tasks will wait until a task is finished.