Pipeline examples¶
Note
In this tutorial, please replace the following values:
ZONE_NAMEwith the name of the administrative zone (it starts withocb-).TEAM_NAMEwith the name of the team used for the deployment.
Build and push a Docker image¶
We want to build a Docker image with a Dockerfile in a git repository. Once the image is built, we push it to the Quay docker registry.
Step 1: define the resources¶
For this pipeline, we need to define two resources:
git-master: this is a reference to the git repository where our Dockerfile is located.latest-image: the typeregistry-imagerepresents a Docker repository. If you want to use Quay, we recommend you to create a robot account and use it to push your image.
resources:
- name: git-master
type: git
icon: git
source:
uri: https://git.ZONE_NAME.caascad.com/myorg/myrepo.git
username: ((git-username))
password: ((git-password))
branch: master
- name: latest-image
type: registry-image
icon: docker
source:
repository: docker-registry.ZONE_NAME.caascad.com/myorg/myimage
username: ((quay-robot-username))
password: ((quay-robot-password))
tag: latest
All the values between parenthesis, (( and )), are variables. The values of these variables are passed with the command fly set-pipeline.
Step 2: fetch the git repository¶
Fetch the git repository. The trigger option will trigger this job every time there is new commit on master.
jobs:
- name: build
plan:
- get: git-master
trigger: true
Step 3: generate credentials file¶
This file will be used by the task that actually build the image. This step is only useful if the base image of your Dockerfile is located on a private Docker repository.
- task: generate-docker-creds
config:
platform: linux
image_resource:
type: registry-image
source:
repository: docker-registry.ZONE_NAME.caascad.com/myorg/alpine
username: ((quay-robot-username))
password: ((quay-robot-password))
outputs:
- name: docker-creds
run:
path: ash
args:
- -c
- |
AUTH="$(echo -n "((quay-robot-username)):((quay-robot-password))" | base64 | tr -d '\n')"
echo '{"auths": {"docker-registry.ZONE_NAME.caascad.com": {"auth": "'$AUTH'", "email": ""}}}' > docker-creds/config.json
The resulting file is located in docker-creds/config.json. It contains a JSON structure with the Quay robot username and password encoded into base64. This structure is the same that the official Docker client uses.
Step 4: build the image¶
For building the Docker image, we use a task using the image vito/oci-build-task. This will use the Dockerfile available in the git repository and export the image in image/image.tar.
- task: build-docker-image
privileged: true
config:
platform: linux
image_resource:
type: registry-image
source:
repository: vito/oci-build-task
caches:
- path: cache
inputs:
- name: git-master
- name: docker-creds
outputs:
- name: image
params:
CONTEXT: git-master
DOCKER_CONFIG: docker-creds
run:
path: build
This task uses two inputs:
git-master: it contains the content of the git repository.docker-creds: it contains the JSON file with the Docker credentials that we have previously generated.
The params key contains environment variables to configure the tasks. You can refer to the documentation to know the values accepted.
Step 5: push the image¶
This step take the image file and push it to the repository defined in the resource with the name latest-image.
- put: latest-image
params:
image: image/image.tar
Step 6: apply the pipeline¶
You can copy the full pipeline and put it in a file:
Full pipeline
resources:
- name: latest-image
type: registry-image
icon: docker
source:
repository: docker-registry.ZONE_NAME.caascad.com/myorg/myimage
username: ((quay-robot-username))
password: ((quay-robot-password))
tag: latest
- name: git-master
type: git
icon: git
source:
uri: https://git.ZONE_NAME.caascad.com/myorg/myrepo.git
username: ((git-username))
password: ((git-password))
branch: master
jobs:
- name: build
plan:
- get: git-master
trigger: true
# Create a file with the credentials used to authenticate to the docker registry
# This step is mandatory only if you need to fetch an image from a private registry
- task: generate-docker-creds
config:
platform: linux
image_resource:
type: registry-image
source:
repository: docker-registry.ZONE_NAME.caascad.com/myorg/alpine
username: ((quay-robot-username))
password: ((quay-robot-password))
outputs:
- name: docker-creds
run:
path: ash
args:
- -c
- |
AUTH="$(echo -n "((quay-robot-username)):((quay-robot-password))" | base64 | tr -d '\n')"
echo '{"auths": {"docker-registry.ZONE_NAME.caascad.com": {"auth": "'$AUTH'", "email": ""}}}' > docker-creds/config.json
# Build the docker image using the credentials we previously forged
- task: build-docker-image
privileged: true
config:
platform: linux
image_resource:
type: registry-image
source:
repository: vito/oci-build-task
caches:
- path: cache
inputs:
- name: git-master
- name: docker-creds
outputs:
- name: image
params:
CONTEXT: git-master
DOCKER_CONFIG: docker-creds
run:
path: build
- put: latest-image
params:
image: image/image.tar
You can apply the pipeline using fly:
fly -t caascad set-pipeline -p pipeline-name -c pipeline.yaml \
-v quay-robot-username=ROBOTUSERNAME \
-v quay-robot-password=ROBOTPASSWORD \
-v git-username=USERNAME \
-v git-password=PASSWORD
Deploy to K8s using Rancher¶
This pipeline fetch a Helm chart from a git repository and deploy it on the Kubernetes cluster using Rancher.
Step 1: define the resource¶
For this pipeline, we need to define one resource:
git-master: this is a reference to the git repository where our Dockerfile is located.
resources:
- name: git-master
type: git
icon: git
source:
uri: https://git.ZONE_NAME.caascad.com/myorg/myrepo.git
username: ((git-username))
password: ((git-password))
branch: master
Step 2: fetch git repository¶
Fetch the git repository. The trigger option will trigger this job every time there is new commit on master.
jobs:
- name: deploy
plan:
- get: git-master
trigger: true
Step 3: apply the Helm chart¶
Use a task to deploy the Helm chart using an image containing Helm and a shell called dtzar/helm-kubectl.
- task: deploy manifest
config:
platform: linux
inputs:
- name: git-master
image_resource:
type: registry-image
source:
repository: dtzar/helm-kubectl
tag: 3.2.1
run:
path: bash
dir: git-master
args:
- -ec
- |
echo "((caascad-kubernetes-TEAM_NAME.kubeconfig))" > /tmp/kubeconfig
export KUBECONFIG="/tmp/kubeconfig"
helm upgrade --install --wait --create-namespace -n demo release-name ./helm/demo
The string ((caascad-kubernetes-TEAM_NAME.kubeconfig)) will be replaced automatically by Concourse. This variable has already been provided for you as a secret. It will allow you to retrieve the kubeconfig used to configure the command kubectl and helm. It contains the credentials to connect to Kubernetes using Rancher.
Step 4: apply the pipeline¶
You can copy the full pipeline and put it in a file:
Full pipeline
resources:
- name: git-master
type: git
icon: git
source:
uri: https://git.ZONE_NAME.caascad.com/myorg/myrepo.git
username: ((git-username))
password: ((git-password))
branch: master
jobs:
- name: deploy
plan:
- get: git-master
trigger: true
- task: deploy manifest
config:
platform: linux
inputs:
- name: git-master
image_resource:
type: registry-image
source:
repository: dtzar/helm-kubectl
tag: 3.2.1
run:
path: bash
dir: git-master
args:
- -ec
# The secret "caascad-kubernetes-TEAM_NAME.kubeconfig" has already been created for you
- |
echo "((caascad-kubernetes-TEAM_NAME.kubeconfig))" > /tmp/kubeconfig
export KUBECONFIG="/tmp/kubeconfig"
helm upgrade --install --wait --create-namespace -n demo release-name ./helm/demo
You can apply the pipeline using fly:
fly -t caascad set-pipeline -p pipeline-rancher -c pipeline-rancher.yaml \
-v git-username=USERNAME \
-v git-password=PASSWORD
Create Kubernetes secret to pull images from a private registry¶
This pipeline shows how to get docker registry credentials from Vault and deploy it as Kubernetes secret on specific namespace in order to pull an image from a private registry.
Docker registry credentials (in Docker JSON authentication format) must first be added in Vault.
Note
Vault secrets that will be used in Concourse pipelines must be placed in Vault KV store in specific path according to the desired use. Example :
kv/concourse/TEAM_NAME/
The secret will be accessible and available in all Concourse pipelines of the TEAM_NAME
You can copy the following pipeline code and put it in a file (ex: pipeline-regcred.yaml):
jobs:
- name: add-regcred-to-k8s
plan:
- task: push-regcred
config:
platform: linux
image_resource:
type: registry-image
source:
repository: dtzar/helm-kubectl
run:
path: /bin/sh
args:
- -cx
- |
# delete secret first and ignore if not found
kubectl --server="((caascad-kubernetes-TEAM_NAME.url))" --token="((caascad-kubernetes-TEAM_NAME.token))" -n ((namespace)) delete secret regcred --ignore-not-found=true
# create secret
kubectl --server="((caascad-kubernetes-TEAM_NAME.url))" --token="((caascad-kubernetes-TEAM_NAME.token))" -n ((namespace)) create secret generic regcred --from-literal=.dockerconfigjson='((YOUR_SECRET.KEY))' --type=kubernetes.io/dockerconfigjson
Replace TEAM_NAME with the name of Concourse team where you want to apply this pipeline, and replace YOUR_SECRET.KEY with your Vault secret.key .
Now you can apply the pipeline using the command line tool fly and specify the namespace where you want to deploy the secret:
fly -t caascad set-pipeline -p pipeline-regcred -c pipeline-regcred.yaml \
-v namespace=default