deployctl is no longer a
gitlab-runner helper, but a new dedicated runner for static deployments.
Read further about the reasons for this development.
During the development of
deployctl was heavily used as the system to auto deploy releases/branches for testing and releases, and during that development it became apparent that we needed a specific runner for a couple of reasons.
deployctl strives for key-/password-/token-less deployment, the runner is located on the web-server, where as
deployctl configures the web-server and put’s the content in the right place.
This called for the use of a
gitlab-ci-multi-runner with a
shell executer, causing a real concern regarding security in terms of abuse/misuse of the
gitlab-ci-token and unauthorized project content access, through stolen token and/or accessing data outside the project-dir.
deployctl runner does not run user scripts directly in bash, only
deployctl commands are sent to the
deployctl command processor and
deployctl commands are simply ignored, depriving the script access to bash execution and thus the system, while allowing custom content locations by means of special variables:
- DEPLOY_PUBLISH_PATH : path(s) to content for
- DEPLOY_REPO_PATH: path to location of the directories rpm and deb
- DEPLOY_RELEASE_PATH: path(s) to the release content
These variables can contain bash substitutions and our variables.
better access-control protection for deployctl:
Access control of the deployments is done through the repository name and when using
gitlab-runner all variables are thrown into a bash shell, making it impossible to distinguish between real and fraud namespace.
With the deployctl-runner, we have access to the full
job, that provides info asides from the CI-variables, making it a full-proof solution.
needs to run as privileged user
Gitlab-runner is started as a servive with that needs root privileges to protect the runner configuration from a shell runner. But it also means that we can not run gitlab on a docker system (openshift/kube) that does not allow privileged containers.
deployctl runner, runs as a non privileged user, with a
sudo (non password exception) to reload and config-test for
nginx, allowing it to run as an un-privileged container.
For that same reason
deployctl switched from
certbot to the
acme.sh client for auto https configurations with
Let's Encrypt, that is till we have a c-lib to take care of the
acme requests as the downside of
acme.sh is quiet a verbose output.
Clean projectdir on start deployment.
Gitstrategy: none in the
.gitlab-ci.yml, and only using
job artifacts as the deployment source, the project-dir does not get cleaned causing deleted files still to be present in a new deployment.
Furthermore, on a shell runner, there is no work around to get it cleaned before the artifacts are downloaded, making it necessary to add some extra annoying scripting:
e.g. when artifacts provides the content in
output/, we need to:
rm -rf newoutput_dir mkdir newoutput_dir cp -r output/ newoutput_dir/
deployctl runner always starts with a fresh and clean project-dir. A choice is made as
deployctl runner only does deployments of output and thus mostly the source for the deployment is provided by
artifacts so it does not need long
git clone for large repositories.
Clean project-dir on finish deployment.
as per previous point, data is left behind by the
gitlab-runner, with no means to delete, if one would need artifacts to be produced.
deployctl runner always cleans the project-dir, not leaving any data behind other then the intended release in the deployment locations.
When we use a server with more then one runner registered, runners check for new jobs sequentially, with a mechanisms to backup of, unfortunately this back-off mechanism has an effect on the healthy
gitlab-instances, causing an delay of deployment and status
Pending, up to 60 sec.
deployctl runner has a back-off system / reduced contact on errors per registered runner!
Secondly since Gitlab 9.xx, long requests have been introduced to reduce the database load/ API polling request of runners. Essentially a very nice solution that blocks a request for a job up to 50 seconds and returns immediately with a job if a job is available. This works kinda like a semi polling/pushing, which has superb improvement on timing of starting a runner very quickly without the overload of very frequent poll’s.
But, it does not work when more then one runner is registered on a server as it leads up to 50 seconds delay per registered runner on the server, as the multi-runner polls sequentially one by one.
deployctl runner enables long-polling in multiple threads per registered runner, with a maximum of 8 registered runners and per runner individual back-off timing.
This results in a very fast start of new builds, where as
deployctl put locks in place for deployment (web-config and repositories for RPM/apt should only executed one job at a time.)
To enable long polling add to
/etc/gitlab/config.rb on your private repository:
gitlab_workhorse['api_ci_long_polling_duration'] = "60s"
Basically a cancel job does not work in Gitlab once the job has started or on a docker build after the authorization of the docker-image has been done.
Furthermore there is no way for a runner to known when a job has been cancelled till the moment the runner announces that the
succeeded. By using the
gitlab-runner, there is nothing in place to take action, like remove the new deployment and putting the previous deployment back in place. (One can check this by refreshing the build trace after canceling the job or pipeline.)
deployctl runner could do is when receiving a cancel notification on a successful or failed job, to put the previous deployment, if there was on back in place.
Remark not implemented yet at this moment.(0.3.4)
No color output
prior to 0.3.0, when using
gitlab-runner all commands are executed with a script with a stdout redirection, resulting in a plain white output, making error’s and warnings difficult to spot.
With 0.3.0 onwards, by integrating the runner into
deployctl, we can display colors:
very anoying trace output:
When artifacts are downloaded, they are unzipped, preserving the file attributes. Meaning if the content was build in a docker container, artifacts are zipped with the file-attributes (owner), pulling the artifacts on a shell runner, where a non privileged user tries to get ownership of those files results in:
extract from build heloworld_c with gitlab-runner: