140 lines
6.8 KiB
Markdown
140 lines
6.8 KiB
Markdown
# Simple CI (Continous Integration)
|
|
Jenkins, Travis, GitHub Actions, GitLab CI. The list goes on.
|
|
There are many CI systems.
|
|
All of them overcomplicate an extremely simple use case:
|
|
"I want to trigger custom scripts (usually just a list of simple shell commands) on a server.
|
|
Either automatically, or manually.
|
|
And then I want to be follow and see any errors that may happen."
|
|
|
|
There's no need to invent the wheel 10 different times.
|
|
Most of the actions required by the CI use case can be handled with most UNIX tools such as `tail`,
|
|
`cat`, `inotify` etc.
|
|
At the core of CI must be a triggering system that can be dynamically configured to run a pipeline script.
|
|
|
|
This program provides that simple triggering system where you simply `touch` a file, and the associated
|
|
pipeline will then be executed.
|
|
You can then follow the log via `tail`.
|
|
|
|
The operation of sci is configured through a pipelines.conf configuration file (see sci(7) for
|
|
configuration language details) and each pipeline will have an associated pipeline trigger file
|
|
that can be By default, pipeline triggers are placed in /tmp/sci but this can be overridden with
|
|
the `-T` flag (WIP feature). For more commandline options see the manpage `sci(1)` or use the `--help` option.
|
|
|
|
## Dependencies
|
|
Just GNU/Linux! `sci` is just using the POSIX api, so it should work with *NIX, but it has only been tested on GNU/Linux
|
|
systems.
|
|
NOTE: Not tested with `muslc` systems yet, `sci` use some libc GNU extensions, but it would be nice to support alpine
|
|
linux some time in the future.
|
|
|
|
## Build
|
|
Build the project using `make`:
|
|
```sh
|
|
make
|
|
```
|
|
|
|
## Install
|
|
Install the project:
|
|
```sh
|
|
make
|
|
sudo make install
|
|
```
|
|
|
|
By default, `sci` is installed to `/usr/local` but this can be overridden by providing the `PREFIX` var:
|
|
```sh
|
|
make
|
|
sudo make install PREFIX=/some/path
|
|
```
|
|
|
|
## Package
|
|
`sci` can be packaged for multiple distributions:
|
|
|
|
### Arch Linux
|
|
It is recommended that you use an arch linux distribution when building.
|
|
<!-- TODO: add a dockerfile for arch building -->
|
|
```sh
|
|
```
|
|
|
|
### Debian
|
|
It is recommended that you use the `deb-builder.dockerfile` docker image to build the debian image.
|
|
```sh
|
|
```
|
|
|
|
#### Building the debian builer docker image
|
|
```sh
|
|
docker build -t debbuilder deb-builder.dockerfile .
|
|
```
|
|
|
|
## Brainstorm
|
|
If you dont want to congest your CI server. Too bad. Write faster ci suites. (TODO: implement runners)
|
|
|
|
I would like to try to avoid writing a million REST APIs, as that just results in bloat usually.
|
|
|
|
Could this also be used as a systest replacement? The value of systest is the drivers and the configuration files. Not
|
|
really the controllers themselves. But yes, this system could replace the systest runner scripts. But not the drivers or
|
|
the configuration files themselves...
|
|
|
|
### Using
|
|
`/etc/sci/conf.d/pipelines.conf`
|
|
```txt
|
|
aaltitoad-release https://github.com/sillydan1/aaltitoad onpush-main /home/sci/pipelines/cpp-dev-git.sh
|
|
aaltitoad-develop https://github.com/sillydan1/aaltitoad onpush-dev /home/sci/pipelines/cpp-rel-git.sh
|
|
schoolnotes https://gitlab.com/sillydan/schoolnotes manual-name "echo hello"
|
|
horse https://gtz.dk/horse" manual-name "curl -X POST https://example.com"
|
|
cool-horse https://gtz.dk/horse" manual-name "docker run --rm -v $SCI_PIPELINE_DIR:$SCI_PIPELINE_DIR -v /home/sci/pipelines:/pipelines alpine /pipelines/your-script.sh"
|
|
rad-horse https://gtz.dk/horse" manual-name "/home/sci/pipelines/dockerized-horse.sh"
|
|
<name> <url> <trigger-filename> <pipeline-command>
|
|
```
|
|
Triggers are just bare files that reside in `/etc/sci/triggers/` - this means that multiple pipelines can use the same
|
|
trigger. When a trigger file is being written to or `touch`ed, this triggers the associated pipeline. Trigger files are
|
|
automatically created by the `scid` daemon during startup or reconfiguring.
|
|
|
|
Any webhooks should be managed by shim-APIs / adapter APIs that are very slim (possibly flask) REST APIs that just
|
|
listen for webhook event triggers and does a simple `touch <associated file>`. It could even read the same configuration
|
|
file as `scid` uses to autogenerate these webhooks. But it is explicity NOT part of the core system, as that would be
|
|
unneeded bloat.
|
|
|
|
How do I get information about a running pipeline? Perhaps a `/var/tmp/sci/<pipeline-name>` is continually updated with
|
|
runtime information? Then you can `cat`, or `tail -f` it to get the information.
|
|
|
|
`scid` manages running subprocesses, so it must be implemented in a language that handles that nicely.
|
|
|
|
Pipelines - Should pipelines have a custom language? NO! God already gave us shell scripts! Why would we reinvent the
|
|
wheel? Take a page out of Jenkins' book. `scid` should just call some command on trigger. That command could / should
|
|
usually be a quick shell script (these calls are executed in the dispatched separate thread, so dont worry about
|
|
sleeping or whatever you need to do). These scripts are equivalent to the Jenkins pipelines, and should update their
|
|
current state in `/var/tmp/scid/$PIPELINE_NAME` by simply writing to the file (when implementing runners, we probably
|
|
need to `rsync` these files automagically). Oh, and dont worry about cleaning up. Each pipeline execution will be placed
|
|
in a `/tmp/<pipeline-name>-<uuid>/` directory and will be executed by a gimped user. If you need to interact with the
|
|
linux stuff and it might break other pipelines or whatever, then your pipeline-command should just start a docker
|
|
container, volume mount the tmpdir and what else you need and run your command in that.
|
|
|
|
Need multistep pipelines? Then write a multistep shell script! You're the boss of your own CI server man.
|
|
|
|
```sh
|
|
#!/bin/bash
|
|
# cpp-dev-git-pipeline,sh
|
|
# NOTE: Will be executed from /tmp/$SCI_PIPELINE_NAME-<uuid>/
|
|
git clone -b dev "$SCI_PIPELINE_URL" "$SCI_PIPELINE_NAME"
|
|
echo "clone success" > "/var/tmp/sci/$SCI_PIPELINE_NAME" # You can have any kind of data in the pipeline status file.
|
|
cd "$SCI_PIPELINE_NAME"
|
|
cmake -B build
|
|
cmake --build build
|
|
```
|
|
|
|
## Secrets
|
|
Not planned yet. But they could be as simple as a keystore.
|
|
|
|
## Development
|
|
What language should I implement `scid` in?
|
|
- `c` slow devtime, decent tooling, very minimal and forces you to be minimal also. Maybe
|
|
- `c++` slow devtime, good tooling, very familiar. Can easily get out of hand. Maybe
|
|
- `python` quick, easy, slow performance, terrible multiprocessing capabilities. Maybe
|
|
- `java` slow devtime, bloated. No.
|
|
- `rust` slow devtime (not familiar), great tooling, too many risks for a first prototype. No.
|
|
- `shell` quick, easy, instant spaghetti. No.
|
|
|
|
I choose `c`!
|
|
|
|
I also choose `Makefile`s! - Just to force myself to use another build system than CMake!
|
|
If you want `compile_commands.json` files, you should use [bear](https://github.com/rizsotto/Bear) as it works well
|