Lines

A pure bash clojureish CI pipeline.

Main Features

Complete CI engine
Execute local or remote
Pure data syntax edn
Clojure script syntax
Easy command line integration like (docker, kubectl, apt)
Concurrency execution
Modular and extensible

Why?

Alternative method to install clojure, using “clojure”, but without clojure
Tired to write yaml every day
Create something cool with Fleck

Installation
Job keywords
Modules
- shell
- docker
- scp
- template
- user module
EDN Pipeline
- Targets file
Clojure Pipeline
- Lines functions
Environment vars
Extensions
Development

Installation

requirements

bash 4
coreutils
git
docker ¹
ssh ²
scp ²

¹ docker module
² remote execution

install script

# download

curl -L https://github.com/rosineygp/lines-sh/releases/latest/download/lines > lines
chmod +x lines
sudo mv lines /usr/bin/lines

minimal usage

Create a file .lines.edn with followin data.

[{:apply ["echo hello world!"]}]

Just execute command lines

name: lines *******************************************************************
target: local
start: ter 22 dez 2020 21:38:01 -03
  cmd: echo hello world!
    hello world!
  exit-code: 0
status: true
finished: ter 22 dez 2020 21:38:04 -03 ****************************************

Job keywords

A job is defined as a hashmap of keywords. The commons keywords available for job are:

keyword	type	description
apply	array	the tasks that job will handler
name	string	optional job name
module	string	module name
target	hashmap	where the job will run
vars	hashmap	set env vars (branch name changes conforme current branch)
args	hashmap	arguments modules, each module has own args
ignore-error	boolean	when job fail the execution continue
retries	integer	number of retries if job fail, max 2

apply

Is the only required keyword that job needs. It’s an array of objects that will be executed by a module. Each element of array is a tasks and the job can handler N tasks. If any tasks has a exit code different than 0, the job will stop and throw the error. ignore-error and retries helps to handler errors.

{:apply ["uname -a"
         "make build"]}

using default module shell, apply is a array of strings.

{:module "scp"
 :apply [{:src "program.tar.gz" :dest "/tmp/program.tar.gz"}
         {:src "script.sh"      :dest "/tmp/script.sh"}]}

scp uses a list of hashmaps

name

name is just a label for the job.

{:name "install curl"
 :module "shell"
 :apply ["apk add curl"]}

module

module is the method executed by job (default is shell)

{:module "shell"
 :apply ["whoami"]}

Docker module

{:module "docker"
 :args {:image "node"}
 :apply ["npm test"]}

the builtin modules are:

module	description
shell	execute shell commands
docker	start a docker instance and execute shell commands inside it
template	render lines template and copy to it to destiny
scp	copy files over scp

is possible create custom modules

target

Host target is the location where the job will run. If any target passed the job will run at localhost.

{:target {:label "web-server"
          :host "web.local.net"
          :port 22
          :method "ssh"}}

Targets can be defined in separated file, during execution is possible to merge data with job and execute the same job in n hosts.

keywords	description
label	host label, just for identify
host	ip or fqdn for access host
user	login user
port	method port 22 is default
method	connection method, ssh is default

Is possible set another keywords for filter like group, dc or any other value you need to organizer targets.

After job executed it return themself with result values

vars

Variables will be inject in environment during tasks execution.

{:vars {MY_VAR_0 "lines"
        MY_VAR_1 "go"}
 :apply ["echo $MY_VAR_0"
         "echo $MY_VAR_1"]}

BRANCH_NAME and BRANCH_NAME_SLUG are inject in environment.

args

Args is the parameters of modules.

{:args {:sudo true}
 :apply ["apt-get update"
         "apt-get install htop -y]}

All tasks will run with sudo

ignore-error

If some task fail, lines will not stop the pipeline, just return the current task failed.

{:ignore-error true
 :apply ["whoami"
         "exit 1"
         "dpkg -l"]}

The tasks after error will not be executed.

retries

If some task fail, retry will run it again.

{:retries 2
 :apply ["ping -c 1 my-host"]}

The max retries are 2, but it can be increase setting LINES_JOB_MAX_ATTEMPTS at environment vars.

Modules

shell

Is it the default module, just spawn scripts to shell.

{:module "shell"
 :apply ["date"]}

Command will be executed inside a heredoc block

<command_header> bash -s <<-'LINES-BLOCK-EOF'
export <env vars>;
commands
LINES-BLOCK-EOF

arguments

keyword	type	description
sudo¹	boolean	apply commands using sudo
user¹	string	change current user
entrypoint	array	change initial entry command (default is `["bash" "-s"]`)

¹ needs pre configured sudoers (without password)

docker

Create a docker instance and execute commands inside it.

single instance

{:module "docker"
 :apply ["whoami"]}

start docker instance with default image (alpine)
run command whoami inside container

services

{:module "docker"
 :args {:image "ubuntu"
        :services [{:image "nginx"
                    :alias "nginx"}]}
 :apply ["apt-get update"
         "apt-get install curl -y"
         "curl http://nginx"]}

start docker instance with nginx image as a service and set network alias as nginx
start another docker instance with ubuntu image
install ubuntu packages
execute curl at service from ubuntu instance

download artifacts

Download files or directory from a docker instance.

{:module "docker"
 :args {:artifacts {:paths ["file"
                            "directory"]}}
 :apply ["touch file"
         "mkdir directory"
         "touch directory/file"]}

arguments

keyword	type	description
image	string	docker instance path name
entrypoint	array	change initial entry command (default is sh)
privileged	boolean	run job with privileged access and mount docker socket
services	array	services description
artifacts	hasmap	download artifact from docker instance

services description

Services is an array of hashmaps, is possible up N services with docker module, the following keywords can be used to build a service.

keyword	type	description
image	string	docker instance path name
vars	hashmap	like job vars but exclusive from instance service
alias	string	network alias name, otherwise slug image name
entrypoint	string	service entrypoint string, otherwise ‘’

artifacts description

Download a files or directories from a docker instance.

keyword	type	description
paths	array	file or folder relative or full path

scp

Copy files and folders to remote host over scp.

{:module "scp"
 :apply [{:src "./dist/command.bin"
          :dest "/usb/bin/command"}]}

apply arguments

keyword	type	description
src	string	file or directory source
dest	string	file or directory destiny
recursive	boolean	set true for directory copy

template

Simple template file that only replace values inside double brackets ``.

Hello !

template file

{:module "template"
 :vars {NAME "lines"}
 :apply [{:src "./hello-world.j2"
          :dest "/tmp/hello-world.txt"}]}

user module

Lines provides interface with custom user module.

Just put additional clojure scripts at .lines/modules/<module_name>/module.clj, like following example.

; .lines/modules/git/module.clj

; create a boilerplate function for git command
(str-use ["git"])

; custom user function, params: job (receive job definition), i (apply index)
; the function to return a string eg. `git clone -v git@github.com:rosineygp/mkdkr.git mkdkr`
(defn str-git-command-line [job i]
  (git ["clone"
        "-v"
        (get i :repos)
        (get i :dest)]))

; lines will call this function `lines-module-<module_name>`
(defn lines-module-git [job]
  (lines-task-loop job str-git-command-line)) ; loop handler

Using user module.

{:module "git"
 :apply [{:repos "git@github.com:rosineygp/lines.git" :dest "lines"}
         {:repos "git@github.com:rosineygp/mkdkr.git" :dest "mkdkr"}]}

EDN Pipeline

Pipelines is an array of jobs, and can be described using only edn.

; file: node.edn

[{:name "build"
  :apply ["npm install"]}
 [{:name "unit test"
   :group ["test"]
   :apply ["npm unit"]}
  {:name "mocha test"
   :group ["test"]
   :apply ["npm mocha"]}]
 {:name "deploy"
  :apply ["npm deploy"]}]

Jobs inside pipeline can be executed in parallel, just group then with [ array ], is possible use custom keywords for better notation like :groups.

# execute all jobs
lines -p node.edn

# filter only tests
lines -p node.edn -j group=test

Targets file

Is possible describe hosts targets using edn files, like pipeline.

; file: hosts.edn

[{:label "vm-0" :host "192.168.1.4" :method "ssh" :user "ubuntu"}
 {:label "vm-1" :host "192.168.1.5" :method "ssh" :user "ubuntu"}]

ssh only works with authorized_keys pre-configured, command ssh-copy-id can help configure it.

# execute pipeline in all hosts
lines -p node.edn -i hosts.edn

# filter only wm-0 host
lines -p node.edn -i hosts.edn -l label=wm-0

# filter jobs deploy and host wm-1
lines -p node.edn -i hosts.edn -l label=wm-1 -j name=deploy

EDN files are not safe is possible call functions dynamically.

; unsafe pipeline, extensions are based on this behavior

[{:apply [(str "echo " (time-ms))]}]

Clojure Pipeline

Powerful and dynamic pipelines with complex scenarios.

; generate single job for each yml found and test with docker image yamllint

(lines-pp (parallel
    (map (fn [x] (assoc {}
                        :name (get x :object)
                        :module "docker"
                        :args {:image "my/yamllint"}
                        :apply [(str "yamllint " (get x :object))]))
            (filter (fn [i] (= "yml" (get i :type))) (list-dir "src/")))))

This project use this case to test itself at .lines.clj.

Lines functions

The following functions will help to build lines clojure scripts.

name	description
job	execute edn job
parallel	execute jobs in parallel
str-use	mapping command line to function, return str not execute
use	mapping command line to function, run immediately
pipeline	execute end pipeline
lines-pp	beautiful print pipeline
lines-pp-minimal	beautiful print pipeline (single line)
merge-job-targets	generate a mixed list with edn pipelines and targets
isremote?	check if target will run local or in remote machine

src/core.clj and src/includes/lang-utils.clj has more useful functions.

Additional functions included in Lines.

filter

pmap

println_stderr

exit!

trap!

unset

str-join

str-subs

range

str-ident

mod

file-exists

file-write

unlink

and

hashmap-list

merge

key-name

call

callable?

get-in

even?

odd?

load-once

spit

The result of function job is the following data:

({:attempts 1
  :args {}
  :module "shell"
  :status true
  :apply ["echo hello world!"]
  :name "lines"
  :retries 0
  :target {:label "local" :method "local"}
  :pipestatus ((0))
  :finished 1608684590507
  :vars {"BRANCH_NAME_SLUG" "master" "BRANCH_NAME" "master"}
  :ignore-error false
  :start 1608684587657
  :result (({:exit-code 0
             :finished 1608684590253
             :cmd "echo hello world!"
             :stdout "hello world!"
             :stderr ""
             :start 1608684590233
             :debug "  bash -s <<-'LINES-BLOCK-EOF'\n export BRANCH_NAME_SLUG=\"master\" BRANCH_NAME=\"master\" ;\n echo hello world! \nLINES-BLOCK-EOF"}))})

Environment vars

Lines variables definition.

name	default	description
LINES_JOB_TTL	3600	max time that a job can run (only docker module)
LINES_JOB_MAX_ATTEMPTS	2	max number of retries
LINES_MODULES_DIR	“.lines/modules/”	modules default location
LINES_EXT_DIR	“.lines/ext/”	extensions default location
FLK_MAX_THREADS	undefined	max number of threads in parallel execution

Extensions

Is possible create functions and extend usability, all extensions must be keep at .lines/ext/, it is just a clojure script with useful functions.

; file: .lines/ext/apt-helper.clj

(str-use ["apt-get"])

(defn apt-update []
  (apt-get ["update"]))

(defn apt-args [options packages]
   (apt-get  (concat options packages)))

(defn apt-install [packages]
  (apt-args ["install" "-y"] packages))

(defn apt-remove [packages]
  (apt-args ["remove" "-y"] packages))

Now edn files or pipeline can call those functions.

; edn pipeline

[{:name "apt with extensions"
  :apply [(apt-update)
          (apt-install ["python"
                        "mysql-server"
                        "memcached"])]}]

Development

Build this project

./build.sh

> result
  flk lines

flk is same of Fleck with patches/ applied and lines is flk with all src/ included.

Testing changes in src/ without build.

./flk src/main.clj <params>

lines

A pure bash clojureish CI pipeline

Lines

Main Features

Why?

Table of contents

Installation

requirements

install script

minimal usage

Job keywords

apply

name

module

target

vars

args

ignore-error

retries

Modules

shell

arguments

docker

single instance

services

download artifacts

arguments

services description

artifacts description

scp

apply arguments

template

user module

EDN Pipeline

Targets file

Clojure Pipeline

Lines functions

Environment vars

Extensions

Development