We recently completed a migration from supervisor to pebble on our production hosts.

In this article, we will share how we set it up, why we did so, and what we anticipate forward for our infrastructure moving forward.

At NNDI we build systems that typically run on Linux based operating systems, though we have also shipped Windows services as part of our Consulting Work. Most of our services are built with Go and so far we have taken advantage of Go’s ability to build single binaries to power our simple deployment pipeline which involves just taking the binary and putting it on the server. However, running these binaries reliably requires a separate process, usually refered to as a service manager or process manager. An example of these is systemd which comes pre-installed with most Linux distributions as the primary process manager for the whole operating system.

Historically, we have used Supervisor for our service management needs and though it has served us well over the years, it still had some few things lacking that we needed. Pebble, a new-ish project from the folks at Canonical is a process manager like supervisor but with a few interesting features and some new ideas. When we came across it, we initially tried it but it wasn’t ready for us to use - but recently with updates to the project and a more recent round of testing, we were confident it was good enough for us to migrate to. We intend to contribute upstream if we run into any issues.

What is Pebble ?

To quote the Pebble documentation

Pebble is a lightweight Linux service manager. It helps you orchestrate a set of local processes as an organised set. It resembles well-known tools such as supervisord, runit, or s6, in that it can easily manage non-system processes independently from the system services. However, it was designed with unique features such as layered configuration and an HTTP API that help with more specific use cases.

How to install Pebble

This section details how to install pebble.

Pebble is available for Linux Operating systems and you can find binaries from the GitHub releases. The following instructions (series of commands), will help you setup pebble on a Linux host effectively

# Starting installation of pebble github.com/canonical/pebble"
export PEBBLE_VERSION=1.19.2

# Downloading pebble from GitHub releases"
wget "https://github.com/canonical/pebble/releases/download/v${PEBBLE_VERSION}/pebble_v${PEBBLE_VERSION}_linux_amd64.tar.gz"
tar zxvf "pebble_v${PEBBLE_VERSION}_linux_amd64.tar.gz"

rm "pebble_v${PEBBLE_VERSION}_linux_amd64.tar.gz"

#  Installing pebble to /usr/local/bin"
$ sudo mv pebble /usr/local/bin
$ sudo ln -s /usr/local/bin/pebble /usr/bin/pebble

#  Creating a pebble system user account" 
$ sudo adduser --system --group pebble

#  Creating pebble required directories PEBBLE_DIR at /var/lib/pebble" 
$ sudo mkdir -p /var/lib/pebble/default/ 
#  Creating pebble socket file as PEBBLE_DIR/.pebble.socket \
#          you can customize using PEBBLE_SOCKET=/path/to/socket"
$ sudo touch /var/lib/pebble/default/.pebble.socket

#  Granting permissions to files to pebble user"
$ sudo chown -R pebble:pebble /var/lib/pebble

#  Creating pebble systemd service unit" 

$ cat > pebble.service <<'EOF'
[Unit]
Description=Pebble
Wants=network-online.target
After=network-online.target

[Service]
Environment=PEBBLE_DEBUG=1
ExecStart=/usr/local/bin/pebble run
KillMode=process
Restart=on-failure
RestartSec=50s

[Install]
WantedBy=multi-user.target
EOF

$ sudo cp ./pebble.service /etc/systemd/system/pebble.service 

#  Enabling pebble systemd service at startup"
$ sudo systemctl enable pebble

#  Starting pebble service via systemd"
$ sudo systemctl start pebble

#  pebble installed sucessfully, congratulations!! executing 'pebble services'"

Comparing pebble and supervisor

Now that we have pebble installed, let’s see how to use the two to accomplish the same task of running a program. If you are following along this article, we are going to assume you already have supervisor installed via your OS package manager.

Using supervisor to run a program

A typical supervisor program has configuration that looks something like this.

NOTE: This is not a comprehensive example of all the options supervisor provides. See Configuration

[program:myprogram]
cmd="/path/to/myprogram server"
directory="/opt/myprogram/"
user="myuser"
group="myuser"

This configuration tells supervisor to run your program using the given command, within the specific working directory and as a specific user/group. You have to place this configuration in the appropriate directory for supervisor to pick it up, usually one of the following

• /etc/supervisord/conf.d/ e.g. /etc/supervisord/conf.d/myprogram.conf
• /etc/supervisor.d/ e.g. /etc/supervisor.d/myprogram.conf

With that in place, you can start or restart supervisor and start your program like so:

$ sudo systemctl restart supervisor

$ sudo supervisorctl start myprogram

$ sudo supervisorctl status
myprogram                      RUNNING   pid 126591, uptime 2 days, 9:52:20

Using pebble to run a program

Pebble uses the concept of layers. Programs are configured as layers which can be overriden and/or merged. Layers allow you to modify aspects of the configuration for a single or group of services being run via pebble. This flexibility allows you to patch configuration of an already configured service without having to restart the pebble daemon.

An example of a layer configuration is shown below:

NOTE: This is not a comprehensive example of all the options pebble provides. See Layer Specification

services:
  myprogram:
    override: merge
    startup: enabled
    cmd: "/path/to/myprogram server"
    working-dir: "/opt/myprogram/"
    user: "myuser"
    group: "myuser"

This layer definition tells pebble to run your program using the given command, within the specific working directory and as a specific user/group. You can save this file as 001-myprogram.yaml

$ sudo pebble add myprogram 001-myprogram.yaml

We can then run the program using the following

$ sudo pebble start myprogram

$ sudo pebble services

Way forward for NNDI: Path to kubernetes-like orchestration

We look forward to reaping the benefits of a more modern process management tool.

If you have a small team and less than 100 hosts to manage, Pebble can give you Kubernetes-like features such as:

• Performing health checks
• Management of identities of users who control pebble
• Sending logs from services to Loki
• Control the services via an API
• Integrating using Go client library for controlling/interacting with Pebble

If you have greater than 100 hosts, it would probably be wise to invest in Kubernetes since that scales much better with all the automation. And tools like CAPL make this even easier for large fleets.