Using Git Post-Commit Hooks and Portainer Web API to Update Your Docker Stacks

This project contains the configurations for managing Portainer stacks using docker-compose.yml files. It enables effective versioning of stack configurations and ensures that changes are automatically applied to the Portainer instance via a Git post-commit hook and the Portainer Web API.

Features

• Version Control: Track changes to stack configurations using Git.
• Post-Commit Automation: Automatically applies changes to the Portainer instance when configurations are updated.
• Portainer Web API Integration: Updates stacks directly using Portainer’s API.
• Install script: Automate installation of the whole thing
• Test feature: Integrated Test Functionality when using the --dry-run argument.

My Stacks - An Overview

I run these stacks, those are in the repo, but you need to create your own and change the scripts accordingly.

1. Mediaserver

• Description: Configuration for a Plex Media Server.
• Purpose: Host and manage your media library efficiently.

2. Stats

• Description: Monitoring stack using Grafana, Prometheus, and Node Exporter.
• Purpose: Visualize and monitor server metrics.

3. Torrents Management

• Description: Tools for searching for media files via torrents in an isolated environment with VPN support. Integration with qbittorrentvpn
• Purpose: Enhance privacy and security while torrenting.

3. Firefox VPN

• Description: Container that runs the Firefox web browser in an isolated environment with VPN support. Enhance privacy and security while browsing the web through a specific VPN connection.
• Purpose: Browsing the web anonymously without leaving traces on the host system. Accessing geo-restricted content via a VPN.

Project Structure

.
├── stacks
│   ├── mediaserver           <─────┐   **stack directory names must be the same as the stack name on portainer
│   │   └── docker-compose.yml      │
│   ├── stats                       │ 
│   │   └── docker-compose.yml      │
│   ├── torrents-tracker      <─────┤           
│   │   └── docker-compose.yml      │
│   ├── firefoxvpn            <─────┘         
│   │   └── docker-compose.yml   
├── scripts
│   └── update-stack.py
├── hooks
│   └── post-commit   <─────┐ same file
├── .git/hooks              │
│   └── post-commit   <─────┘ .git/* cannot be commited
├── logs
│   └── post-commit.log
├── .env  [environment variables difinitions file]
├── test
│   └── commit-msg
├── venv  [virtual environment]
└── test
   └── commit-msg

Key Files

• stacks/mediaserver/docker-compose.yml: Configuration for the Plex Media Server stack.
• stacks/stats/docker-compose.yml: Configuration for the Grafana, Prometheus, and Node Exporter stack.
• stacks/torrents-management/docker-compose.yml: Configuration for the Torrents Management stack.
• scripts/update-stack.py: Python script to update stacks using the Portainer Web API.
• .hooks/post-commit: Copy of the git hook that triggers the Python script after changes are committed, should be copied to .git/hooks/post-commit for it to work
• .git/hooks/post-commit: Git hook that triggers the Python script after changes are committed.

How It Works

1. Configure It:
   • Set the environment values and the .env file. You need PORTAINER_API_KEY and PORTAINER_URL. For the specific portainer stacks config, you may need other tokens depending on how you store secrets.
2. Make Changes:
   • Modify the desired docker-compose.yml file in the stacks directory.
3. Commit Changes:
   • When a commit is made, the post-commit hook checks if specific files have been modified.
   • the file is parsed
   • Stack ID if retrieved from the portainer API
4. Apply Changes:
   • If a target file (e.g., stacks/mediaserver/docker-compose.yml) is updated, the update-stack.py script is executed.
   • The script uses the Portainer Web API to update the corresponding stack.

Clone

Clone the repo from git@github.com:arsscriptum/docker-stacks-config.git

git clone git@github.com:arsscriptum/docker-stacks-config.git

Easy install

Use the install script

./scripts/install.sh 

or using the arguments

./scripts/install.sh 10.0.4.65:9000 rwds_45YukkggSSDFFSFD345SFhhsn6offkj0=

Install virtual environment

At the end of the install script, you will be asked if you want to install virtual environment, press ‘y’. Or you can run this script later:

./scripts/setup-venv.sh 

Prerequisites

Software Requirements

• Python 3.7+
• Required Python Libraries:
  • requests
  • python-dotenv

Virtual Environment

I run Python in a virtual environemnt, this is a personal choice, you may want to do the the same. To do so, see the section below on Creating a virtual environment for Python

Environment Variables

You need to provide the following environment variables for the script:

Option 1: Use a .env File

Create a .env file in the project root with the following content:

PORTAINER_API_KEY=your-portainer-api-key
PORTAINER_URL=http://your-portainer-url:9000

Option 2: Set Environment Variables Directly

Set the variables in your shell environment:

export PORTAINER_API_KEY=your-portainer-api-key
export PORTAINER_URL=http://your-portainer-url:9000

Post-Commit Hook Setup

Post-commit hooks are local-only and are not included when cloning the repository. After cloning the repository, you need to create the post-commit hook manually:

1. Navigate to .git/hooks in your repository.
2. Create a post-commit file with the content in this file:

   copy -f hooks/post-commit .git/hooks/post-commit
   

3. Make the hook executable:

   chmod +x .git/hooks/post-commit
   

4. Test the hook:

   .git/hooks/post-commit --test
   

5. Validate if it worked:

   cat logs/post-commit.log
   

Test - commit message file

Avoid commit message file changes during testing.

git update-index --assume-unchanged test/commit-msg

Usage

Updating a Stack

1. Modify a stack’s docker-compose.yml file.
2. Commit the changes:

   git commit -am "Update mediaserver stack configuration"
   

3. The post-commit hook triggers automatically, and the update-stack.py script updates the stack on the Portainer instance.

Commit Message Flags

@noupadte - No Update Option

When writing @noupdate in the commmit message, the stack are not updated.

• @noupdate

Prune @prune and Pull Image @pull-image Options

When updating a container, the --prune and --pull-image options are typically used to control how resources are managed during the update process.

These options are activated when you add the following lines in the commit message:

• @prune
• @pull-image

Here’s a detailed explanation of each:

--prune Option

• Purpose: The --prune option instructs the system to remove unused or dangling resources, such as old containers, networks, or volumes, that are no longer needed after the update.
• Use Case: This is particularly useful for maintaining a clean environment and freeing up disk space by removing resources associated with previous versions of the container.
• Effect: When used during an update:
  • Stops and removes outdated containers or services.
  • Deletes unused images, networks, and volumes that are no longer referenced by active containers.

For example:

• If an update modifies the stack’s configuration or images, the old containers may become redundant. --prune ensures these are cleaned up, leaving only the updated stack.

--pull-image Option

• Purpose: The --pull-image option forces the update process to fetch the latest version of the container image from the registry, even if a local copy exists.
• Use Case: Useful when you want to ensure the container runs the most up-to-date image, especially if the image has been updated in the registry since the last pull.
• Effect: When used during an update:
  • Checks the container registry for the latest version of the specified image.
  • Downloads the image if it has changed or if a newer version is available.
  • Ensures that the container is recreated using the latest image.

When Combined

When both --prune and --pull-image are used, the update process:

1. Pulls the latest image from the registry.
2. Stops the current container(s).
3. Removes old containers, unused images, and other dangling resources.
4. Starts new containers using the updated image.

This ensures a clean and up-to-date deployment environment.

IMPORTANT NOTE REGARDING STACKS NAMES

Your stacks in portainer must have the same names has the directory names in the stacks folder

When creating a stack on portainer, the name will need to be the same as the directory name in the stacks folder. Example

Stacks Creation Step 1

Add Stack

Stacks Creation Step 2

Enter name and script

Stacks Creation Step 3

Deploy

Testing

You can test using the hooks scripts with the --dry-run argument:

   ./hooks/post-commit --dry-run

NOTE When you are in test mode the commit message will be taken from the file test/commit-msg so you can test with @prune and @pull-image options.

testing options

The following options are available:

Usage: ./hooks/post-commit [options]
Options:
  -v, --verbose       Enable verbose output
  -d, --dry-run       Dry-run: will do everything but using fake commit files / msg
  -u, --update        Dry run will not update the server except if this is set
  -h, --help          Show this help message

Test: List Stacks

You can test getting the stack ids, by running the script get-stacks.sh. It will list the folder names and query your portainer server for the stack ids.

Test Examples

Dry run with logs to std out, wil not uptate the portainer stack

./hooks/post-commit -v -d

Dry run with logs to std out, will uptate the portainer stack, you can see if the stack was updated properly

./hooks/post-commit -v -d

Running The Update Script Manually

You can also run the script manually for testing or troubleshooting:

python3 scripts/update-stack.py \
    --stack-id 1 \
    --file-path stacks/mediaserver/docker-compose.yml \
    --portainer-url http://your-portainer-url:9000 \
    --prune \
    --pull-image

Logging

• The post-commit hook logs its execution and results in logs/post-commit.log.
• Logs include:
  • Parameters passed to update-stack.py.
  • Success or failure of the update process.

See the repository docker-stacks-config for more details.

About Guillaume Plante

A developper with a passion for technology, music, astronomy and art. Coding range: hardware/drivers, security, ai,. c/c++, powershell

Email : guillaumeplante.qc@gmail.com

Website : https://arsscriptum.ddns.net