GitLab CI CD Pipelines for Home Lab: A Step-by-Step Guide - Virtualization Howto

type

status

date

summary

What is GitLab?

It is a solution that I have been using in the home lab for quite some time now. It is, in my opinion, one of the best self-hosted code repositories and makes setting up and learning CI/CD very intuitive and easy to learn.

Gitlab cicd

There are many other great solutions you can use as well, such as Jenkins, etc. So the concepts we are learning in this post will apply across the board. Each solution has a slightly different way of implementing the concepts and terminology that goes with a specific solution. So, keep that in mind.

Installing Gitlab

You can learn more about Gitlab here: Download and install GitLab | GitLab.

It is very easy to get started with GitLab. Let’s look at how to easily spin up a Gitlab container image for your home lab, using Docker Compose. Below is an example of Docker compose code I am using.

Gitlab Container Registry

We won’t use the container registry in this demo. However, GitLab has a built-in container registry you can enable.

The project’s container registry plays an important role in managing Docker images used in the pipeline. The registry allows you to store and retrieve Docker images. For instance, your pipeline may build a new container image and push that image to your GitLab repository or another repository, such as Docker Hub.

Overview of GitLab CI/CD

GitLab CI/CD is a tool that is designed with software delivery with continuous integration and continuous deployment in mind. With CI/CD, you can automate the software development process, efficiently commit changes to the code repository, and deploy them to a production environment.

Check out my post that I posted recently covering the top DevOps containers in 2023: Top 10 DevOps Containers in 2023. Gitlab is a part of that list.

Continuous Integration

Continuous Integration (CI) is a crucial GitLab feature that enables developers to merge their changes into the master branch regularly.

Each commit initiates a CI process, automatically triggering a pipeline job that can do many things like verify the integrity of new code changes through a set of predefined tests using some type of test suite, build container images, and push images to a container registry.

Continuous Deployment and Delivery

Continuous Deployment (CD) is the logical next step following CI. It automatically deploys every validated change to the production environment, while Continuous Delivery ensures every change is deployable but requires manual deployment.

Both are essential for achieving an automated, streamlined software delivery process in the development lifecycle.

GitLab Runner

When you install Gitlab, you have to provision what Gitlab calls a GitLab Runner. GitLab runners core component that executes all operations defined in the .gitlab-ci.yml file (the special-purpose file that contains the Gitlab pipeline code).

It handles various tasks, and you can install it on different systems, including your local machine, Docker container, Linux host, Windows host, etc to meet specific project requirements.

Installing a GitLab runner

We can spin up a simple Gitlab runner as part of our Docker Compose code, using similar code to the below.

Connecting a Gitlab runner to your GitLab CI CD pipeline

You need to go through a process to connect your GitLab runner to your GitLab CI CD pipeline. Once you have your Gitlab and Gitlab runner stood up, you can create a project in Gitlab. After you have a project (we will show how to create one in the section below), go to the Project Settings > Runners > Expand.

Expanding runners configuration

Here you can click the ellipse next to New project runner, and then click the copy icon that appears to copy your token. You will use this to add the Gitlab runner to the project.

Copy your registration token for the gitlab runner

Now, you will exec into your Gitlab runner container and run the command gitlab-runner register.

You will be taken through registering the Gitlab runner with your Gitlab instance. Note the following prompts:

Enter the Gitlab instance URL

Enter the registration token

Enter a description for the runner

Enter tags for the runner

Enter optional maintenance note for the runner

Enter the default Docker image

Running the gitlab runner wizard to connect to a project

If you refresh your CI CD settings page for the Runners configuration, you will see your new GitLab runner assigned and it should show as “green”.

Gitlab project runners

GitLab projects are tied to a repository

When you create a project in GitLab, it is tied to a repository. It means that the repo that holds your code files is tied to the project. If you want to run PowerShell, PowerCLI, or other types of scripts, this is where we will check in our code.

In Gitlab, from the left sidebar create your project.

Creating a new project in gitlab

Create a blank project.

Create a blank gitlab project

Name your project and create the project URL and slug. Also, pick a group or namespace.

Blank project details

You will see your Clone button. Here is where we will clone our repository locally so we can work with the project.

Clone your gitlab repository

Now from the command line, you can use the git clone command to clone down your repository. Enter your credentials and authenticate to your GitLab instance.

After you authenticate, your files will be cloned down.

The git clone runs successfully

Open your cloned repo in VS Code

Now, we can open the repo in VS Code.

Open a folder in visual studio code

Building a quick PowerShell example for PowerCLI

So, for a quick example, let’s build a pipeline that will use a PowerCLI container to run a simple PowerCLI script against a home lab vCenter Server. We will create two files in VS Code:

.gitlab-ci.yml

powershell.ps1

The .gitlab-ci.yml file

A .gitlab-ci.yml configuration file is pivotal for creating efficient GitLab CI/CD pipelines. This file serves as a blueprint, guiding the GitLab Runner through various predefined jobs, stages, and tasks within the pipeline. It ensures your code is tested, built, and deployed systematically, using the principles of continuous integration and continuous deployment.

The contents of the .gitlab-ci.yml file will look like the following:

The contents of the powershell.ps1 will look like this:

In the PowerCLI script, I am running a simple query to pull all powered on VMs from the environment. Then emailing the results to myself using my Mailrise server.

Below, we have created both files in VS Code.

Files created in vs code

Stage, commit, and push changes to your GitLab repo

Click on the Source control icon in VS Code. Then, click the plus sign next to Changes. This will stage changes to be ready to commit changes.

Stage and commit your files

Now, from the command line we will commit and push changes to our GitLab repository.

Commit and push your files to your gitlab repository

As soon as you push your commit to your GitLab repo, your CI CD pipeline will attempt to run. If you see a failed pipeline run, you can look at the details of the failure and see what is failing.

Gitlab ci cd failure

If you don’t see any errors, your task will look something like this:

Successful pipeline run

Frequently Asked Questions

How can I configure a GitLab Runner on my local machine?

To configure a GitLab Runner, first, install it on your local machine. Upon installation, register it with your GitLab instance using the registration token obtained from your GitLab project’s settings. Adjust the config.toml file as necessary to meet the requirements of your CI/CD pipelines and the tasks it will execute.

What steps should I follow if my pipeline fails?

When a failed pipeline occurs, begin by inspecting the job logs within the GitLab interface. These logs provide detailed information about the pipeline’s operation and where it failed.

Look for error messages or failed tests, and make the necessary code or configuration file adjustments. Rerun the pipeline once changes are committed.

Can I use Docker commands within the .gitlab-ci.yml file?

Yes, Docker commands can be incorporated within the .gitlab-ci.yml file. You might use Docker to build container images, push them to your project’s container registry, or pull images from public or private registries. Ensure your GitLab Runner is configured to use the Docker executor to run jobs in Docker containers.

How do I manage access controls for my CI/CD pipelines?

Access controls for CI/CD pipelines are managed within the GitLab interface. Navigate to your project’s settings, then access the “CI / CD” section found in the left sidebar. Here, you can define protected branches, add deploy tokens, or set up specific runner access for sensitive jobs.

What’s the role of environment variables in GitLab CI/CD?

Environment variables in GitLab CI/CD are essential for dynamically adjusting your pipeline’s behavior without altering the .gitlab-ci.yml file. These variables can store sensitive data, like API keys, or define deployment environments. Define them within the “Variables” section of your project’s CI/CD settings.

How do I ensure that my CI/CD pipelines are secure?

Security in CI/CD pipelines encompasses protecting your code, sensitive data, and the pipeline processes themselves. Employ access controls, use secure connections (like HTTPS), avoid hardcoding sensitive information in files, and routinely monitor and update your GitLab instance and Runner to the latest secure versions.

Why does my job fail even though the previous stages were successful?

Jobs within CI/CD pipelines might fail due to various reasons, even if previous stages succeeded. Possible causes include code errors in the specific job, misconfiguration of job settings, issues with runner availability, or problems with external services the job interacts with. Inspecting job logs will offer insights into the failure’s root cause.

Is it possible to trigger a pipeline manually?

Yes, GitLab CI/CD allows for manual triggering of pipelines. You can configure manual jobs within your .gitlab-ci.yml file, which will only run when manually started from the GitLab interface. This feature is especially useful for deployment to production or other sensitive operations that require approval or manual review before execution.

Wrapping up

GitLab is an extremely powerful CI CD solution that I have been running for quite some time now in the home lab and in production. It has many capabilities that can allow infrastructure engineers to start getting their feet wet in DevOps processes and using source control to keep track of their code. Hopefully, this tutorial will help anyone who wants to start playing around with GitLab and DevOps to see that it isn’t too difficult to get started.