When and Why Building a CI Pipeline is Important in Web Development

Regardless of where you’re at in understanding web development, we can all appreciate efficiencies. Starting out as a web developer, I was using the good ol’ FTP (file transfer protocol) to move important web files from my local to any server. Drag and drop and wait. Sometimes wait a while. Connections like this can be secure or insecure depending on that port (ex: 21 or 22) you are using, but generally connections just require the IP/host, username, and password, to get onto the server and see what’s there.

I remember when I first learned about SSH, the fingerprint-like validation or “handshake” between any two servers. The keys. No passwords. The public and the private and to never share the private (always keep that local).

When to setup a CI Pipeline

If you’re developing a website or an application and you are going to be making changes to the files of it or updates to it to keep it “alive” and evolving with a brand or company, product or service, then you’ll need to consider a better way to get files from the developer server (their laptop or whatever) to the www publicly accessible IP, aka their domain, on a regular and quick basis over FTP as it’s a bit antiquated in general. Now cloud computing and AWS is a whole other topic that I’m less experienced on, but for basic server-to-server connections, SSH + rsync + github actions on a .yml file will serve the need way better than opening up CyberDuck, FileZilla, Transmit, or whatever other FTP client you are used to.

The importance of Git and branches when building CI Pipelines

So, basically when you build CI (Continuous Integration), you have to consider what is “production ready” and staged development, and when to “push” the updates from your computer to a publicly accessible computer, aka a website on the world wide web. This article is not to get into the weeds on main/master, staging/develop branches, but just know they are super important and that main goes to prod, and ideally you have a staging environment to push to (depending on what the client is using and/or has available for hosting…)

The long end of the short of it is that these factors apply when building a CI pipeline,

• How often you are going to be making updates
  • Usually this applies most to the files, not the database
• Considering security, if you are updating files and need to push them connecting via CI and a SSH handshake is much safer than using FTP.

When not to use a CI/CD pipeline

Well I’m sure there’s other reasons and I’ll speak from mostly a WordPress experience standpoint, but if you’re using a theme that is a “page builder” theme like Divi, Elementor, etc., then most of the content and structure (data and presentation are usually separated for better performance, among other things) of your website is stored in the database. And this demo is not to cover how to build CI/CD with database structures.

Setting up your first CI

So to start, you’ll need to create a .github directory inside your root directory, then in there a workflows directory.

We’ll be building the CI pipelines as .yml files and while you can build actions in Github, Gitlab, Bitbucket, Travis, etc etc, we’ll just do a simple .yml file that will define which branch to deploy, which SSH keys to reference, which directories to push and where, and how to work with secrets.

Specific Instructions for building a CI pipeline

Prerequisites: you are using git with Github and have SSH access to the server you want to push files to via git commands.

1. Create .github/workflows directory in your root folder
2. Create staging.yml and main.yml files

*be sure you have SSH access on the server you want to build a pipeline to.

Generate a SSH key-pair specific to the project

cd to your local .ssh folder and generate new ssh key pair,

ssh-keygen -t rsa -b 4096 -C "yourname@website.com"

Name the key/pair the name of the client name (do not override your id_rsa files, hehe…)

Copy the private key to your clipboard. We are going to share the private key with Github. Remember when I said to never share your private key? Well.. We’re going to add it to Github as a “secret” so it’s only accessible to a *private* repository with permissions setup properly…

Connect Github with production server

Login to GitHub, go to the repo, and go to the Settings -> Secrets and Variables -> Actions
Add new repository secret
Name the secret SSH_PRIVATE_KEY and paste the private key from clipboard to value field, hit save

Next,

Add the public key (.pub) to the server you are connecting to, to the .ssh/authorized_keys file. SSH onto the server by finding the SSH command in Kinsta,

Ex: ssh username@12.345.67.89 -p 12345

Once logged into the server,

cd .ssh
touch authorized_keys
nano authorized_keys

Paste the contents of the .pub key to the file
Write out the file, then save.

Exit

Then test the connection/handshake locally:

On your localhost, if you don’t already have it,
cd .ssh
touch config

Open the folder in your editor and review the following config and replace with the appropriate connection details to the server you are connecting to,

Host websitename_prod
    User serverusername
    Hostname 12.345.67.89.0
    Port 12345
    IdentityFile ~/.ssh/clientsshkeyname  

Save and exit Terminal/bash, relaunch.

Now if you type (example) ssh websitename_prod it will use the details from your config file and ssh to the server specified.

If that works, you’re ready to finalize the staging.yml and main.yml files previously created in the repo.

example of a CI pipeline .yml file

name: domainname.com - branch name pipeline

on:
  push:
    branches:
      - main      

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Sync
      env:
        src: './wp-content/themes/name-of-theme/'
        dest: 'serverUsername@12.345.67.89.0:~/path/to/theme/on/server/name-of-theme'
      run: |
        echo "${{secrets.SSH_PRIVATE_KEY}}" > deploy_key
        chmod 600 ./deploy_key
        rsync -chav --omit-dir-times --delete \
          -e 'ssh -p 22 -i ./deploy_key -o StrictHostKeyChecking=no' \
          --exclude /deploy_key \
          --exclude /.git/ \
          --exclude /.github/ \
          --exclude /node_modules/ \
          --exclude /wp-content/uploads/ \
          ${{env.src}} ${{env.dest}}

When updating the .yml file,

Lines to pay attention to the lines in red. In the order presented,

• be sure to name the CI pipeline so when it shows up in Github > Actions, you’ll identify it from other pipelines on the repository
• be sure you are pushing the right branch and that the .github/workflows directory is on that branch
• for the next two lines, src and dest, match the repositories’ structure and the directory within, to deploy the src file path to the dest file path on the server
  • A good thing to do is to SSH onto the server and cd into the folder you need to deploy to, then running a pwd real quick to check your sanity
  • another thing to report is the forward slashes before/after these paths are important to include either recursively or not
• the name of the secret you added to Github must match the secret reference on the .yml file
• Be sure that the SSH/rsync command includes the correct port you are connecting to.

I hope this helps you and your understanding of web development, CI/CD pipelines, and why we do this stuff in the end.