I committed a code change. Thirty seconds later, Jenkins detected it. Two minutes after that, the application was running on AWS EC2 with a new version number — both the app and database containers live, accessible to users.

Zero manual steps.

Getting to this point meant fighting OpenSSL key formats that worked on my machine but failed in Jenkins, hunting through three different Git configs before finding the right user context, debugging authentication failures caused by a single misplaced quote, and learning that "works on my machine" isn't good enough when your build server runs different software versions.

But now: git push → automated build → versioned deployment → running application. The complete CI/CD workflow. Here's how I built it.

The Goal

Build a pipeline that automatically:

  1. Increments version in pom.xml
  2. Builds the JAR with Maven
  3. Builds a Docker image with the new version
  4. Pushes versioned artifact to Docker Hub
  5. Deploys to EC2 using Docker Compose
  6. Commits version bump back to Git

All from one commit. No manual intervention.

This is how modern software gets deployed. Not manual SSH, not clicking through UIs, not copying files with scp. Code gets committed → Jenkins automates everything → application runs on infrastructure where users can access it.

What I had before this project: Docker fundamentals, Jenkins configuration, shared library pattern, reusable pipeline functions. What was missing: the deployment piece — actually getting code onto servers and automating the complete flow.

Part 1: AWS Infrastructure Setup

I already had an AWS account from the Docker/ECR project. This time I set it up properly.

IAM Best Practice: Admin User Instead of Root

Created an IAM admin user instead of using root for everything. This is the first project where I actually followed that security practice instead of just knowing I should.

Why this matters: If the admin account gets compromised, I can delete it and create a new one. Can't do that with root — root account compromise means unrestricted access to everything, including billing.

Setup:

EC2 Instance Creation

Instance config:

Security group: security-group-docker-server

Inbound rules:

chmod 400 ~/.ssh/docker-server.pem
ssh -i ~/.ssh/docker-server.pem ec2-user@<public-ip>
Why ec2-user, not root?

Amazon Linux 2's default — a user with sudo access but no direct root login. Logging in as root directly is disabled by default, which is a security best practice.

Docker Installation and Configuration

sudo yum update -y
sudo yum install docker -y
sudo service docker start

# Add ec2-user to docker group (avoid needing sudo for every command)
sudo usermod -aG docker $USER

# Log out/in for group change to apply
exit
ssh -i ~/.ssh/docker-server.pem ec2-user@<public-ip>

# Verify
docker run hello-world
Docker Group Grants Root-Equivalent Access

Adding a user to the docker group means they can run privileged containers — effectively root-level access to the host. This is acceptable in a learning environment but is carefully controlled in production.

Part 2: Manual Deployment Baseline

Before automating, I deployed manually to understand what Jenkins would need to replicate.

# Authenticate to Docker Hub
docker login
# Creates /home/ec2-user/.docker/config.json

# Pull and run
docker pull nvastola/demo-app:1.0
docker run -d -p 8080:8080 nvastola/demo-app:1.0

# Verify
docker ps

Opened browser: http://<ec2-ip>:8080. Application loaded. Manual deployment successful.

This process — authenticate, pull, run, verify — is exactly what the pipeline will automate.

Part 3: The Architecture Decision That Taught Me Production Thinking

My initial thought: "Run Jenkins on the same EC2 instance as the app. No need for a separate server."

I actually set this up. Jenkins on EC2, application on EC2, everything worked.

Then I stopped and asked: What am I actually learning here?

If Jenkins and the app run on the same instance, Jenkins either SSHs into itself (circular and pointless) or skips SSH entirely and runs Docker commands locally (not remote deployment). Either way, I'm missing the learning objective.

Why Separation Matters

The curriculum separates Jenkins (DigitalOcean) from the app (EC2) deliberately. This teaches the production pattern: build infrastructure separate from runtime infrastructure.

I deleted the EC2 Jenkins setup. Created a DigitalOcean droplet instead. The "simpler" approach would have worked functionally — but it would have skipped understanding why production systems are architected with separation.

Sometimes the harder path teaches you the principles behind the patterns.

Part 4: The OpenSSL Key Format Battle

With Jenkins on DigitalOcean, I needed SSH access to EC2 for deployment. This is where I hit an environment-specific issue that took 30 minutes to debug.

Jenkins Setup on DigitalOcean

Created Ubuntu droplet, installed Docker, and ran Jenkins as a container:

docker run -d \
  -p 8080:8080 \
  -p 50000:50000 \
  -v jenkins_home:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --name jenkins \
  jenkins/jenkins:lts

Mounting /var/run/docker.sock gives access to the Docker daemon but doesn't install the docker CLI inside the container.

Error
docker: not found
The Fix

Install the Docker CLI inside the Jenkins container:

docker exec -it -u 0 jenkins bash
apt-get update && apt-get install -y docker.io

Now Jenkins can execute Docker commands against the host daemon.

After that: installed SSH Agent plugin, configured Maven, added GitLab/Docker Hub credentials, created a multibranch pipeline.

SSH Credentials for EC2

Created an SSH credential in Jenkins:

Updated EC2 security group to allow port 22 from the DigitalOcean droplet IP only. Then added the first deploy stage:

stage("deploy") {
    steps {
        script {
            def dockerCmd = 'docker run -p 8080:8080 -d nvastola/demo-app:1.0'
            sshagent(['ec2-server-key']) {
                sh "ssh -o StrictHostKeyChecking=no ec2-user@<ec2-ip> ${dockerCmd}"
            }
        }
    }
}
Why StrictHostKeyChecking=no?

Without this, SSH prompts "Are you sure you want to continue connecting?" Jenkins can't answer interactive prompts — the build would hang forever. This flag tells SSH to trust the host without asking, which is required for automation.

Committed, pushed, ran the build.

Error
Running ssh-add
Error loading key: error in libcrypto

The Investigation

Could I SSH from my local machine? Yes. Worked perfectly. Was the key correct in Jenkins? Re-copied it multiple times, deleted and recreated the credential. Same error. Was the username right? Yes.

I was stuck. Same key works locally, fails in Jenkins. Both using identical .pem file contents.

The Root Cause

Checked OpenSSL versions:

# Inside Jenkins container
docker exec -it jenkins bash
openssl version
# OpenSSL 3.5.4

# Local machine
openssl version
# OpenSSL 1.1.1

Different versions handle key formats differently.

AWS EC2 .pem files use PKCS#1 format (older):

-----BEGIN RSA PRIVATE KEY-----

OpenSSL 3.x prefers PKCS#8 format (newer):

-----BEGIN PRIVATE KEY-----

My local OpenSSL 1.1.1 tolerated both. Jenkins container's OpenSSL 3.5.4 was strict.

The Fix

Convert PKCS#1 to PKCS#8:

openssl rsa -in ~/.ssh/docker-server.pem -out ~/.ssh/docker-server-converted.pem

Updated the Jenkins credential with the converted key content. Deploy stage succeeded.

Lesson Learned

30 minutes of debugging to learn: "works on my machine" isn't enough. Software versions matter. The same key file behaved differently depending on the OpenSSL version reading it. Always test SSH credentials inside the actual build environment, not just locally.

Part 5: Building the Complete Pipeline

With SSH working, I built the pipeline incrementally, adding complexity step by step.

Docker Compose for Multi-Container Deployment

The initial deploy stage used docker run for a single container. Real applications need databases and supporting services alongside them.

Installed Docker Compose on EC2:

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Created docker-compose.yaml in the repository:

version: '3.8'
services:
  java-maven-app:
    image: nvastola/demo-app:java-maven-1.0
    ports:
      - 8080:8080
  postgres:
    image: postgres:15
    ports:
      - 5432:5432
    environment:
      - POSTGRES_PASSWORD=my-pwd

Updated the deploy stage to copy the file and run Docker Compose over SSH:

sshagent(['ec2-server-key']) {
    sh "scp docker-compose.yaml ec2-user@<ec2-ip>:/home/ec2-user"
    sh "ssh -o StrictHostKeyChecking=no ec2-user@<ec2-ip> 'docker-compose -f docker-compose.yaml up --detach'"
}

Shell Script Extraction

The deploy stage was getting complex. What if we need to stop old containers, clean up images, verify health? The Jenkinsfile would become unreadable.

Created server-cmds.sh:

#!/usr/bin/env bash

export IMAGE=$1
docker-compose -f docker-compose.yaml up --detach
echo "Success"

Updated docker-compose.yaml to use a variable:

services:
  java-maven-app:
    image: ${IMAGE}  # Now dynamic

Updated deploy stage:

def shellCmd = "bash ./server-cmds.sh ${IMAGE_NAME}"
def ec2Instance = "ec2-user@<ec2-ip>"

sshagent(['ec2-server-key']) {
    sh "scp server-cmds.sh ${ec2Instance}:/home/ec2-user"
    sh "scp docker-compose.yaml ${ec2Instance}:/home/ec2-user"
    sh "ssh -o StrictHostKeyChecking=no ${ec2Instance} ${shellCmd}"
}
Why This Is Better

Deployment logic is version-controlled, testable independently, and doesn't clutter the pipeline definition. Jenkinsfile orchestrates — scripts contain business logic. Adding complexity later doesn't touch the pipeline structure.

Dynamic Versioning

Still hardcoding nvastola/demo-app:java-maven-1.0. Every build overwrites the previous version. No traceability.

Added increment version stage:

stage('increment version') {
    steps {
        script {
            sh 'mvn build-helper:parse-version versions:set \
                -DnewVersion=\\\${parsedVersion.majorVersion}.\\\${parsedVersion.minorVersion}.\\\${parsedVersion.nextIncrementalVersion} \
                versions:commit'
            def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
            def version = matcher[0][1]
            env.IMAGE_NAME = "nvastola/demo-app:$version-$BUILD_NUMBER"
        }
    }
}

How it works:

Every push creates a unique, traceable version. If users report a bug in 1.1.2-14, I can trace back to that exact build and code state.

Committing Version Bumps

The version increments in the workspace, but that change isn't persisted. The next build would start from 1.1.0 again. Added a commit stage:

stage("commit version update") {
    steps {
        script {
            withCredentials([usernamePassword(credentialsId: 'gitlab-creds',
                                              passwordVariable: 'PASS',
                                              usernameVariable: 'USER')]) {
                sh 'git remote set-url origin https://$USER:$PASS@gitlab.com/nvastola/java-maven-app-aws.git'
                sh 'git add .'
                sh 'git commit -m "ci: version bump"'
                sh 'git push origin HEAD:jenkins-jobs'
            }
        }
    }
}

Now the incremented version commits back to Git. Continuous progression: 1.1.1 → 1.1.2 → 1.1.3 → ...

Part 6: The Debugging Marathon

With the structure complete, I hit five different failure types. The Jenkins Stage View tells the story: builds #7–#20, multiple red builds before finally hitting green. Each failure revealed a specific knowledge gap about how Jenkins, Docker, and Git interact.

Issue 1: Missing Repository Name

Error
docker build -t 1.1.1-13 .
docker push 1.1.1-13
denied: requested access to the resource is denied

What happened: Image name was 1.1.1-13 instead of nvastola/demo-app:1.1.1-13. I set IMAGE_NAME to $version-$BUILD_NUMBER, forgetting the repository prefix. Docker tried pushing to docker.io/library/1.1.1-13 (the default public namespace), which I don't have access to.

The Fix
env.IMAGE_NAME = "nvastola/demo-app:$version-$BUILD_NUMBER"
Lesson Learned

Docker image format is registry/repository:tag. Missing any part causes unexpected routing failures — Docker silently assumes defaults that may not be what you intended.

Issue 2: Misplaced Quotes in dockerLogin()

Error
denied: requested access to the resource is denied

Credentials were correct — I tested them manually. The problem was in my shared library:

sh "echo '${script.PASS} | docker login -u '${script.USER} --password-stdin"

The closing single quote comes after the pipe. Bash sees:

echo 'mypassword | docker login -u 'nvastola --password-stdin

The pipe is inside quotes — treated as literal text, not a shell operator. Docker login receives nothing through stdin.

The Fix
sh "echo '${script.PASS}' | docker login -u '${script.USER}' --password-stdin"

Pipe is now outside quotes. Authentication works.

Lesson Learned

I spent 10 minutes checking credentials, the Jenkinsfile, the Git branch — everything except the shared library code. Quote placement matters: one character can break an entire authentication flow without giving you a useful error message.

Issue 3: The Typo (usernamePassword vs usernameVariable)

Error
groovy.lang.MissingPropertyException: No such property: usernamePassword
withCredentials([usernamePassword(credentialsId: 'gitlab-creds',
                                  passwordVariable: 'PASS',
                                  usernamePassword: 'USER')]) {  // ← wrong

I typed usernamePassword twice. The third parameter should be usernameVariable.

The Fix
withCredentials([usernamePassword(credentialsId: 'gitlab-creds',
                                  passwordVariable: 'PASS',
                                  usernameVariable: 'USER')]) {
Lesson Learned

Took 15 minutes to find. I checked the credential ID, repository URL, and Git commands — everything except the actual parameter name. When failing with a missing property error, check for typos in method calls and named parameters first.

Issue 4: The Git Config Hunt (Three Tries)

Error
*** Please tell me who you are.
Run
  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

Git needs to know who's committing. But where exactly do you set that when Jenkins is running as a container on a droplet?

Attempt 1: DigitalOcean droplet

git config --global user.email "noahvastola@gmail.com"
git config --global user.name "Noah Vastola"

Ran build. Same error.

Attempt 2: Jenkins container as root

docker exec -it -u 0 jenkins bash
git config --global user.email "noahvastola@gmail.com"
git config --global user.name "Noah Vastola"

Ran build. Same error.

Attempt 3: Jenkins container as jenkins user

docker exec -it jenkins bash
git config --global user.email "noahvastola@gmail.com"
git config --global user.name "Noah Vastola"

Ran build. Success.

The Fix

Set the Git config as the jenkins user inside the container — not the droplet's root, not the container's root, but the actual jenkins user that runs the pipeline workspace.

Lesson Learned

Jenkins pipeline steps run as the jenkins user inside the container. That's the user whose config matters. When debugging environment or config issues in Jenkins, always think about which user context is actually executing the command.

Issue 5: postgres:lts Doesn't Exist

Error
manifest for postgres:lts not found: manifest unknown

I assumed PostgreSQL used lts for latest stable, like some other images do.

The Fix
image: postgres:15

PostgreSQL uses numbered versions (postgres:15, postgres:16). There is no lts tag.

Lesson Learned

Don't assume Docker tag naming conventions carry over between images. Every image maintainer chooses their own tags. Verify the tag exists on Docker Hub before using it.

The Pattern

Looking at builds #7–#20, every failure fits a category:

Each failure revealed a specific knowledge gap about how Jenkins, Docker, Git, and bash interact. Not "debugging is hard" — each taught something concrete about system integration.

Part 7: The Complete Pipeline

With all five failure types resolved, the pipeline ran clean.

Final Jenkinsfile:

#!/usr/bin/env groovy

library identifier: 'jenkins-shared-library@master', retriever: modernSCM(
    [$class: 'GitSCMSource',
    remote: 'https://gitlab.com/nvastola/jenkins-shared-library.git',
    credentialsID: 'gitlab-creds']
)

pipeline {
    agent any
    tools {
        maven 'Maven'
    }
    stages {
        stage('increment version') {
            steps {
                script {
                    sh 'mvn build-helper:parse-version versions:set \
                        -DnewVersion=\\\${parsedVersion.majorVersion}.\\\${parsedVersion.minorVersion}.\\\${parsedVersion.nextIncrementalVersion} \
                        versions:commit'
                    def matcher = readFile('pom.xml') =~ '<version>(.+)</version>'
                    def version = matcher[0][1]
                    env.IMAGE_NAME = "nvastola/demo-app:$version-$BUILD_NUMBER"
                }
            }
        }
        stage('build app') {
            steps {
                buildJar()
            }
        }
        stage('build image') {
            steps {
                script {
                    buildImage(env.IMAGE_NAME)
                    dockerLogin()
                    dockerPush(env.IMAGE_NAME)
                }
            }
        }
        stage("deploy") {
            steps {
                script {
                    def shellCmd = "bash ./server-cmds.sh ${IMAGE_NAME}"
                    def ec2Instance = "ec2-user@<ec2-ip>"

                    sshagent(['ec2-server-key']) {
                        sh "scp server-cmds.sh ${ec2Instance}:/home/ec2-user"
                        sh "scp docker-compose.yaml ${ec2Instance}:/home/ec2-user"
                        sh "ssh -o StrictHostKeyChecking=no ${ec2Instance} ${shellCmd}"
                    }
                }
            }
        }
        stage("commit version update") {
            steps {
                script {
                    withCredentials([usernamePassword(credentialsId: 'gitlab-creds',
                                                      passwordVariable: 'PASS',
                                                      usernameVariable: 'USER')]) {
                        sh 'git remote set-url origin https://$USER:$PASS@gitlab.com/nvastola/java-maven-app-aws.git'
                        sh 'git add .'
                        sh 'git commit -m "ci: version bump"'
                        sh 'git push origin HEAD:jenkins-jobs'
                    }
                }
            }
        }
    }
}

server-cmds.sh:

#!/usr/bin/env bash
export IMAGE=$1
docker-compose -f docker-compose.yaml up --detach
echo "Success"

docker-compose.yaml:

version: '3.8'
services:
  java-maven-app:
    image: ${IMAGE}
    ports:
      - 8080:8080
  postgres:
    image: postgres:15
    ports:
      - 5432:5432
    environment:
      - POSTGRES_PASSWORD=my-pwd

The Moment Everything Worked

I committed a small code change and pushed to GitLab. Jenkins detected it. The Stage View updated in real time:

Stage: increment version     - 4s  ✓
Stage: build app             - 7s  ✓
Stage: build image           - 5s  ✓
Stage: deploy                - 6s  ✓
Stage: commit version update - 2s  ✓

Finished: SUCCESS

All green. After twenty failed builds, three environment investigations, five different debugging issues, and six hours of work.

I SSH'd into EC2:

docker ps
CONTAINER ID   IMAGE                           STATUS         PORTS
a3f2c1b4d5e6   nvastola/demo-app:1.1.1-20     Up 2 minutes   0.0.0.0:8080->8080/tcp
7d8e9f0a1b2c   postgres:15                     Up 2 minutes   0.0.0.0:5432->5432/tcp

Both containers running. Version 1.1.1-20 matched the Jenkins build number exactly.

Opened browser: http://18.221.211.198:8080

The page loaded. Not localhost. Not manually deployed. Automatically built, versioned, pushed, and deployed — all from one commit. I checked Docker Hub: image nvastola/demo-app:1.1.1-20 present with fresh timestamp. Checked GitLab: new commit visible — ci: version bump. The pom.xml showed version 1.1.2, ready for the next build.

The Complete Flow

Code commit → Jenkins detects → Version increments → JAR builds → Image builds and pushes → Deploys both containers to EC2 → Version bump commits → Application accessible to users. One commit. Zero manual steps.

What I Learned

1. Environment-Specific Issues Are Real Production Concerns

The OpenSSL key format issue demonstrates this perfectly. Same key file, different behavior based on environment.

"Works on my machine" isn't good enough. You have to test in actual deployment environments. This is exactly why containerization matters — it ensures environment consistency.

2. Architecture Decisions Teach Production Thinking

Running Jenkins on the same EC2 as the app would have worked. But it would have skipped the fundamental lesson about separating build and runtime infrastructure. Understanding why systems are architected with separation matters more than just making code work.

3. Failures Fall Into Patterns

Every build failure across this project fit a category: naming, authentication, configuration, assumptions, or typos. Recognizing which category you're in narrows the debug surface immediately. When credentials are failing, check quote placement before re-entering passwords.

4. Small Scripts Beat Large Pipeline Files

Extracting deployment logic to server-cmds.sh means the logic is version-controlled, testable independently, and easy to extend without touching the pipeline definition. Jenkinsfile orchestrates. Scripts contain business logic. This separation scales.

5. Dynamic Versioning Enables Real Traceability

nvastola/demo-app:1.1.1-20 tells me the application version from pom.xml and the Jenkins build number. If users report a bug, I can trace to the exact build and code state that deployed it. Static tags like :latest don't provide this. In production incident response, that traceability matters.

Production Considerations

This works for learning. Production deployment needs more.

Secrets Management

That POSTGRES_PASSWORD=my-pwd hardcoded in docker-compose.yaml is in Git. Anyone with repository access sees it. Production approach: AWS Secrets Manager or HashiCorp Vault — inject secrets at runtime, never commit them.

Rollback Strategy

If this deployment breaks production, I'd need to find the previous working build, manually revert pom.xml, and retrigger. Better: keep previous containers running until the new version passes health checks (blue/green deployment), or implement automatic rollback on health check failure.

Health Checks and Verification

The pipeline succeeds if docker-compose starts containers — it doesn't verify the application actually responds. Production needs: HTTP health endpoint, database connectivity checks, automated smoke tests post-deployment, and automatic rollback if checks fail.

Multi-Environment Pipeline

This deploys to one EC2 instance. Real pipelines have dev (auto-deploy on every commit), staging (manual approval, production-like environment), and production (manual approval + passing smoke tests). Each environment has separate infrastructure, and promotion requires human sign-off.

Reflection

This project connects everything from projects 1–4:

This project adds: AWS infrastructure, remote deployment automation, dynamic versioning, the complete CI/CD workflow end to end.

What's Next

Container orchestration. Instead of running containers directly on EC2, I'll deploy to Kubernetes (AWS EKS).

Kubernetes handles things that become painful at scale: auto-scaling based on load, self-healing (restart failed containers automatically), load balancing across instances, rolling updates for zero-downtime deployments, and declarative configuration where you define the desired state and Kubernetes figures out how to get there.

I'm expecting networking complexity, YAML overload, storage challenges, and a steep learning curve. But I'm also seeing how all these pieces connect: Docker packages the apps, Jenkins builds and deploys them, Shared Libraries keep the automation DRY, and Kubernetes will handle the orchestration at scale.

GitLab Repository →