WindFonts CI/CD Pipeline Enhancement: Automated Build, Deployment Rollback, and Security Scanning

feiCode
,
1

Background

Previously, WindFonts (Wenfeng Font Library) deployments relied solely on an automated deployment script triggered by a Forgejo webhook, lacking CI validation, rollback mechanisms, and deployment notifications. This update completes the entire deployment pipeline.

Architecture Overview

Developer pushes to main
    │
    ├──→ Forgejo Actions (CI)
    │     └─ typecheck → lint → test → build
    │
    └──→ Forgejo Webhook → cravatar-prod
          └─ webhook.py (port 9800) → deploy-fonts-vault.sh
               ├─ git pull
               ├─ tag old image as fonts-vault:prev
               ├─ podman build → replace container
               ├─ health check (30s)
               │   ├─ success → clean up old image (keep prev)
               │   └─ failure → automatically roll back to prev
               └─ logs → /var/log/deploy-fonts-vault.log

New Additions

1. CI Workflow (.forgejo/workflows/ci.yml)

Automatically runs on push or PR to main:

  • TypeCheck: npm run typecheck — Type checking
  • Lint: npm run lint — Code style validation
  • Test: npm run test — Unit tests (using vitest)
  • Build: npm run build — Build verification

Runners execute on feicode-prod (forgejo-runner-01) with label ubuntu-latest; CI jobs use the node:20-alpine container.

2. Enhanced Deployment Script (/opt/deploy-hooks/deploy-fonts-vault.sh)

Compared to the previous version, new features include:

  • Rollback image retention: Automatically tags the current image before deployment: podman tag fonts-vault:latest fonts-vault:prev
  • Extended health check: Increased from 15s to 30s; now accepts HTTP status codes 200 and 302
  • Automatic rollback: On health check failure, automatically rolls back to fonts-vault:prev
  • Optimized image cleanup: Pruning retains the prev image, preventing accidental deletion of rollback candidates

3. Manual Rollback Script (/opt/deploy-hooks/rollback-fonts-vault.sh)

One-command rollback to the previous version:

ssh cravatar-prod '/opt/deploy-hooks/rollback-fonts-vault.sh'

4. Security Scan Fixes

The original Trivy (dependency vulnerability scanning) and Gitleaks (secret leakage detection) workflows used runs-on: docker, which was incompatible with the runner configuration and thus never executed. Fixed by changing to runs-on: ubuntu-latest.

Common Operations Commands

# View deployment logs
ssh cravatar-prod 'tail -30 /var/log/deploy-fonts-vault.log'

# Manually trigger deployment
ssh cravatar-prod '/opt/deploy-hooks/deploy-fonts-vault.sh'

# Manually roll back
ssh cravatar-prod '/opt/deploy-hooks/rollback-fonts-vault.sh'

# Check webhook service status
ssh cravatar-prod 'systemctl status deploy-webhook'

# Check CI execution status
curl -s 'https://feicode.com/api/v1/repos/Windfonts/fonts-vault/actions/tasks' -u 'feibisi:<token>'

Pitfalls Encountered

  • Runner label mismatch: The Forgejo runner is registered with labels ubuntu-latest/ubuntu-22.04, but the workflow specified runs-on: docker, causing jobs to remain unassigned. Resolution: Workflow labels must match runner labels.
  • Accidental deletion of rollback images by podman image prune: prune -f removes all dangling images—including the newly tagged prev. Solution: Before pruning, explicitly verify existence of fonts-vault:prev; restrict pruning scope using --filter until=24h.
  • Forgejo memory bloat: Encountered 502/504 errors during pushes due to Forgejo container memory usage ballooning to 12 GB. Restarting resolved it temporarily. Planned upgrade to Forgejo v11.0.10.

Scope of Impact

  • Repository: Windfonts/fonts-vault
  • Server: cravatar-prod (140.210.20.196)
  • Runner: feicode-prod (forgejo-runner-01)