Fei Publish: Automated Release Pipeline from Warehouse to Marketplace

feiCode
1

Background

The product update pipeline for the WeiXiaoDuo Mall (mall.weixiaoduo.com) has long been broken — 83% of files on dl1 were uploaded between 2017 and 2022, and the feicode repository contains 457 code-bearing repositories with zero releases. Manually updating a single product involves six error-prone and time-consuming steps: download source code → package into ZIP → upload to dl1 → create a GitHub/Gitea Release → update the WooCommerce download link → update the local YAML file.

After successfully running the full workflow manually for Elementor Pro as a pilot, we formalized it into the fei publish command.

Architecture

We adopted a hybrid delegation pattern:

  • ~/bin/fei (Bash) adds a lightweight cmd_publish() wrapper that execs into a Python script.
  • ~/Projects/mall-products/feicode/fei-publish.py (Python) implements the core logic.
  • Reuses existing data sources: repos.json (mapping 530 repositories), product.yaml (defining 371 products), and .env.secrets (containing credentials).

This keeps the fei CLI unified at the entry point while delegating complex operations — such as API calls, ZIP handling, and YAML read/write — to Python.

Single-Product Publishing Workflow (8 Steps)

fei publish <slug>
  1. Lookup Mappings — Locate org/repo in repos.json; retrieve product_id, current version, and zip_top_dir from product.yaml.
  2. Extract Version Number — Remotely fetch metadata via Gitea’s raw API (no clone required), trying in order of priority:
    • File specified by metadata.main_file → extract Version: header
    • <slug>.php → extract Version: header
    • style.css → extract Version: header (for themes)
    • readme.txt → extract Stable tag:
    • Filter out invalid values (trunk, stable, dev, master, main)
  3. Package ZIP — Download main.zip via Gitea’s archive API → extract → rename top-level directory to zip_top_dir → repackage (excluding .git*) → output {slug}-{version}.zip.
  4. Create Gitea Release — POST to /repos/{org}/{repo}/releases, tagging as v{version} and uploading the ZIP as an attachment. Skips creation if release already exists.
  5. Upload to dl1 — SCP the ZIP to weixiaoduo-prod:/www/wwwroot/dl1.weixiaoduo.com/{YYYY}/{MM}/.
  6. Update WooCommerce — PUT to /wp-json/wc/v3/products/{id}, replacing only download entries whose names contain “original” (e.g., “original plugin”) — updating their URL and version number — while preserving other downloads (e.g., Chinese language packs).
  7. Update Local product.yaml — Update upstream_version, download URLs/names, and date_modified.
  8. (Optional) Trigger Customer Notification — With --notify, invoke prod-query to enqueue a notification via wdpum.

Batch Operations

fei publish --check              # Concurrently scan version differences across all products (20 threads; completes 297 products in ~3 seconds)
fei publish --auto --limit 10    # Auto-publish the first 10 products with version mismatches
fei publish <slug> --dry-run     # Simulate execution without making real changes
fei publish <slug> --force       # Force publishing even if version hasn’t changed

Pitfalls Encountered

1. The "trunk" Trap in Version Extraction

Some plugins list Stable tag: trunk in readme.txt, which is not a valid version.
Solution: Maintain a blacklist of invalid version strings ({trunk, stable, dev, master, main}); skip matches and proceed to the next candidate file.

2. Connection Pool Exhaustion During Concurrent Scanning

--check uses 20 concurrent threads to call the Gitea API, but requests.Session defaults to a connection pool size of only 10 — causing many "Connection pool is full, discarding connection" warnings.
Solution: Mount a custom HTTPAdapter with pool_connections=20 and pool_maxsize=20.

3. Inconsistent ZIP Top-Level Directory Names

ZIPs downloaded via Gitea’s archive API use {repo}/ as the top-level directory, but WordPress plugins/themes require the top-level directory name to match the slug (some products specify a custom zip_top_dir, e.g., Astra’s child theme uses astra-wei).
Solution: After extraction, rename the top-level directory according to metadata.zip_top_dir in product.yaml.

4. Precise Replacement of WooCommerce Download Items

A product may have multiple downloads (e.g., original plugin, Chinese language pack, child theme). Updates must replace only entries whose names contain “original”, leaving others untouched. If no “original” entry exists, insert a new one.

Data Dependencies

repos.json (530 repositories)
    ↓ slug → org/repo
product.yaml (371 products)
    ↓ id, downloads, metadata.upstream_version, metadata.zip_top_dir
.env.secrets
    ↓ FEICODE_TOKEN, WC_CONSUMER_KEY/SECRET, WC_STORE_URL

Logging and Rollback

  • Each publication writes a JSON log to publish-log/{date}-{slug}.json.
  • Before updating WooCommerce, the current download list is backed up into the log.
  • Uploads to dl1 are append-only (new files go to new paths); failure during WooCommerce update does not delete already-uploaded files.

Next Steps

  • Explore triggering automatic publishing via Gitea webhooks upon new code pushes to repositories.
  • Schedule --check as a recurring cron job (e.g., daily), notifying stakeholders when updates are detected.
  • Introduce rate limiting for batch publishing to prevent overwhelming APIs with bursts of requests.