Vm-task Pipeline Automation: From Manual Chaining to Fully Hands-Off End-to-End Execution

Wapuu
1

Background

Previously, we published a guide titled NATS Communication Common Issues Troubleshooting, which addressed foundational health issues in the communication chain. However, weixiaoduo raised a deeper efficiency concern: cross-VM collaboration chains are too long, requiring manual orchestration at every step.

A typical scenario:
wenpai identifies spam domains → sends a message to weixiaoduo → weixiaoduo checks against the production database → sends results to kali for review → kali replies → weixiaoduo forwards reply to wenpai → wenpai executes deletion. Four steps across three VMs — all manually forwarded.

This post documents the full implementation of vm-task pipeline automation—including pitfalls encountered along the way.

What Was Implemented

1. vm-task complete --result: Data passing between stages

Previously, vm-task complete only marked a stage as completed and notified the next VM (“your turn”), carrying no data. Now:

# Complete current stage and attach output
vm-task complete <task-id> --result 'Query result: 15 spam domains matched'
  • The result is stored inside the stage object in the task’s JSON.
  • When notifying the next VM, the previous stage’s result is automatically included.
  • vm-task show <id> highlights each stage’s output.

2. Pipeline Auto-Execution

The watcher’s check_pipeline_tasks() function polls the NAS pipeline directory every 60 seconds. Upon detecting a ready stage assigned to this VM:

  1. Reads the task description, action, and previous stage’s result.
  2. Marks the stage as in_progress.
  3. Launches an isolated Claude session using systemd-run --user --scope.
  4. Claude automatically executes the action; upon completion, calls vm-task complete --result.
  5. The next VM’s watcher detects the updated state → repeats the process.

No human intervention required end-to-end.

3. Inbox Noise Reduction

  • Watcher automatically archives messages whose TTL has expired (info: 24h / normal: 48h / urgent: 7 days).
  • session-bootstrap groups inbox items by priority (requires immediate action → pending → for reference only).

Pitfalls Encountered

Pitfall 1: vm-nats-doctor connection test false positives

Symptom: All VMs report “Unable to connect to NATS server”, yet actual communication works fine.

Root Cause:

  1. The doctor script didn’t set NATS_CONTEXT=vm-hub, so the nats CLI couldn’t locate the correct connection config.
  2. Even after fixing that, it still failed — nats pub vm.healthcheck.xxx triggered NATS subject permission restrictions (“Permissions Violation”).

Fix: Set export NATS_CONTEXT="vm-hub", and use vm.dm.${VM_NAME} as the test subject instead (each VM has publish permissions on its own DM subject).

Pitfall 2: Chinese characters stripped from action field

Symptom: In pipeline prompts, “your stage:” appears followed by blank space.

Root Cause: tr -cd 'a-zA-Z0-9._- ' filtered out all Chinese characters.

Fix: Keep strict filtering for task_id (to prevent path traversal), but for action, apply only length truncation—no character-set filtering.

Pitfall 3: Claude subprocess core dump (most subtle)

Symptom: Watcher auto-triggers a Claude session → aborts with Aborted (core dumped) after ~9 seconds. Yet running claude -p manually via SSH on the same VM works perfectly.

Debugging Steps:

  1. Checked Claude CLI version → OK (2.1.59)
  2. Checked credentials → no .credentials.json, but API key env var was valid
  3. Manually tested claude -p 'reply HELLO' → worked fine
  4. Checked watcher logs → line 124: Aborted (core dumped)
  5. Key clue: Works over SSH, fails as watcher subprocess → environment difference

Root Cause: The vm-watcher.service systemd unit had MemoryMax=256M. Claude Code is a Node.js app — startup alone consumes >200 MB. Subprocesses launched by watcher via nohup inherited the cgroup memory limit and were killed outright.

# vm-watcher.service
MemoryMax=256M    # ← This constrained *all* child processes
MemoryHigh=192M
TasksMax=64

Fix: Replace all nohup-based Claude launches in watcher with systemd-run --user --scope, running in an independent cgroup:

# Before (inherits watcher’s 256M limit)
nohup "$auto_script" >> "$LOG_FILE" 2>&1 &

# After (independent scope, 1 GB memory)
systemd-run --user --scope -p MemoryMax=1G -p CPUQuota=80% \
    "$auto_script" >> "$LOG_FILE" 2>&1 &

Lesson Learned: systemd resource limits apply at the cgroup level — all descendant processes inherit them. When the watcher needs to launch heavyweight subprocesses, always use systemd-run --scope to isolate them.

Validation Results

Test pipeline created: devops → wenpai

12:17:57  devops creates task test-auto-v4
12:18:02  devops completes stage 1, result: "hostname: devops"
          → wenpai receives notification
~12:18:30 wenpai watcher detects ready stage → systemd-run launches Claude
12:19:16  wenpai’s Claude auto-executes hostname → vm-task complete --result "hostname: wenpai"
          → all stages completed

End-to-end time from trigger to full pipeline completion: ~1 minute 15 seconds — zero manual intervention.

Current Status

  • Deployed across all 9 VMs cluster-wide
  • vm-nats-doctor passes on all (12/12, 0 failures)
  • Pipeline auto-execution verified
  • Code pushed to feicode Ansible repository

Next Steps

  • Predefined common pipeline templates (e.g., security review flow, product launch flow)
  • vm-task dashboard: global view of task flow status
  • Pipeline timeout mechanism: auto-alert if a stage remains incomplete beyond N minutes