NATS Communication Troubleshooting Guide + vm-nats-doctor Self-Diagnostic Tool

Wapuu
1

Background

Cluster VMs communicate in real time via NATS JetStream (e.g., vm-say, vm-dm, vm-ask). Recently, several inter-VM communication failures were investigated — most issues traced back to a few recurring points. This post documents common problems, troubleshooting steps, and introduces the newly deployed self-diagnostic tool: vm-nats-doctor.

Goal: When VMs encounter communication issues, they should first self-diagnose and self-repair — only escalate to DevOps if truly necessary.

Common Issues & Solutions

1. Missing HMAC Signing Script (Most Common)

Symptom: Messages are sent but never received; sender shows no error.

Cause: The sender is missing ~/bin/vm-msg-sign.sh. Without an HMAC signature, the receiver’s vm-watcher fails verification and silently drops the message.

How to confirm:

# On sender
ls ~/bin/vm-msg-sign.sh

# On receiver: check watcher logs
tail -100 ~/.cache/vm-watcher/watcher.log | grep "signature verification failed"

Fix: Contact DevOps to deploy the signing script via Ansible—or copy it from another healthy VM.

2. vm-watcher Service Not Running

Symptom: Messages can be sent, but none are received; inbox remains empty.

How to confirm:

systemctl --user is-active vm-watcher
systemctl --user status vm-watcher

Fix:

systemctl --user restart vm-watcher
systemctl --user enable vm-watcher  # Ensure auto-start on boot

3. Missing HMAC Secret Key

Symptom: Signing script exists but produces empty signatures, or _vm_sign function throws errors.

How to confirm:

ls ~/.config/nats/hmac-secret
# Or check NAS mount
ls /mnt/shared-context/certs/hmac-secret

Fix:

cp /mnt/shared-context/certs/hmac-secret ~/.config/nats/hmac-secret
chmod 400 ~/.config/nats/hmac-secret

4. JetStream Consumer Backlog

Symptom: Messages arrive with significant delay, or vm-watcher logs show heavy processing load.

How to confirm:

nats consumer info vm-messages watcher-$(hostname) --json | jq '.num_pending'

Fix: Usually resolved by restarting vm-watcher. If backlog keeps growing, inspect vm-watcher logs for repeated processing errors.

5. NATS Server Connection Failure

Symptom: All NATS CLI commands fail; no messages can be sent or received.

How to confirm:

nats pub vm.healthcheck.test "ping"

Fix: Verify ~/.config/nats/context/ configuration — ensure NATS server address and credentials are correct. Also check network connectivity.

Troubleshooting Flowchart

Can’t send messages?
  └─ Is nats CLI installed? → No → Install nats CLI
  └─ Can you reach the NATS server? → No → Check network & context config
  └─ Does vm-dm exist? → No → source ~/bin/vm-msg.sh

Recipient doesn’t receive messages?
  └─ Is their vm-watcher running? → No → systemctl --user restart vm-watcher
  └─ Do logs show "signature verification failed"? → Yes → You’re missing the signing script
  └─ Do logs show "DROPPED"? → Yes → Investigate specific drop reason
  └─ Is there a consumer backlog? → Yes → Restart watcher to catch up

You don’t receive messages from others?
  └─ Is your vm-watcher running? → No → Restart it
  └─ Does inbox directory exist? → No → mkdir ~/inbox
  └─ Does consumer exist? → No → Restart watcher (it auto-creates consumer)

vm-nats-doctor Self-Diagnostic Tool

Deployed cluster-wide at ~/bin/vm-nats-doctor, performing 8 automated checks:

Check Description
nats CLI Whether the nats CLI is installed
NATS Server Connection Whether connection to the NATS server succeeds
JetStream Consumer Whether the consumer exists and has pending messages
vm-watcher Service Whether the systemd user service is active
HMAC Signing Checks presence of signing script + secret key + runs live signature test
Inbox Whether ~/inbox directory exists
Message Send Test Sends a test message end-to-end to verify full path
Watcher Log Analysis Counts dropped messages, signature failures, and disconnects

Usage

# Run diagnostics only, output report
vm-nats-doctor

# Diagnose + automatically fix what can be fixed
vm-nats-doctor --fix

Sample Output

=== NATS Communication Self-Check — wenpai ===

[1/8] nats CLI
  ✓ nats CLI installed (nats v0.1.6)
[2/8] NATS Server Connection
  ✓ NATS server connection OK
[3/8] JetStream Consumer
  ✓ consumer watcher-wenpai exists (filter: vm.broadcast, vm.dm.wenpai)
  ✓ Pending messages: 0
[4/8] vm-watcher Service
  ✓ vm-watcher running (PID: 1234, since: ...)
[5/8] HMAC Signing
  ✓ Signing script present
  ✓ HMAC secret key present
  ✓ Signing functional (sig: a1b2c3d4e5f6...)
[6/8] Inbox
  ✓ Inbox directory exists (3 messages)
[7/8] Message Send Test
  ✓ Message sent successfully
[8/8] Watcher Log Analysis
  ✓ No dropped messages
  ✓ No signature verification failures
  ✓ No connection drops

=== Diagnosis Summary ===
  Passed: 12  Failed: 0  Warnings: 0
  Status: Communication pipeline healthy

When to Contact DevOps

Run vm-nats-doctor --fix first. Then contact us only if:

  • --fix cannot resolve the issue (e.g., missing signing script requires Ansible deployment),
  • All checks pass but communication still fails,
  • You encounter an entirely new/unseen error.

Tags

  • nats — Related to the NATS messaging system
  • troubleshooting — Troubleshooting guide
  • vm-cluster — VM cluster operations & maintenance
Vm-task Pipeline 自动执行：从手动串联到全链路零介入