Bot Fleet & VPS Operations

Verified runtime snapshot from 2026-08-08 UTC. This page ties the Discord bot fleet to the services it depends on and records audit findings that should not be mistaken for completed fixes.

📌 Snapshot, not configuration management

Service state and ports can change after this audit. Confirm with systemctl, ss, health endpoints, and current unit/nginx files before an operation. Do not copy secrets or raw Discord content into this public wiki.

Live bot fleet

Ten instances of /etc/systemd/system/crossworlds-bot@.service are active and run the same /opt/crossworlds-bot/engine.js as user ubuntu:

Unit keyDisplay personaSurface / specialtyElevated feature flags
muffindevHelpCustomer-facing #helpdesk; only configured channel ID; website chatdeveloper DM, alert webhook 127.0.0.1:3002, web chat 127.0.0.1:3003, Kuma heartbeat
dorisDorisCustomer-facing development/testing counterpartDiscord membership verification 127.0.0.1:3004
nadiaMuffinDevInternal CTO/engineering personaexec knowledge, code grounding, /discuss, review-only proposals, /codex-task, voice
garthGarthInternal strategyexec knowledge + read-only code grounding
devinDevinInternal delivery/operationsexec knowledge + read-only code grounding
burtBurtInternal brand/experienceexec knowledge + read-only code grounding
jaylenJaylenInternal AI/dataexec knowledge + read-only code grounding
piperPiperInternal productexec knowledge + read-only code grounding
ajrunAjrunInternal platformexec knowledge + read-only code grounding
samSamInternal security/riskexec knowledge + read-only code grounding

dennis.json and .env.dennis exist, but crossworlds-bot@dennis is disabled and inactive with no journal history. Dennis is staged configuration, not a live dispatcher. Older text saying only Help and Doris are live is obsolete; text saying Dennis currently orchestrates work is also premature.

All active persona configs currently use mentionOnly: true, riffOnBanter: false, and no wikiFocusPages restriction. Internal execs therefore retrieve from the whole game wiki plus nine exec-knowledge pages; customer personas retrieve from the public-safe candidate set. The process logs correctly show 43 Markdown pages for public personas and 52 entries for exec personas (43 wiki + 9 exec knowledge), not two different deployed wikis.

Message, retrieval, and isolation flow

  1. Messages authored by any bot are discarded immediately. Direct mentions and bounded direct continuations can proceed; all personas are mention-only.
  2. Per-message lock files and a shared cooldown state prevent two personas from racing or feeding one another. A hot-reloaded tuning.json provides a global kill switch and a 20 replies / 10 minute per-channel circuit breaker.
  3. Haiku chooses up to three relevant wiki titles. Pages are read in full; the old 1,500-character truncation no longer applies.
  4. Exec personas merge nine internal knowledge files and may retrieve up to three current source files from auth staging, game source, bot source, and dashboard source. Ordinary answers remain read-only prompt context.
  5. Project-channel knowledge is loaded per request and per persona to avoid cross-project leakage. Learned notes append to that project's persona-specific file and are capped to 200 characters.
  6. Customer personas exclude internal architecture, auth, database, deploy, dashboard, tooling, issue, implementation, and bot pages in code before semantic selection.
⚠️ New internal page exclusion

Historical Context & Decision Register and this page were added after the hard-coded customer exclusion list was written. Until the code list includes both titles, they are eligible for public retrieval despite containing operational detail. Treat this as a security follow-up, not as protection supplied by the prompt.

Write-capable paths

  • Normal exec answers only read code. Nadia/MuffinDev is the sole persona configured for proposal and Codex-task features.
  • /discuss is bounded, uses file locks, and can turn an explicitly requested target into a proposal only in the pinned MuffinDev guild.
  • /codex-task is owner-gated and calls a shared-secret worker on 127.0.0.1:3021.
  • The worker fixes the repository path to the dedicated /game/Development/Crossworlds-codex-worktree, allows one task at a time, rejects dirty starting state, uses isolated branches, compiles through Unity, and can commit. It never merges, pushes, deploys, restarts production, or writes a database.
  • Task result detail persists under codex-pipeline/runs for 30 days. The unauthenticated worker health endpoint exposes only status, busy, and current task ID; all task routes require the worker secret.

The worker is active but its unit is disabled, meaning a normal reboot will not start it automatically. Its Git working tree also had local modifications to server.js and format-for-discord.js at audit time. Preserve and review those changes before reinstalling, pulling, or resetting anything.

Bot audit findings

SeverityFindingEvidence / impact
HighShared /opt/crossworlds-bot/.env is mode 664; persona/auth/dashboard env files are correctly 600.Any local user can read the shared secrets file. Tighten ownership/mode after identifying which values remain in it; rotate any historically exposed credentials.
HighNew operational wiki pages are not in CUSTOMER_EXCLUDED_WIKI_PAGES.Public Help/Doris can semantically retrieve them. Add both titles and regression-test customer isolation.
MediumBot package has no automated tests (npm test intentionally fails); worker package only has start.Syntax checks pass, but routing, privacy boundaries, locks, discussion flow, chunking, tickets, voice budgets, and proposal approval lack a repeatable test suite.
MediumWorker is active but disabled./codex-task disappears after reboot unless started manually; either enable deliberately or document intentional on-demand start.
MediumSource-of-truth drift: bot code/config is live under /opt, worker has uncommitted changes, and most bot code is not in a visible project Git repository.Rollback, peer review, and provenance are weaker than for the Unity repo. Capture the bot/dashboard stack in controlled version history without committing secrets or state.
MediumloadWiki() overwrites/adds entries but does not clear wikiPages first.A deleted/renamed page can remain retrievable in a long-running bot until restart. Build a fresh map and atomically replace it on reload.
MediumWebsite chat rate limiting trusts the leftmost X-Forwarded-For value.Safe only when nginx overwrites that header and direct port 3003 stays loopback-only. Verify proxy header policy; never expose 3003 directly.
MediumDiscord username verification has no shared-secret authentication of its own and can return a Discord user ID.It is loopback-only, so the caller boundary is nginx/local services. Keep 3004 private, pin DISCORD_GUILD_ID, rate-limit the public caller, and consider authenticated service-to-service requests.
Lowbot.js, temporary test scripts, old .env.dorispringle, lock/state files, and discussion artifacts remain beside live source.They increase operator ambiguity and indexing risk. The code index skips _tmp-, state, backup, and run directories, but operational cleanup/archival is still warranted.
Low/health returns current worker task ID without authentication.Loopback-only today; keep it that way or remove the task identifier if proxied publicly.

Verified VPS runtime

ServiceVerified state / bind
Production gamecrossworlds-server, active; /game/server/CrossWords.x86_64; UDP 7777
Production authcrossworlds-auth, active; TCP 3000; /api/health reports DB connected
Staging gamecrossworlds-server-staging, active; /game-staging/CrossWords.x86_64; UDP 7778
Staging authcrossworlds-auth-staging, active; TCP 3010; isolated DB health connected
Dashboardcrossworlds-dashboard, Node on 127.0.0.1:4001; nginx TLS on public 4000
Website/wikinginx on public 80/443; Markdown and _site under /var/www/crossworlds/wiki
Realtimerod-realtime, active on 127.0.0.1:5000
MonitoringUptime Kuma, active and publicly allowed on 3001
DataMySQL 127.0.0.1:3306, PostgreSQL 127.0.0.1:5432, Redis loopback, SpacetimeDB 127.0.0.1:3500

Brain map service-context audit (2026-08-08)

The MuffinCore 3D map previously mixed live process facts with descriptions too short to explain architectural meaning. ROD Realtime was the clearest example: the process was correctly shown as active, but the inspector did not explain that ROD means Rate of Decay, the former project name, or distinguish its Socket.io channel from the authoritative Unity/Mirror game server.

The curated service/database nodes now expose separate right-panel fields for Naming context, What it does, Important caveat, and Depends on, followed by live systemd status and raw configuration metadata. Coverage includes the bot engine, dashboard, production/staging auth and game servers, ROD Realtime, Kuma, nginx, agent bridge, Codex worker, wiki webhook, SMS worker, SpacetimeDB, MySQL, PostgreSQL, and Redis.

Important distinctions recorded in the map:

  • rod-realtime is a legacy technical name, not a second/currently branded game. It is the Socket.io companion on port 5000; Mirror/KCP on UDP 7777 remains authoritative gameplay.
  • SpacetimeDB is installed, running, and proxied, but is not proven to participate in the current documented game persistence path.
  • MySQL is canonical Crossworlds persistence. PostgreSQL and Redis are running VPS capabilities, not evidence that character data or durable brain memory lives in them.
  • A green systemd/Kuma status proves process availability, not correct multiplayer gameplay, persistence, scene travel, or build compatibility.
  • Staging services must not be presented as production state, even though they share the host and may share the MySQL daemon through separate schemas/configuration.
  • Generated Codex commits are reviewable proposals. The agent bridge and developer identities do not receive implicit production authority merely because their services are active.

This context is curated in MuffinCore's catalog and is re-seeded into its local SQLite brain on service startup. Future service additions are incomplete until these architectural context fields are populated; a name, port, and active badge alone are insufficient. | MuffinCore | dashboard 127.0.0.1:4100; reasoning worker on Unix socket; hourly index timer active | | Codex lane | worker 127.0.0.1:3021, active/healthy, idle at audit time |

Host snapshot: 92 GiB RAM, about 3.5 GiB used and 87 GiB available; no swap; root/game share a 388 GiB filesystem at about 50% usage (194 GiB used, 195 GiB available).

VPS audit findings and drift

  1. The old root CROSSWORLDS.md runbook is dangerously stale. It names /opt/rod-auth, /opt/rod-dashboard, /game/Builds, crossworlds.service, direct HTTP/basic-auth dashboard access, and obsolete deploy commands. Current paths are /opt/crossworlds-auth, /opt/crossworlds-dashboard, and /game/server; current game unit is crossworlds-server; dashboard Node is internal behind nginx TLS/session auth.
  2. Historical handoffs are snapshots, not live runbooks. DEV_ENV_VPS_SETUP.md says staging TCP 3010 is still closed and UDP 7778 was reserved/free in one table, while the current firewall allows both and both services are active. The 2026-07-27 wiki handoff says 32 pages; the current build has 44 after this page.
  3. Public exposure is broader than the normal HTTPS surface. UFW permits TCP 3000, 3001, 3010, and 4000 plus UDP 7777/7778 from anywhere, for IPv4 and IPv6. This may be intentional for game clients/staging, but auth and monitoring should be explicitly justified, TLS-protected or restricted, and reviewed regularly.
  4. Monitoring contains an obsolete label. Kuma repeatedly warns about “Game Server — Unity Dedicated :55000”, while the verified game ports are 7777 and 7778. Confirm whether this push monitor is intentionally logical rather than port-based; otherwise rename/reconfigure it to avoid false incident signals.
  5. Staging logs show Unknown arg: -batchmode, -nographics, and -authurl at process start. The server nevertheless binds 7778 and remains active. Verify that RodNetworkManager actually consumes Environment.GetCommandLineArgs() before Unity reports unknown flags and that staging validates JWTs against 3010; do not infer correctness solely from process uptime.
  6. Dashboard service has several persistent tail -f/journalctl -f child processes. This may be active websocket log viewers, but stale viewers can accumulate. Confirm cleanup on client disconnect and impose a bounded subscriber count.
  7. MuffinCore reasoning logged a Claude exit 143. One terminated request is not proof of a recurring fault; correlate request timeout/cancellation behavior before changing the service.
  8. No swap is configured. Current memory headroom is enormous, so this is not an incident, but the operational choice should be explicit for a host running Unity builds and model workers.

Safe verification commands

systemctl list-units 'crossworlds-bot@*' --all --no-pager
systemctl status crossworlds-server crossworlds-auth crossworlds-server-staging \
  crossworlds-auth-staging crossworlds-dashboard crossworlds-codex-worker --no-pager
ss -lntup
curl -sS http://127.0.0.1:3000/api/health
curl -sS http://127.0.0.1:3010/api/health
curl -sS http://127.0.0.1:3021/health
journalctl -u 'crossworlds-bot@nadia' -n 50 --no-pager

Never print .env contents, tokens, dashboard session secrets, worker secrets, database passwords, or raw Discord exports into logs or tickets.


Related: AI Agent Tooling · Help Bot · Architecture & Topology · Build & Deploy · Admin Dashboard · Historical Context & Decision Register