GlassBox Verify Shell

Open source transparency shell for Linux servers.

Verify Shell is a public verification system that proves a server has nothing to hide. Anyone with reviewer access can SSH into a hardened read-only shell and inspect live server configuration: firewall rules, running services, DNS, WireGuard status, file integrity reports, and more.

Built by Open Source Security Inc.

Live instance: glassbox.oss-ops.net Security reports: glassbox.oss-ops.net/verify/ Company: opensourcesecurity.net

What It Does

Verify Shell gives reviewers a restricted SSH shell (called the "verify shell") that allows read-only inspection of a live server. The shell whitelists 13 commands and blocks everything else. All sessions are logged and the server owner is alerted on every connection.

The goal: replace "trust us" with "verify it yourself."

Security Model

Verify Shell uses 9 layers of defense. Layers 1-3 are filter layers in the shell script itself. Layers 4-9 are kernel, daemon, and network enforcement that operate independently of the script.

1. inspect-shell command whitelist - 13 allowed commands only
2. inspect-shell injection filter - blocks shell operators, command substitution, and hex/escape sequences
3. inspect-shell protected path filter - blocks access to sensitive paths
4. Chroot jail - verify user is jailed to a minimal filesystem at /var/chroot/glassbox/. Sensitive files do not exist inside the jail.
5. AppArmor (enforce mode) - explicit deny rules on protected files, no network access
6. sshd ForceCommand - every SSH session is forced into inspect-shell
7. sshd ChrootDirectory - jail enforced at the SSH daemon level
8. UFW firewall - SSH only accessible via WireGuard tunnel
9. File permissions - protected files are 600 root:root

Red Team Status

Three confirmed bypasses to date. All three stopped at layer 3 of 9. No attack has reached the chroot, AppArmor, or any of the kernel-enforced layers below the shell. No bounty file has been read.

Round	Finding	Layer Bypassed	Layer That Held
1	Skip: /proc/thread-self filter bypass via symlink path	3 (protected path filter)	4 (chroot)
2	Tux: AIDE meta.json reporting falsified (transparency attack, not isolation)	N/A - dashboard claim	N/A - isolation untouched
3	Skip: hex-encoded grep filter bypass	3 (protected path filter)	4 (chroot)

This is the design working as intended. Layers 1-3 are bash regex filters in one script; we expect them to fail eventually under sustained attack. Layers 4-9 are the layers we are actually counting on, and so far they have held in every confirmed bypass.

We want more attacks. Three rounds is not enough. The bounty is open. If you can read any of the three protected files (see Bug Bounty below), the cash and the credit are yours, and the patch goes into the next release with your name on it.

Bug Bounty

Verify Shell runs a $1,000 USD bounty challenge. Three files are protected on the server. Read any one of them through the verify shell and you win.

Bounty targets:

• /etc/wireguard/wg0.conf
• /root/glassbox-customers.csv
• /root/.ssh/id_rsa

To request reviewer access, email glassbox@opensourcesecurity.net with the subject "Verify Shell reviewer access."

Allowed Commands in the Verify Shell

ls [path]                  Directory listing
cat [file]                 File contents
head [file]                First lines of file
tail [file]                Last lines of file
grep [pattern] [file]      Search file contents
wc [file]                  Word/line count
wg show                    WireGuard status
systemctl status [service] Service status
iptables -L                Firewall rules
ipset list                 IP blocklist contents
uname -a                   Kernel version
help                       Show this help
exit / quit / q            End session

Note: uptime, df, and free were removed from the whitelist after the Attack 1 patch unmounted /proc from the chroot entirely. See Threat Model and Attack 1 writeup below for the reasoning.

Philosophy

Verify Shell exists because VPN providers, hosting companies, and infrastructure operators should not ask customers to trust them. They should prove they have nothing to hide. Verify Shell makes that proof possible.

Requirements

• Ubuntu 24.04 LTS (tested)
• WireGuard
• OpenSSH
• AppArmor
• Bash

Optional (for security reporting): AIDE, Lynis, rkhunter

Files in This Repo

File	Server Path	Purpose
inspect-shell	/usr/local/bin/inspect-shell	The verify shell script
usr.local.bin.inspect-shell	/etc/apparmor.d/usr.local.bin.inspect-shell	AppArmor profile
glassbox-monitor.sh	/opt/glassbox-scripts/glassbox-monitor.sh	External session alert service
generate-security-reports.sh	/opt/glassbox-scripts/generate-security-reports.sh	Weekly security report generator
sshd_config	/etc/ssh/sshd_config	SSH daemon configuration

License

Open source. BSD-2 Licensed.

---

Deep Detail (for reviewers)

The remainder of this README is the security depth a serious reviewer needs to assess the design. It is meant to be read top-to-bottom only by someone evaluating the system; for everyone else the top of the README is sufficient.

Threat Model

The reviewer is an authenticated adversary, not a public-internet attacker. UFW and sshd handle public-internet defense. Verify Shell's job begins after both of those gates have been cleared.

What the attacker has:

• The shared glassbox-reviewer SSH private key (intentionally public, distributed in the repo)
• A WireGuard peer config from us, putting them on the 10.13.37.0/24 tunnel with an assigned reviewer IP (10.13.37.240 through 10.13.37.254)
• A working SSH session into the verify user
• Full source code of inspect-shell (readable from inside the shell itself via cat /usr/local/bin/inspect-shell)
• Knowledge of the bounty target paths
• Unlimited time and unlimited attempts

What the attacker does NOT have:

• Root on the server
• A real shell (sshd ForceCommand routes them into inspect-shell)
• A writable filesystem (mounts inside the chroot are all ro except the session log)
• Network egress from inside the jail (no curl, no nc, AppArmor has no network rules)
• Visibility outside the chroot (no /etc/wireguard/, no /root/, no /etc/shadow inside the jail)
• Ability to invoke other users' processes (hidepid=invisible originally; /proc later unmounted entirely)
• Ability to escalate (no SUID binaries, no sudo, no su)

Win condition: read any of /etc/wireguard/wg0.conf, /root/glassbox-customers.csv, or /root/.ssh/id_rsa.

Explicitly out of scope:

• Compromise of the host kernel
• Compromise of the WireGuard handshake itself
• Physical or IPMI access
• Social engineering of OSS staff
• Side-channel attacks on the underlying hardware (Psychz shared tenant)

Layers 1-3 Share One Codebase

Layers 1-3 (whitelist, injection filter, protected-path filter) all live in /usr/local/bin/inspect-shell - one bash script, roughly 258 lines. A single logic flaw in that script bypasses all three at once, which is exactly what Attack 3 (hex-encoded grep) proved: one regex gap took out the whole filter trio.

This is by design, not a flaw. Keeping the filter logic in one auditable file is more reviewable than three separate components. But reviewers should know that layers 4-9 (chroot, AppArmor, sshd ForceCommand, sshd ChrootDirectory, UFW, file permissions) are the layers that have actually held when the filter has been beaten.

High-Risk Whitelist Entries

The 13-command whitelist is not uniformly risky. Some commands have been the persistent attack surface and deserve naming.

cat, head, tail, grep accept arbitrary file paths as arguments. Attack 1 (/proc/thread-self) and Attack 3 (hex-encoded grep) both came through this class. Every filter bypass to date has been about getting one of these four to read something the filter did not anticipate.

grep specifically was the source of two of three confirmed bypasses. It accepts patterns with backslash sequences (\x, \NNN octal) that the bash-level filter does not see decoded. The hex bypass is patched; octal is flagged in the Thread 3 summary as untested.

systemctl status [service] is flagged as un-red-teamed. systemctl status historically pipes through a pager (less) by default, and less has a !command shell-escape. We pass --no-pager, but no red team thread has explicitly probed whether a crafted service name argument or env-var manipulation can bring the pager back. Open item for the next red team thread.

The lower-risk commands (wg show, iptables -L, ipset list, uname -a, help, exit) take either no arguments or fixed flags, and have not produced any bypasses.

AppArmor and Chroot Specifics

Chroot: /var/chroot/glassbox/. Mounted in (all ro): bin, lib, lib64, usr/lib, etc/unbound, etc/nginx, etc/fail2ban, etc/iptables, var/www. NOT mounted in: /etc/wireguard/, /root/, /opt/, /etc/letsencrypt/.

The /proc mount was originally bind-mounted with hidepid=invisible. After the /proc/thread-self finding in Attack 1, it was unmounted entirely and removed from fstab. This forced removal of uptime, df, and free from the whitelist since those depend on /proc. The tradeoff was deliberate: losing three low-value status commands eliminated the entire /proc attack surface from inside the jail.

AppArmor profile (/etc/apparmor.d/usr.local.bin.inspect-shell) - in enforce mode.

What it allows: read on the chroot's bin, lib, lib64, usr/lib, etc, opt, var/www; execute on bin, usr/bin, usr/sbin; rw on /dev/null and /dev/tty; write to /var/log/glassbox-sessions.log only.

What it explicitly denies (overrides any allow):

deny /etc/wireguard/ r,
deny /etc/wireguard/** rwlx,
deny /root/** rwlx,
deny /root/.ssh/** rwlx,
deny /root/glassbox-customers.csv rwlx,
deny /etc/shadow r,
deny /etc/shadow w,
deny /etc/shadow x,
deny /var/chroot/glassbox/bin/* w,
deny /var/chroot/glassbox/usr/bin/* w,
deny /var/chroot/glassbox/usr/sbin/* w,
deny /var/chroot/glassbox/etc/** w,

No network rules. The original network inet stream / network inet dgram rules were removed after Thread 1 - the ntfy alert that needed them was moved out of the shell into glassbox-monitor.service, which runs as root outside the chroot. curl was also removed from the chroot binary tree. The verify user has zero network-capable binaries and zero AppArmor permission to use the network even if one appeared.

The @{PROC} AppArmor variable is not used in the profile - paths are all literal. The kernel running this is Linux 6.8.0 on Ubuntu 24.04. The AppArmor network_v8 feature file is not present, which means IP-level network filtering in the profile is not available even if we wanted it - another reason network was removed entirely instead of narrowed.

Verifiability Honesty Gap

A verify-shell user cannot independently verify the verify shell itself. They can read the source of inspect-shell, and should. But from inside the shell they cannot run aa-status to confirm AppArmor is actually enforcing, cannot run mount or findmnt to confirm the chroot is set up as described, and cannot confirm the kernel binary on disk matches what uname -a reports. They are trusting that the layers underneath the shell are configured the way we say they are.

Three mitigations partially close this gap, none of them fully:

1. AIDE report at https://glassbox.oss-ops.net/verify/aide-report.txt - a hash manifest of every file on the server, regenerated weekly. A reviewer can confirm the kernel image, AppArmor profile, and inspect-shell script all match their published hashes.
2. Server snapshot at https://glassbox.oss-ops.net/verify/server-snapshot.txt - includes aa-status, mount, systemctl list-units, wg show, iptables -L, ufw status output. Regenerated weekly.
3. Lynis and rkhunter reports at the same URL.

All three of these are produced by us on the server we are asking you to verify. Anything we generate, we can fake. The verify shell exists because of this problem - to give reviewers a live, interactive view that is harder to fake than a static report - but the verify shell itself is still configured by us.

This is the unresolved tension at the heart of Glass Box v1, and is the main reason v2 is being designed around showing raw kernel state (via standard tools with fixed arguments) rather than around a hardened custom shell. See the v2 design doc.

WireGuard Pre-Credentialing

Bounty challengers do not start from the open internet. They start from inside our WireGuard tunnel, on 10.13.37.0/24, with an SSH key we issued them and a reviewer IP we assigned (10.13.37.240 through 10.13.37.254). The public IP (108.181.0.99) does not accept SSH at all - UFW blocks port 22 except on wg0.

One gate has already been cleared by the time the attack begins: the attacker has been authenticated as a peer at the network layer and as a key holder at the SSH layer. Glass Box's claim is that even with those two gates cleared, the bounty files remain unreachable. It is not a claim that the server resists arbitrary unauthenticated attack from the public internet - that is UFW's and sshd's job, not Glass Box's.

Expanded Attack Writeups

Attack 1 - /proc/thread-self Filter Bypass

Thread 1, Claude Opus 4.6

Category: Isolation attack (filter layer bypass). Stopped at layer 4 (chroot).

Vector: check_protected_path() blocked /proc/self and /proc/[0-9] but not the sibling path /proc/thread-self. The attacker discovered this by listing /proc, noticing thread-self as an entry, and trying it on a hunch.

What it leaked: Full process environ (SSH_CONNECTION, PATH, USER, HOME), open file descriptors, memory maps, mount info, the host TCP socket table via /proc/thread-self/net/tcp, process status including PID, PPid, UID, capabilities, and seccomp state.

Why it mattered (and didn't): It bypassed three filter layers in one step. But every bounty target was outside the chroot - /proc/thread-self/root resolved back to the chroot root, so escape was contained.

Patch: Added a single line to check_protected_path():

[[ "$input_lower" =~ /proc/thread ]] || \

Broader than thread-self specifically - also catches any future /proc/thread-* variant the kernel might introduce. Subsequent decision: unmount /proc from the chroot entirely, since uptime, df, and free were the only consumers and they were dropped from the whitelist. After the unmount, the entire /proc attack surface ceased to exist inside the jail.

Attack 2 - meta.json AIDE Status Falsified

Thread 2, Tux + Claude Sonnet 4.6

Category: Transparency attack, not isolation attack. The shell, chroot, and AppArmor were untouched. What failed was the dashboard's claim about the server state, not the server state itself. This distinction matters: a system can be perfectly hardened and still lie about being hardened, and that lie is its own class of vulnerability.

Vector: Tux noticed that meta.json (the dashboard's status summary) was reporting aide_status: CLEAN while the actual AIDE report showed 14 changed binaries. If the dashboard lies, the whole transparency claim is undermined.

Root cause: generate-security-reports.sh was running aide --check without a --config flag. AIDE errored out silently with no config, the script's grep -qE '^(New|Removed|Changed):' "${OUTDIR}/aide-report.txt" found nothing, and the script defaulted to CLEAN. Compounding bug: AIDE was spinning indefinitely on /var/chroot/glassbox/proc/ pseudo-files because the baseline did not exclude them.

The 14 "changed binaries" themselves: Not a compromise. sshd, sudo, curl, systemctl, and others had been updated by apt upgrade between server build and Tux's scan. AIDE baseline was stale. Tux's Claude initially read it as a rootkit; we corrected that. The distinction between "AIDE is reporting changes" and "the server has been compromised" is one a careful reviewer needs to make every time, and we want it documented here that we made it correctly.

Patch: Added --config /etc/aide/aide.conf to the AIDE invocation. Updated the grep pattern to AIDE 0.18 output format. Added /var/chroot/glassbox/proc to the AIDE exclusion list. Rebuilt the baseline (aide --init then mv aide.db.new aide.db). Same baseline refresh applied to rkhunter (rkhunter --propupd). meta.json now reports accurately.

Attack 3 - Hex-Encoded Grep Filter Bypass

Thread 3, Claude Opus 4.6

Category: Isolation attack (filter layer bypass). Stopped at layer 4 (chroot).

Vector: The filter checks input for literal substrings like shadow, wg0.conf, /proc/self. But grep -rP accepts Perl-compatible patterns with \x hex escapes that the bash-level filter never sees decoded. The grep binary interprets the hex internally after the filter has already passed the input through.

Proof:

• grep -rP "\x73\x68\x61\x64\x6f\x77" /etc/ (hex for "shadow") - filter did not catch it
• grep -rP "\x77\x67\x30\x2e\x63\x6f\x6e\x66" /etc/ (hex for "wg0.conf") - filter did not catch it
• grep -rP "\x2f\x70\x72\x6f\x63\x2f\x73\x65\x6c\x66" /proc/ (hex for "/proc/self") - bypassed the /proc/self block, grep recursed into /proc and leaked PID directories, AppArmor settings, BPF JIT status, interface names including wg0

Why it mattered (and didn't): Same shape as Attack 1 - bypassed the filter trio. Bounty files remained out of reach because they are not in the chroot. But if any sensitive content were ever mounted in, this bypass would search it without triggering alerts.

Patch: Added \x detection to check_injection():

|| [[ "$input" =~ \\x ]]

Any input containing \x is now flagged as an injection attempt and denied. Octal encoding (\NNN) was identified as the same bypass class and flagged in the Thread 3 summary as untested - open item for the next red team round.

Also patched in the same session (proactive hardening, not in response to a confirmed attack):

• IFS=$' \t\n' set explicitly at the top of inspect-shell to prevent SSH-environment IFS manipulation from changing how read -ra parts tokenizes input
• trap '' SIGINT SIGTSTP SIGQUIT added to ignore Ctrl+C, Ctrl+Z, Ctrl+\ - closes the possibility of killing the shell and dropping to a parent process

Open Items for Future Red Team Rounds

• systemctl status pager-escape via crafted service name or env-var manipulation
• \NNN octal encoding as a sibling bypass class to the patched \x hex bypass
• Any other backslash escape sequence that grep, sed, awk, or other text tools decode after the bash filter has cleared the input
• Behavior of the AppArmor profile when the network_v8 feature file becomes available in a future kernel

If you find something not on this list, we want to know about it. Email glassbox@opensourcesecurity.net.

---

Open Source Security Inc. - opensourcesecurity.net