NSSPC System Adminsitrator Experiences

Context

Back in April, a senior from NASA (Network And System Administration Team) asked if I’d like to join the Computer Center’s NSSPC Project. The plan: host the online preliminary round in mid-September and then fly to Kuala Lumpur for the on-site finals. Honestly, the first thing I noticed was “sponsored trip back to Malaysia” — which lead to instant yes XDD.

NTU Team

The contest was organized by the Malaysian Alumni Association, in collaboration with National Taiwan University and UTAR. NTU’s responsibility was the programming contest.

Our team split into two groups:

• Judging Team: wrote problems and reviewed translations.
• Systems Team (my team): built the prelim and final environments.

Inside Systems, we further split into Infra and Services. I’m not the “image laptops & wire switches” kind of person, and I was already the Web Administration Team lead in NASA, so I joined Services (web services, deployments, etc.). A few days later, they asked if I could lead the Services group… and welp, here we are XDD.

Web Architecture

In previous years, everything was set up directly on VMs. This time, we decided to go with Kubernetes. VM-only setups get messy fast and are painful to reproduce; with Kubernetes, we can turn multiple services into a few config files, then deploy with one command. Plus, we get built-in knobs for high availability.

To keep prelim and finals similar (and reduce cognitive load), we used the same Kubernetes architecture for both.

Why Kubernetes instead of VM-only?

• Declarative, repeatable deploys: everything as YAML/Helm; “git pull + apply” beats “SSH + snowflake servers.”
• Scale & self-healing: if a pod dies, it restarts; if load spikes, replicas scale. On VMs, you babysit processes.
• Service discovery built-in: no hand-crafted /etc/hosts spaghetti; Services/Ingress handle routing cleanly.
• Rolling updates / rollbacks: ship changes gradually, roll back in one command if something breaks.
• Portable between venues: prelim on campus VMs, finals on three laptops acting as masters — same manifests.
• Resource isolation: limits/requests help keep noisy neighbors from taking everything down.
• Observability: standardize on Prometheus/Grafana operators, sidecars, and logs with consistent labels.
• Team collaboration: infra and services speak the same YAML; fewer “it works on my VM” moments.
• Disaster tolerance: restarting pods across nodes is way easier than resurrecting a pet VM with mystery configs.

The Pain of Deploying DOMJudge

MySQL NDB Said “Nope” QQ

We used DOMJudge 8.3.1 (a newer release) and immediately ran into weirdness. First, the database layer: DOMJudge doesn’t exactly shout “No NDB,” so we tried MySQL NDB first. Everything “deployed,” but the app wouldn’t work. After stepping through logs and behaviors, we realized DOMJudge doesn’t support NDB. Switched to InnoDB, and finally the app could talk to the DB… or so we thought.

Broken Startup Script…

DOMJudge still couldn’t communicate properly under Kubernetes. More debugging later, we discovered the init script dj_setup_database creates the DB user as domjudge@localhost by default. That’s fine for a single box, but not for a distributed setup. Fix: I patched the DOMJudge Docker image to create domjudge@% instead. No more permissions rabbit hole. The following is the patched dj_setup_database script:

#!/bin/sh
# Generated from 'dj_setup_database.in' on Tue Jun 10 06:09:00 UTC 2025.

# This script allows one to perform DOMjudge database setup actions.
# See usage below for possible commands and options.

set -e

BINDIR="/opt/domjudge/domserver/bin"
ETCDIR="/opt/domjudge/domserver/etc"
WEBAPPDIR="/opt/domjudge/domserver/webapp"
EXAMPLEPROBDIR="/opt/domjudge/domserver/example_problems"
DATABASEDUMPDIR="/opt/domjudge/domserver/db-dumps"

PASSWDFILE="$ETCDIR/dbpasswords.secret"

verbose()
{
	if [ -z "$QUIET" ]; then
		echo "$@"
	fi
}

usage() {
	cat <<EOF
Usage: $0 [option]... <command> [argument]

Commands:
  status           check database installation status
  genpass          generate DB,API,Symfony,admin password files
  create-db-users  create (empty) database and users
  install          create database, example contest and users if not existing
  bare-install     create database, setup defaults if not existing
  uninstall        remove database users and database, INCLUDING ALL DATA!
  install-examples install examples only
  install-loadtest configure for load testing. WARNING: CREATES A LOT OF EXTRA ITEMS!
  upgrade          upgrade MySQL database schema to current version
  dump [filename]  backup the current database to file (without .sql.gz suffix)
  load [filename]  load a backup from file (without .sql.gz suffix), REMOVES ALL PREVIOUS DATA!

Options:
  -u <user>  connect to MySQL with DB admin <user>
  -p <pass>  use password <pass> for DB admin user
  -q         be (mostly) quiet
  -r         read DB admin password from prompt
  -s         connect via local socket (do not specify port)

Note: you may want to store your credentials in ~/.my.cnf in order to
not have to pass any of the options above.

EOF
}

mysql_options()
{
	local user pass

	# shellcheck disable=SC2153
	if [ -n "$DBUSER" ]; then
		user="-u $DBUSER"
	else
		user="${DBA_USER:+-u ${DBA_USER}}"
	fi
	# shellcheck disable=SC2153
	if [ -n "$PASSWD" ]; then
		pass="-p$PASSWD"
	else
		[ -n "$PROMPT_PASSWD" ] && pass="-p"
		[ -n "$DBA_PASSWD" ]    && pass="-p$DBA_PASSWD"
	fi

	[ -z "$USE_SOCKET" ] && port="-P$DBPORT"
	echo $user ${pass:+"$pass"} -h "$DBHOST" ${port:+"$port"}
}

# Wrapper around mysql command to allow setting options, user, etc.
mysql()
{
	command mysql $(mysql_options) --silent --skip-column-names "$@"
}

# Quick shell hack to get a key from an INI file.
# This is not perfect and ignores sections at least!
get_ini_file_key()
{
	(
	FILE="$1"
	KEY="$2"

	grep "^[[:space:]]*${KEY}[[:space:]]*=" "$FILE" 2>/dev/null \
		| sed -r "s/^[[:space:]]*${KEY}[[:space:]]*=[[:space:]]*//"
	)
}

# Wrapper around the Symfony console to allow setting the connection and passing options
symfony_console()
{
	if [ -n "$QUIET" ]; then
		ARG="-q"
	else
		ARG=""
	fi
	DATABASE_URL=

	# If we do not have an explicit user and password set, determine them from passed arguments,
	# ~/.my.cnf or the current user

	if [ -z "$DBUSER" ]; then
		if [ -n "$DBA_USER" ]; then
			if [ -n "$PROMPT_PASSWD" ]; then
				stty -echo
				printf "Enter password: "
				read -r DBA_PASSWD
				stty echo
				printf "\n"
			fi
		fi

		# If we do not have a user or password yet, try to read it from ~/.my.cnf
		if [ -f ~/.my.cnf ]; then
			[ -z "$DBA_USER"   ] && DBA_USER=$(  get_ini_file_key ~/.my.cnf 'user')
			[ -z "$DBA_PASSWD" ] && DBA_PASSWD=$(get_ini_file_key ~/.my.cnf 'password')
		fi

		if [ -z "$DBA_USER" ]; then
			DBA_USER=$(whoami)
		fi

		if [ -n "$DBA_USER" ]; then
			if [ -n "$DBA_PASSWD" ]; then
				DATABASE_URL=mysql://${DBA_USER}:${DBA_PASSWD}@${domjudge_DBHOST}:${domjudge_DBPORT}/${domjudge_DBNAME}
			else
				DATABASE_URL=mysql://${DBA_USER}@${domjudge_DBHOST}:${domjudge_DBPORT}/${domjudge_DBNAME}
			fi
		fi
	fi

	if [ -n "$DATABASE_URL" ]; then
		DATABASE_URL=$DATABASE_URL ${WEBAPPDIR}/bin/console -v $ARG "$@"
	else
		${WEBAPPDIR}/bin/console -v $ARG "$@"
	fi

	# Make sure any generated cache data has the right permissions if we ran as root.
	if [ `id -u` -eq 0 ]; then
		${BINDIR}/fix_permissions
	fi
}

read_dbpasswords()
{
	if [ ! -r "$PASSWDFILE" ]; then
		echo "Error: password file '$PASSWDFILE' not found or not readable."
		echo "You may want to run: $0 genpass"
		return 1
	fi
	OLDIFS="$IFS"
	IFS=":"
	# Don't pipe $PASSWDFILE into this while loop as that spawns a
	# subshell and then variables are not retained in the original shell.
	while read -r role host db user passwd port dummy; do
		# Skip lines beginning with a '#'
		[ "x$role" != "x${role###}" ] && continue
		domjudge_DBHOST=$host
		domjudge_DBPORT=$port
		domjudge_DBNAME=$db
		domjudge_DBUSER=$user
		domjudge_PASSWD=$passwd
		DBHOST=$host
		DBNAME=$db
		DBPORT=$port
	done < "$PASSWDFILE"
	IFS="$OLDIFS"
	if [ -z "$domjudge_DBPORT" ]; then
		domjudge_DBPORT=3306
	fi
	if [ -z "$DBPORT" ]; then
		DBPORT=3306
	fi
	if [ -z "$domjudge_PASSWD" ]; then
		echo "Error: no login info found."
		return 1
	fi
	verbose "Database credentials read from '$PASSWDFILE'."
}

status()
{
	if [ ! -r "$PASSWDFILE" ]; then
		echo "Error: cannot read database password file '$PASSWDFILE'."
		return 1
	fi
	read_dbpasswords || return 1

	printf "Trying to connect to the server as DB admin: "
	mysql -e 'SELECT "success.";' || return 1

	printf "Trying to connect to the server with user '%s': " "$domjudge_DBUSER"
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD mysql -e 'SELECT "success.";' || return 1

	printf "Searching for database '%s': " "$DBNAME"
	mysql -e "USE \`$DBNAME\`; SELECT 'found.';" || return 1

	printf "Searching for data in database: "
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD \
		mysql -e "USE \`$DBNAME\`; SELECT name FROM team WHERE name = 'DOMjudge';" || return 1

	printf "MySQL server version: "
	mysql -e 'SELECT version();'
}

create_db_users()
{
	(
	# The MySQL character set and collation are hardcoded here, but
	# can be changed in the database and their configuration settings
	# in etc/domserver-config.php updated after installation.
	echo "CREATE DATABASE IF NOT EXISTS \`$DBNAME\` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"

	echo "CREATE USER IF NOT EXISTS '$domjudge_DBUSER'@'%' IDENTIFIED BY '$domjudge_PASSWD';"
	echo "GRANT SELECT, INSERT, UPDATE, DELETE ON \`$DBNAME\`.* TO '$domjudge_DBUSER'@'%';"

	echo "FLUSH PRIVILEGES;"
	) | mysql
	verbose "DOMjudge database and user(s) created."
}

remove_db_users()
{
	(
	echo "DROP DATABASE IF EXISTS \`$DBNAME\`;"
	echo "DROP USER IF EXISTS '$domjudge_DBUSER'@'%';"
	echo "FLUSH PRIVILEGES;"
	) | mysql -f
	verbose "DOMjudge database and user(s) removed."
}

install_examples()
{
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD symfony_console domjudge:load-example-data
	"$EXAMPLEPROBDIR"/generate-contest-yaml
	( cd "$EXAMPLEPROBDIR" && yes y | "$BINDIR"/import-contest )
}

uninstall_helper()
{
	read_dbpasswords
	remove_db_users
}

create_db_users_helper()
{
	read_dbpasswords
	create_db_users
	verbose "Created empty database and users."
}

create_database_dump () {
	sudo mysqldump $(mysql_options) --opt --skip-lock-tables domjudge | pv | gzip > "$DATABASEDUMPDIR/${1}.sql.gz"
}

### Script starts here ###

# Parse command-line options:
while getopts ':u:p:qrs' OPT ; do
	case "$OPT" in
		u)
			DBA_USER=$OPTARG
			;;
		p)
			DBA_PASSWD=$OPTARG
			;;
		q)
			QUIET=1
			;;
		r)
			PROMPT_PASSWD=1
			;;
		s)
			USE_SOCKET=1
			;;
		:)
			echo "Error: option '$OPTARG' requires an argument."
			usage
			exit 1
			;;
		?)
			echo "Error: unknown option '$OPTARG'."
			usage
			exit 1
			;;
		*)
			echo "Error: unknown error reading option '$OPT', value '$OPTARG'."
			usage
			exit 1
			;;
	esac
done
shift $((OPTIND-1))

case "$1" in

status)
	if status ; then
		echo "Database check successful: database and users present and accessible."
		exit 0
	else
		echo "Database status check failed."
		exit 1
	fi
	;;

genpass)
	$ETCDIR/gen_all_secrets
	;;

uninstall)
	uninstall_helper
	;;

install-examples)
	read_dbpasswords
	install_examples
	;;

install-loadtest)
	read_dbpasswords
	create_db_users
	export DB_FIRST_INSTALL=1
	symfony_console doctrine:migrations:migrate -n
	unset DB_FIRST_INSTALL
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD symfony_console domjudge:load-default-data
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD symfony_console domjudge:load-gatling-data
	;;

create-db-users)
	create_db_users_helper
	;;

bare-install|install)
	read_dbpasswords
	create_db_users
	export DB_FIRST_INSTALL=1
	symfony_console doctrine:migrations:migrate -n
	unset DB_FIRST_INSTALL
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD symfony_console domjudge:load-default-data
	if [ "$1" = "install" ]; then
		install_examples
		verbose "SQL structure and default/example data installed."
	else
		verbose "SQL structure and defaults installed (no sample data)."
	fi
	;;

upgrade)
	# check for legacy dbpasswords.secret content
	if grep -Eq ^team: $PASSWDFILE >/dev/null 2>&1 ; then
		echo "Warning: please remove all non-jury users from $PASSWDFILE"
		echo "You may also remove those users from MySQL."
	fi
	read_dbpasswords

	# Check if we need to upgrade the Doctrine migrations table
	if ! echo "SHOW CREATE TABLE \`doctrine_migration_versions\`" | mysql "$DBNAME" >/dev/null 2>&1; then
		symfony_console doctrine:migrations:sync-metadata-storage -n
		# shellcheck disable=SC2016
		echo 'INSERT INTO `doctrine_migration_versions`
			(version, executed_at, execution_time)
			SELECT concat("DoctrineMigrations\\\\Version", version), executed_at, 1
			FROM migration_versions;' | mysql "$DBNAME"
		echo "DROP TABLE \`migration_versions\`" | mysql "$DBNAME"
	fi

	symfony_console doctrine:migrations:migrate -n
	DBUSER=$domjudge_DBUSER PASSWD=$domjudge_PASSWD symfony_console domjudge:load-default-data
	verbose "DOMjudge database upgrade completed."
	;;

dump)
	read_dbpasswords
	DUMPNAME="$2"
	if [ -z "$DUMPNAME" ]; then
		usage
		exit 1
	fi

	if [ -f "${DATABASEDUMPDIR}/${DUMPNAME}.sql.gz" ]; then
		while true; do
			read -p "Overwrite existing database dump (y/N)? " yn
			case $yn in
				[Yy]* ) break ;;
				''|[Nn]* ) exit 0;;
			esac
		done
	fi
	create_database_dump $DUMPNAME
	exit 0
	;;

load)
	DUMPNAME="$2"
	FILE=""
	if [ -z "$DUMPNAME" ]; then
		databases=$(find "$DATABASEDUMPDIR" -name "*.sql.gz" -type f -print0)
		if [ -z "$databases" ]; then
			echo "No files with .sql.gz suffix found in '$DATABASEDUMPDIR'"
			exit 1
		fi
		ind=1
		for i in "$databases"; do
			echo "$ind) $i"
			: $((ind+=1))
		done
		while true; do
			read -p "Which database should be loaded? " db
			ind=1
			for i in "$databases"; do
				if [ "$ind" = "$db" ]; then
					FILE="$i"
					break
				fi
				: $((ind+=1))
			done
			if [ -n "$FILE" ]; then
				break
			fi
		done
	else
		FILE="$DATABASEDUMPDIR/${DUMPNAME}.sql.gz"
	fi

	if [ ! -f "${FILE}" ]; then
		echo "Error: file ${FILE} not found."
		exit 1
	fi

	uninstall_helper
	create_db_users_helper
	pv "${FILE}" | gunzip | mysql domjudge
	;;

*)
	echo "Error: Unknown subcommand '$1'"
	usage
	exit 1
	;;

esac

Redis… Not Actually Used by Default

Even after logging in fine, Redis didn’t seem to be used. Sessions worked, but DOMJudge was still using file-based sessions despite our Redis URL and PHP settings. Digging through Symfony docs (DOMJudge is Symfony-based) showed we also needed to update Symfony’s framework config.

So in the patched image, I overrode a few config files (e.g., framework.yaml and service.yaml) to switch sessions to Redis. Only then did the architecture finally behave like the diagram in my head. Documentation really matters…

framework.yaml:

# see https://symfony.com/doc/current/reference/configuration/framework.html
framework:
    secret: '%env(APP_SECRET)%'
    esi: false
    fragments: false
    http_method_override: true
    annotations: true
    handle_all_throwables: true
    serializer:
        enabled: true
        name_converter: serializer.name_converter.camel_case_to_snake_case

    # Enables session support. Note that the session will ONLY be started if you read or write from it.
    # Remove or comment this section to explicitly disable session support.
    session:
        handler_id: Symfony\Component\HttpFoundation\Session\Storage\Handler\RedisSessionHandler

    php_errors:
        log: true

    assets:
        version: "v=%domjudge.version%"

when@test:
    framework:
        test: true
        session:
            storage_factory_id: session.storage.factory.mock_file

and services.yaml:

# This file is the entry point to configure your own services.
# Files in the packages/ subdirectory configure your dependencies.

# Put parameters here that don't need to change on each machine where the app is deployed
# https://symfony.com/doc/current/best_practices.html#use-parameters-for-application-configuration
imports:
    - { resource: static.yaml }

parameters:
    locale: en
    # Enable this to support removing time intervals from the contest.
    # This code is rarely tested and we discourage using it.
    removed_intervals: false

services:
    # default configuration for services in *this* file
    _defaults:
        autowire: true      # Automatically injects dependencies in your services.
        autoconfigure: true # Automatically registers your services as commands, event subscribers, etc.

    # makes classes in src/ available to be used as services
    # this creates a service per class whose id is the fully-qualified class name
    App\:
        resource: '../src/'
        exclude:
            - '../src/DependencyInjection'
            - '../src/Entity'
            - '../src/Migrations'
            - '../src/Kernel.php'

    Symfony\Component\HttpFoundation\Session\Storage\Handler\RedisSessionHandler:
        arguments:
            - '@Redis'

    Redis:
        # you can also use \RedisArray, \RedisCluster, \Relay\Relay or \Predis\Client classes
        class: \Redis
        calls:
            - connect:
                - 'redis.default.svc.cluster.local'
                - 6379

Prelim Prep

With web services stable, we set up monitoring and load testing:

• Monitoring: Prometheus + Grafana
• Load test: Locust, simulating 800 concurrent users for 2 minutes

Preliminary Round Day

Prelim was fully online, so we camped at the Computer Center, watched dashboards, and ate fried chicken + pizza until it ended XD.

No major fires. One recurring annoyance: some judgehosts occasionally failed to fetch test data from DOMJudge; rejudge fixed it. We also learned to set judgehost resource requests = limits; otherwise, the same submission might pass sometimes and fail other times… not fun.

Finals Prep

We held a few syncs to go over prelim issues and finalize the finals plan. On Sept 13, the advance team (3 of us, including me) flew to Malaysia to prep contestant laptops and deploy servers. The rest (Judging Team, three faculty leads, and remaining Systems Team) arrived on Sept 17.

Before flying, we prepped the server laptops (acting as master nodes) with Kubernetes and deployed DOMJudge on top. Thanks to the manifests we’d already written for the prelim, it was just a matter of tweaking resource allocations for the laptop server hardware. One week before departure, we ran extra stress tests and HA drills. All green.

Mini rant about reimbursements: pay first, attach receipt, then wait (forever?) for reimbursement. If a student’s tight on budget, that’s rough QQ. (Yes, sponsor means not fronting money in a perfect world…)

Advance Team, Wheels Up!

Sept 13: first time flying STARLUX — so exciting XD.

Finally tried the in-flight Mahjong mini-game (usually I fly budget so no fancy screens QQ).

The meal was great too… I’m worried I’ll get spoiled and forget how to “budget airline” XDD

At KLIA, I zipped through immigration (Malaysian perk ><) and waited for the other two teammates. Then straight to the alumni office to set up contestant laptops and the server.

Food pics from those days — welcome to Malaysia, where the cuisine is a personality trait:

Main Group Arrives!

On Sept 17, we (the advanced team) headed to SMK Seri Seremban to start venue setup. Our job: place the server laptops in a side area next to the stage, set up the switches and router, confirm network details with the school’s IT (gateway IP, available IPs), and coordinate with the printer vendor to lock in IP settings.

The printer vendor spoke only a little English — good thing I could switch to Malay. Not sure how they managed this part in previous years XD

And here’s the “pre-finals midnight snack” shot:

Test Contest

Sept 19: we hosted a test contest so students could check their laptop environments and the DOMJudge setup. After it ended, we collected all laptops, re-imaged them (just in case someone planted “creative” things during testing XD), and set final credentials and problems & contest. We worked till around 11 pm before heading back to the hotel.

Finals Day

Sept 20, we arrived at 6:30 am… and immediately hit chaos. After breakfast, we got back to find one server laptop had basically reset — its OPNSense instance reverted to initial state (as if nothing had been configured), and the contest started in 30 minutes.

We split tasks: Infra sprinted through network checks and tried to pull the latest OPNSense backup from the other nodes (pve1/pve3) to restore pve2. I focused on ensuring DOMJudge stayed up despite pve2 being down (thanks to HA), but the judgehosts started spamming Warnings → Errors.

The final fix: sync clocks across OPNSense and all services, and make sure MySQL Routers were spread across different machines. After that initial scare, the rest of the finals went smoothly — no service outages, no network splits, no late contestants. Deep breaths, big smiles.

When the contest ended, the Judging Team headed to the ceremony for solution explanations and awards. Systems packed up contestant laptops, server gear, and cabling. Around 7 pm we went to the celebration dinner. The NTU team was supposed to go back to a hotel in KL, but I’d requested to stay a few more days in Malaysia (since I was home anyway), so I split from the group and headed to a different hotel.

A Quick “Going Home” Interlude

Escaping the start of the semester… just a bit XD. I spent a couple of days chilling at home, stocking up on snacks and essentials before flying back to Taiwan — and of course, hanging with the cats.

Closing Thoughts

Kubernetes absolutely paid off: same manifests for prelim and finals, portable between data center VMs and on-site laptops, and recoverable when one node decided to cosplay a brick. Sure, we had to patch DOMJudge and fight some Symfony configs, but the consistency, observability, and HA were worth it. Would choose K8s again.