Platform migration
Migrating 35+ microservices, one version at a time
The platform ran 35+ services on an aging major version of Spring Boot, and every one sat a
different distance from the target: different gaps between the production branch, the mainline, and
the upgrade branch. I co-owned the tracking and the method. Two pages held the truth, a three-way
branch comparison for the reporting services and a merge-status board for all 35. That board
surfaced nine repositories with conflicts blocking the upgrade, four where the mainline already
contained the change, and one abandoned service with no active certificate and no commits in a
year, which I cut from scope entirely. I also wrote the team's upgrade guide, because I had watched
engineers try to jump several major versions at once and hit cascading breakage they could not
isolate: step one version at a time, run the automated namespace migration at each step, let the
dependency BOM manage versions instead of hardcoding them, and run a full build and test pass
before moving on.
Result
One source of truth for 35 parallel efforts, with blockers visible before the release window
instead of during it.
Observability
Knowing your dependencies are up, not just your service
The platform had grown to 22 services across four deployment targets, and during an incident
engineers were losing time hunting for the right log index or APM process group. I audited all 22,
confirmed every log index was live and every process group reporting, and put every link in one
place. The audit turned up real gaps: services missing the network egress rule their APM collector
needed, and one scheduled job that is not a long-running deployment at all, so APM does not apply
and only log search does. Then I went a layer down, because knowing our services are up is not the
same as knowing their dependencies are. I wrote a custom Spring Boot health indicator that checks
20+ downstream dependencies in parallel with CompletableFuture and reports
per-dependency status and response time on its own endpoint, separate from the framework's, plus a
Python job that calls it across the fleet and mails a per-service table.
Result
One bookmark instead of a scavenger hunt, and dependency-level visibility the team did not have
before.
Automation at scale
One config change, 35 repositories
A platform mandate meant removing a legacy block from the deployment config of roughly 35
repositories. Thirty-five hand-written pull requests is a poor use of a week, so I scripted it as a
batch change across all of them at once. Getting it right took three attempts, and the wrong ones
are the interesting part. The first had a malformed pattern that appended without respecting YAML
nesting. The second used a range delete, which is fragile: it removes everything between two
markers and will happily swallow an unrelated section. The third used awk to flag on
the key and stop at the next non-indented line, identifying the block by indentation and staying
correct regardless of how many lines it ran to. Alongside it I ran a Python audit across 38
repositories for deprecated cluster references. It hit an authentication wall, and rather than drop
it I moved to token-based API access, which is now how I do all programmatic repository work.
Result
All 35 updated by automated pull request in days rather than weeks. Eight violations found, six
real and fixed, two false positives already commented out.
Environment provisioning
Five services, five completely different failures
I was asked to make a five-service stack work in a development environment where it never had.
Every service was broken for a different reason, which is the honest lesson of distributed systems.
The front end had no dedicated auth client for that environment at all. The gateway had four
separate problems stacked on each other: a wrong database hostname, a missing network egress rule,
a service account never provisioned in that database, and an audit table that simply did not exist,
which I created by hand. One service was still pointed at a decommissioned cluster and needed new
ingress addresses ordered. One had a connection timeout that needed both an egress fix and an
access request. The method was unglamorous: read the actual error out of each pod instead of
guessing, then fix them one at a time. I did it again later for an eight-service environment wired
to a database refreshed nightly from production, which catches data-shaped bugs a static test
dataset never will.
Result
All services healthy, all seven screens validated, and the investigation written up so nobody
starts from zero next time.
Release engineering
Compliance-gated releases, 13 services at a time
I have served as release engineer across two very different ecosystems, one on on-prem Kubernetes
and one on AWS. The largest release I ran covered 13 services under a single change request. The
gates are real and none of them are optional: code coverage above threshold on new code and
overall, security scans with no unresolved critical or high findings, a change request carrying
every signed-off ticket and scan link, then approval from the deployment control platform before
the pipeline will run in production at all. After that it is notification, deploy from the release
branches, sanity-check the screens that matter, confirm, then merge back and tag. A separate
release carried ten tickets at once, well above the usual batch of two to four, which meant
justifying the size on the grounds that all ten had been validated together in the same sprint.
Result
Clean releases with no post-release incidents, and a written SOP the team still follows.
Tooling for operations
A bulk-update tool built for whoever cleans up after it
Corporate actions like stock splits used to mean an engineer hand-running SQL across four tables. I
built the wizard that replaced that, and most of the design went into what happens when it goes
wrong. The interface works in human-readable instrument identifiers and resolves them to internal
keys server-side, so a request can never carry an arbitrary key that touches the wrong security. It
shows a small sample of rows before and after, enough for the operations team to sanity-check
without dumping the dataset. It names the instrument on a confirmation screen before anything
irreversible runs. On a related onboarding tool I mapped all 35 input fields before writing a line
of code, which is how I noticed one selection could derive three others and take the form from 35
inputs down to 32.
Result
A multi-day, engineer-dependent process became self-service for the team that actually owns it.
Security
Certificates, and making them boring on purpose
I owned the full lifecycle of the mutual TLS certificates behind a managed file-transfer
integration: requesting them, installing them, rotating them, and testing them. Certificate work is
easy the first time and painful every time after, because whoever does it next is usually doing it
under pressure with an expiry date behind them. So I wrote it down as a nine-step SOP with the
exact commands, including the keystore format conversions the authentication flow needs, plus a
companion guide for exercising certificate auth from an API client before trusting it in a running
service.
Result
A rotation that used to be an investigation is now a checklist.
Reliability
Two clusters, one datacenter, zero redundancy
Services on the platform are meant to deploy to a pair of clusters sharing a base identifier but
sitting in separate availability zones, so losing one zone leaves the other serving. Auditing the
deployment config of every service against the load-balancer and cluster dashboards, I found pairs
built from mismatched bases, and one service spread across three clusters on three different bases,
which means inconsistent patch levels and maintenance windows on top of the redundancy problem. I
did not own those services. I documented the pairing rule, ordered the replacement ingress
addresses, raised the pull requests myself, and asked the owners to review. I also kept the cluster
rebuild SOP, recording each service's release branch and pipeline before a repave so the platform
could be brought back in minutes.
Result
Correctly paired across availability zones, with the rule written down so the next person checks
it before it becomes an incident.
