Real-Time Data Platform for a Global Enterprise Client

The Challenge

The client was running a legacy, on-premises operational database that held their core business data, but getting that data into a usable, analyzable form meant manual, periodic exports. Reporting was always looking backward, and there was no way for stakeholders to see current numbers without waiting on a scheduled pull.

The client wanted to modernize: move to a cloud-based data platform that could connect securely to their existing on-prem system, pull data continuously instead of in batches, and give their team a foundation to build real dashboards on top of.

A vendor had already attempted the initial setup using a standard implementation guide. It didn't work: the cloud data environment was built in isolation, with no actual network path back to the client's on-prem database, and the connectivity model it assumed didn't match the client's real network architecture. I inherited this as a clean-slate rebuild.

Architecture & Approach

The client's network followed a hub-and-spoke model with a central firewall appliance handling all internet-bound traffic. That firewall couldn't consume the cloud provider's dynamic service tags the way a native cloud firewall could, which meant the network design had to route the new platform's specific outbound dependencies directly, while still sending everything else through the client's existing security stack.

• Direct network injection into the client's existing virtual network, rather than a standalone environment with a bolt-on connection, so the platform could reach the on-prem database over private, internal routing with no public exposure.
• A dedicated NAT gateway and route table, built so only the platform's required control-plane and service traffic took the direct path out; everything else still hit the client's existing firewall.
• Secrets management through the cloud provider's key vault service, with credentials retrieved at runtime and rotation handled entirely on the vault side, no redeployment needed.
• Governed, least-privilege access, using per-user grants during rollout to keep permissions auditable.

Security: Pushing Back on a Risky Shortcut

Partway through the project, the vendor pushed for the simplest path for them: open the database port and web-based access directly to the public internet, no IP restrictions, no gating. It would have made their own access easier, but internet-facing database instances are among the most heavily targeted infrastructure on the internet, constantly hit by automated port scanning, credential-stuffing attempts, and ransomware campaigns that specifically look for exposed database ports.

As the sole engineer and project lead on the engagement, I pushed back, brought the risk and my proposed alternative to leadership, and got the go-ahead to implement it a different way.

• VPN access with individual accounts issued per user and per vendor, rather than shared or generic logins, so every connection was attributable to a specific person.
• Access restrictions applied directly on the database host and the cloud workspace and its resources, so even an authenticated VPN session only reached what it needed.
• Directory-based authentication layered on top of the VPN connection, adding a second identity check beyond network-level access alone.

Troubleshooting: Two Root Causes That Weren't Obvious at First

Compute couldn't reach the private network. The first test runs used the platform's default serverless compute, and every connection attempt failed. Tracing the network path showed the compute was landing on an address range that had nothing to do with the client's network, a completely different, non-routable space. Switching to a dedicated compute cluster that actually provisioned inside the client's virtual network resolved the routing mismatch immediately.

Environment setup silently broke its own network config. Once compute was correctly placed, health checks still failed with a generic connectivity error. The actual cause: the initial environment provisioning process had quietly detached the network route associations it needed to function, no error, no warning, it just came up broken. Re-associating the network correctly fixed it, and from there, direct database connectivity tested successfully end to end.

Both of these were the kind of failures that don't show up in vendor documentation, generic errors that could mean five different things, and the only way through was methodically ruling out each layer: compute placement, network routing, then the actual database connection.

Real-Time Ingestion

Once basic connectivity was solid, the next phase was building an actual change-data-capture pipeline, so the platform wasn't just reachable, it was continuously ingesting updates as they happened in the source system. This required enabling change tracking at the database level, a prerequisite the ingestion tool's error messages didn't make obvious when missing, and configuring per-table capture for the specific datasets the client's team needed.

Outcome

• The client's team went from periodic manual exports to near-instant access to live operational data.
• They were able to build dashboards and review business metrics directly against current data for the first time.
• The platform was left in a state ready for the client to expand access and reporting on their own, with a full runbook and architecture reference handed off for their internal team.
• Documented a small number of accepted-risk items for the client's awareness and future action, rather than leaving them undocumented.

What This Demonstrates

• Recognizing an insecure shortcut when a vendor pushed for convenience over security, and having the standing and judgment to say no and implement something better.
• Diagnosing multi-layered connectivity failures across compute, networking, and application layers without clear error messaging to go on.
• Designing network architecture that respects a client's existing security posture instead of routing around it.
• End-to-end delivery: architecture, implementation, troubleshooting, documentation, and knowledge transfer, solo-led from the vendor side.
• Working across an international team and client stakeholders in multiple time zones to deliver on a fixed timeline.

Client name, industry, and identifying details have been withheld or generalized to protect confidentiality.