A containerized enterprise data platform that modernizes legacy SOAP services, flat-file batch feeds, REST APIs, and OLTP database changes (CDC) into a governed dimensional warehouse — with orchestrated ETL (Airflow + dbt), data-quality gates (Great Expectations), lineage tracking (Marquez), and full observability (Prometheus/Grafana). Debezium CDC → Kafka → bronze/silver/gold lake, with every claim backed by measured runs.
its a personal portfolio project: a containerized data platform where four sources pretend to be a messy enterprise, and (in later phases) Debezium, Airflow and dbt move all of it into a proper warehouse with quality gates, lineage and dashboards. The rule I set for myself: every claim in the docs has to be reproducible with a make target, and every number has to come from a measured run. No “would work in prod” hand-waving. If it doesn’t run, it doesn’t count.
phase 0 was the docker compose skeleton. This entry is phase 1, the sources. Everything below is real output from the running stack:
The first source is an OrderManagement SOAP service, because apparently I hate myself. Python + spyne, SOAP 1.1 envelopes, HTTP basic auth, its own little SQLite store so it feels like a real legacy system that owns its data. Seeded it with 3 years of order history: 382k orders, 785k line items, done in 43 seconds on first boot.
The honest bit: people say “WSDL-first” but spyne is code-first, so instead of pretending, I freeze the served WSDL into the repo as a golden artifact and a test fails if the served contract ever drifts from it. That frozen file IS the contract. I think that’s a more defensible story than a dead WSDL nobody checks.
My favorite part is the idempotent CreateOrder: send the same client_reference twice and you get the original order back, no duplicate. Four parallel requests with the same reference? Exactly one order survives. Retries are side-effect-free, which will matter when the ingest layer shows up.
War story of the evening: lost 20 minutes to pip install spyne==2.14.5. That version does not exist. PyPI has 2.14.0. The internet confidently suggested 2.14.5. Cool.
The OLTP database. 5.4 million rows across users, orders, order_items and payments. Row-by-row INSERTs would have taken forever, so it’s one big COPY from stdin inside a single transaction: 5,399,992 rows in 199 seconds, about 558 MB on disk. Crash mid-load? Rollback, rerun, clean. Constraints and indexes get built after the load, and identity sequences get re-synced so the mutation loop can insert without collisions.
Then the fun part: a mutator container that ticks every 2 seconds, walks orders through statuses, touches logins, inserts new orders, deletes old cancelled ones. Measured WAL churn: about 232 KB/s, continuously. It exists purely so Debezium has something to eat in phase 2.
The REST API. FastAPI “Pricing & Promotions” service built to misbehave like a real partner API: cursor pagination over 2,500 promotions, token-bucket rate limiting, and flaky 500s from a seeded hash so the failures are deterministic and testable. Filling the default bucket of 30 with rapid-fire requests gets you a 429 at exactly request 31, Retry-After header included. The smoke client walks all 25 pages, no dupes, no gaps, and it had to retry 2 genuine 500s on the way. Retry logic you can’t turn off isn’t retry logic, it’s decoration.
The file drop. A nightly CSV generator for customers and products with realistic dirt baked in: 2% verbatim duplicate rows, a few ragged lines (short AND long), and one row that’s secretly cp1252-encoded inside the UTF-8 file, so strict decoders faceplant exactly like they do with real vendor exports. There’s a --late-offset flag that backdates a batch so files arrive late, because of course they do. Same inputs produce byte-identical files, so when the phase 2 extractor chokes on one of these, I can reproduce it forever.
