JSON{"record_id":"R-1842", "status":"active", "updated_at":"14:00:00Z"}
SQLSELECT * FROM bronze.events
APIGET /api/v2/events 200
EVENT STREAMevent.updated.v3 partition 07 · offset 384920
PYSPARK.withColumn("ingested_at", current_timestamp())
SCHEMArecord_id STRING value DECIMAL(18,2) event_ts TIMESTAMP
DELTAcommit_version 42
DUPLICATErecord_id: R-1842 version: duplicate
DUPLICATErecord_id: R-1842 version: duplicate
INVALIDrecord_id: null value: null
API Database File Stream
Trusted Data Engineered · Reliable · Useful
AnalyticsPower BI · Direct Lake
Data ProductsModels · APIs
Real-TimeStreaming · Events
AI & AppsAgents · Applications

ENGINEERING DATA
FOR WHAT COMES NEXT.

I design and build modern data platforms, reliable pipelines and analytical systems, from ingestion and transformation through to trusted data products.

TRUSTED DATA
AnalyticsPower BI · Direct Lake
Data ProductsModels · APIs
Real-TimeStreaming · Events
AI & AppsAgents · Applications
Scroll to watch fragmented data become engineered systems↓
02 / EXPERTISE

MODERN DATA SYSTEMS,
BUILT END TO END.

I design and build the systems that move data from source to trusted, usable products, covering ingestion, transformation, lakehouse architecture, orchestration, real-time processing, observability and analytics enablement.

DATA INGESTION

APIs · Databases · Files · Batch

TRANSFORMATION

PySpark · Spark · SQL · Delta

LAKEHOUSE ARCHITECTURE

Medallion · OneLake · Data Modelling

ORCHESTRATION

Pipelines · Metadata-driven frameworks · CI/CD

REAL-TIME DATA

Streaming · Eventstream · Event Hub

OBSERVABILITY & GOVERNANCE

Monitoring · Data Quality · Access · Reliability
ENGINEERING PRINCIPLES
RELIABLE. OBSERVABLE.
MAINTAINABLE. BUILT FOR PRODUCTION.

Good data engineering is not only about moving data. It is about building systems that remain understandable, dependable and useful once they reach production.

03 / MICROSOFT FABRIC

BUILT FOR
MICROSOFT FABRIC.

I design and build modern data platforms in Microsoft Fabric, from ingestion and lakehouse architecture to transformation, orchestration, real-time processing, governance and analytics enablement.

REST API
SQL SERVER
ORACLE
FILES
EVENT STREAMS
INGEST Fabric Data Pipelines · Eventstream
LANDINGraw
BRONZEvalidated
SILVERconformed
GOLDcurated
STOREOneLake · Lakehouse
TRANSFORMSpark / PySpark · Notebooks
ANALYTICSPower BI · Direct Lake
DATA PRODUCTSModels · Warehouse
REAL-TIMEStreaming · Events
ENGINEERED DATA → USEFUL SYSTEMS
OBSERVABILITYMonitoring · Data Quality · Reliability
GOVERNANCEAccess · Security · Control
SOURCESREST API · SQL Server · Oracle · Files · Event Streams
↓
INGESTFabric Data Pipelines · Eventstream
↓
LAKEHOUSEOneLake · Lakehouse
↓
LANDING → BRONZE → SILVER → GOLDraw → validated → conformed → curated · Spark / PySpark · Notebooks
↓
SERVEPower BI · Direct Lake · Models · Warehouse · Streaming · Events
PLATFORM-WIDE
OBSERVABILITYMonitoring · Data Quality · Reliability
GOVERNANCEAccess · Security · Control
BUILDS

BEYOND DATA
PLATFORMS.

Beyond core data engineering, I build practical digital products, websites, internal tools, automation and increasingly AI-enabled applications.

BUSINESS WEBSITES

Modern responsive websites for businesses, professionals and small organisations.

Responsive · Modern · Practical

DATA-DRIVEN APPLICATIONS

Lightweight applications and internal tools built around useful data and workflows.

Data-first · Internal tools · APIs

AUTOMATION

Practical workflow automation and engineering utilities that reduce repetitive work.

Workflows · Integration · Productivity

AI & AGENTS

Expanding into AI-enabled applications, agents and intelligent workflows using modern AI tooling.

Agents · AI applications · Automation
This website is one of those builds.
Nidhi Vij
ABOUT

NIDHI
VIJ.

Engineer. Builder. Curious by default.

I'm a Data Engineer based in London, working across modern data platforms, engineering and analytics.

Over the past five years, I've worked across technology, analytics and data. For more than four years, my focus has been designing, building and operating modern production data platforms.

My engineering work spans Microsoft Fabric, PySpark, SQL, lakehouse and medallion architectures, metadata-driven ingestion, real-time data processing, observability, governance and production platform delivery.

I enjoy the point where strong engineering meets useful products: taking complicated systems and making them reliable, understandable and genuinely useful.

Alongside data engineering, I'm expanding my work in AI agents, automation, applications and modern web products.

MICROSOFT FABRIC · PYSPARK · SQL · SPARK · LAKEHOUSE · MEDALLION ARCHITECTURE · REAL-TIME DATA · DATA MODELLING · AZURE · AI AGENTS
DomainData & AI Engineering
PracticeData Engineering
FocusLakehouse · Fabric · Spark
BaseLondon
CONTACT

LET'S BUILD
SOMETHING USEFUL.

Whether you're hiring, building a data platform, exploring an AI solution, creating a digital product or simply want to talk engineering, I'd be happy to hear from you.

EMAIL[email protected]
LINKEDINlinkedin.com/in/nidhi-vij