This post isn’t about whether or not digital twins are cool. Instead, it reflects an observation I didn’t expect going in. The way this particular twin came together, it was really three refinement pipelines feeding one scene: geometry (LiDAR to mesh), telemetry (Eventhouse to tools), and language (tools to agent). I’m not claiming every digital twin looks like this, but in this build, the three pipelines shared one discipline: Reduce the data as far upstream as you can, keep the runtime thin, and generate every downstream artifact from a single source of truth.
In the video above, you’re looking at a real place but a simulated wind farm: a site in Sievi, Finland, its terrain built from open laser-scanning data, its turbines fed by live telemetry from a Fabric Eventhouse. You can orbit it, pull a turbine apart component by component, and ask (in either typed or spoken language) which turbine is underperforming and what it’s costing. The goal: Build a realistic and hands-free digital twin experience within Microsoft Fabric.
Digital twin application embedded in Fabric Portal
Behind the scenes, there’s a stack of features: a React + Vite + Three.js frontend hosted as a Microsoft Fabric app via Rayfin, Fabric Real-Time Intelligence for telemetry, and a managed Microsoft Foundry agent for the language layer.
Architecture overview.
The skeleton: Rayfin
Rayfin is an open-source SDK and CLI for building apps whose backend lives inside Microsoft Fabric. You define entities as decorated TypeScript classes, run npx rayfin up, and it provisions the SQL tables, Fabric SSO, APIs, and static hosting. No infrastructure to stand up before the first pixel:
@entity()
@authenticated('*')
export class Turbine {
@uuid() id!: string;
@text({ max: 40, unique: true }) turbineId!: string; // "WT-01"
@text({ max: 120 }) name!: string;
@decimal() ratedCapacity!: number;
@decimal({ precision: 9, scale: 5, optional: true }) latitude?: number;
@many(() => Component) components?: Component[];
}
The part that paid off later: This data model is the contract for everything downstream. The same Turbine and Component entities that hold seed data become the binding targets for the 3D model (each part is named after a component) and the grounding vocabulary for the agent (tools speak in turbineIds). Get the model right once; reuse it three times.
One constraint shaped the architecture: Rayfin’s functions were not available in the current preview state streaming functions are not supported. Anything cross-origin (streaming chat, voice token minting, tool execution) can’t live there. So those moved into a deliberately thin backend deployed as its own Container App:
Browser (Fabric-embedded React + Three.js)
├─ chat ──► thin backend /api/assistant/* (streamed NDJSON)
├─ voice ──► thin backend /api/realtime/* → WebRTC direct to Azure OpenAI Realtime
└─ tools ──► thin backend /api/tools/:name (deterministic Fabric RTI tools)
Geometry: 7.7 million points you can’t ship
I wanted the terrain to match the physical site, not a procedurally generated stand-in. The National Land Survey of Finland (Maanmittauslaitos) publishes open geodata under CC BY 4.0, and my first attempt used the obvious source: vector map height contours. That didn’t work as the contours were far too sparse to build a convincing surface.
Vector map height contours.
What did work was their LiDAR data: laser-scanned point clouds at 0.5 points/m², delivered as LAZ tiles. For our area, 7.7 million points.
LiDAR point cloud visualized.
For our digital twin, this source data was too detailed to load directly into the browser. A single LAZ tile can contain millions of points, and rendering that raw point cloud in Three.js would be unnecessarily heavy for an interactive web application. Instead, I used the LAZ data as a high-quality source for generating a lighter terrain representation.
The processing step converts the point cloud into a browser-friendly height model. Conceptually, the workflow looks like this:
- Read the LAZ tile.
- Extract the point coordinates: easting, northing, and elevation.
- Crop the data to the area needed by the digital twin.
- Aggregate the points into a regular height grid.
- Convert the real-world coordinates into local Three.js coordinates.
- Build a Three.js terrain mesh from the height grid.
- Color and light the mesh so terrain shape is visible.
The final mesh is much lighter than the original point cloud: over 7.7 million source points reduced to roughly 6,500 vertices and 12,800 triangles (an 80×80 segment plane). That makes it suitable for an interactive browser-based digital twin, where we also need to render turbines, labels, weather effects, camera controls, and live operational data.
The sparse-but-semantic vector layers we rejected for elevation earn their keep on top of it: Roads, rivers, fields, water, and powerlines drape onto the mesh, so the terrain isn’t just shaped like the real place—it also reads like it.
Final generated terrain with roads, contours, trees, and other features. Terrain/map data: National Land Survey of Finland (Maanmittauslaitos), CC BY 4.0.
Modeling the wind turbine
The turbine model had its own failed first attempt: procedural generation directly in Three.js, which produced geometry I can only describe as legally distinct from a wind turbine. The approach that worked was moving the generation offline. I used GitHub Copilot with Opus 4.8 to write a Blender Python script that builds the turbine and exports a compact GLB with separately named parts (rotor, nacelle, tower). Same principle as the terrain: Do the heavy work upstream; ship a small artifact.
From simple geometry to realistic model with BOM.
The scene: Keep Three.js out of React
The rendering layer is intentionally boring: HomePage → Twin3DView → SceneEngine, where Twin3DView is a thin React wrapper and the SceneEngine is ~300 lines of framework-agnostic Three.js that knows nothing about React. The render loop never fights React’s lifecycle, and the engine is testable without mounting a component tree. The GLB loads once as a prototype and is cloned per turbine, so each instance gets its own material state and risk coloring, hover highlighting via raycasting against those named parts, and rotor animation driven by live wind data.
The language layer: An agent that’s as fast as your voice
I was faced with a hard task when deciding what kind of agent should power the digital twin. I knew that the requirement of having real-time capable voice chat would drive my decisions further. The agent is where most of the design decisions live, and it’s where I’d defend them hardest.
Model selection
In order to drive real-time voice chat, I would need a model that supports this and would provide a good (and multilingual) experience. Real-time voice models allow you to build a natural conversation experience where you can interrupt and have more natural conversation. I decided to use gpt-realtime-2 from Microsoft Foundry for this task. It’s OpenAI’s advanced real-time voice model, capable of speech-to-speech interactions with GPT-5-class reasoning and tool calling. This will come in handy when we need to ground our answers in data.
Lightning-fast grounding
In order to keep the conversation flowing and natural, the agent would need to answer fast. Usually building an agent that reasons over data, we would either use some NL2SQL generation or out-of-the box solution like Fabric IQ. But this time we would want the answer in a fraction of a second rather than waiting multiple seconds for an answer.
So instead of NL2SQL, I built deterministic tools that fetch data from Fabric KQL Database. Each of the tools calls a function in KQL database that returns the requested data.
Tool calling latency.
With this approach, the tool calls take just over 100ms for most of the tools, including network latency.
End-to-end agent experience
This approach of having real-time voice model and deterministic tools enables us to build a voice chat experience that feels like actual conversation rather than a simple question-and-answer machine.
End-to-end latency for simulated voice chat.
Lessons learned
Three findings stuck with me after shipping this:
For a voice-enabled twin, generated SQL isn’t fast enough. When someone is speaking to the twin, they expect an answer before the pause gets awkward. I started down the obvious path (NL2SQL, or an out-of-the-box option like Fabric IQ), and the latency killed the conversation. What worked was the opposite of clever: a small set of deterministic, purpose-built tools backed by KQL functions. Most of them return in just over 100ms including network, which is what keeps the whole exchange feeling like a conversation instead of a query.
Open data will carry more of the build than you’d expect. The terrain here isn’t a stand-in. It’s the real ground. Free, openly licensed geodata took this from “a plausible-looking hill” to “the actual site,” and that credibility is what makes the scene feel operational rather than decorative.
Complex, 3D, agentic apps can be built fast now. This is a React and Three.js frontend on a Fabric backend, a Blender-generated turbine, a real-time voice agent, and a KQL tool layer. A year ago, that would have been a multi-week project. GitHub Copilot wrote the Blender script, scaffolded the scene, and helped sequence the whole plan, and this was ready in days, not weeks.
The interesting part isn’t that the AI wrote code. It’s that the bottleneck moved from typing to deciding, so the time went into making architecture choices that mattered.