Keywords: Microsoft Fabric, Databricks, AI Platform, Data Infrastructure, Medallion Architecture, Lakehouse, PoC
After running a data platform for four or five years, issues that were invisible at launch begin to surface. As data and processing grow across departments and use cases, silos form and similar logic appears in multiple places. Years of incremental changes can also turn source code into a black box. Meanwhile, the surrounding technology continues to evolve, making the original architecture look increasingly dated and adding more development and operational overhead whenever the organization adopts new analytics capabilities or AI use cases.
This matters even more as LLMs and AI agents evolve rapidly. They are no longer merely add-on features; they are becoming active consumers that search, interpret, and use data on their own. A data platform that was not designed for this shift will struggle to keep pace with new AI use cases.
What is needed is data platform modernization: a reassessment of how data is stored and processed, how it is governed, and how the platform is developed and operated. At its core, modernization should move the platform toward an AI-ready foundation, where people, AI models, and AI agents can all use data accurately and safely. The key selection criterion is therefore not only whether the current system can be migrated, but whether the resulting Data & AI Platform can support the organization several years from now.
This article describes our investigation of Microsoft Fabric (hereafter, Fabric), Azure Databricks, Snowflake, and Google Cloud as part of modernizing an environment built primarily on Azure SQL Database and Azure Synapse Analytics. We initially began moving toward Fabric because it offered continuity with the existing environment. After a production-oriented proof of concept, however, we decided to bring Databricks back into the comparison. The purpose of this article is to explain how that experience changed our criteria for selecting a solution—not to declare a universal winner between Fabric and Databricks.
⚠️ This article reflects solution capabilities and our validation status as of September 2026. The observations are based on one organization’s requirements and should not be read as an absolute assessment of any service.
The Conclusion Up Front
From the outset, the PoC treated two questions as separate evaluation tracks: Can we migrate the existing platform to Fabric? and Can Fabric serve as an AI-ready data platform for future AI use cases? We concluded that the migration was technically feasible. The second question remained open because Fabric still had maturing capabilities and because operating governance across Ontology and Purview introduced issues that required further evaluation.
| Question | Current assessment |
|---|---|
| Migration to Fabric | Technically feasible, but maturity and operability as an AI-ready platform must be assessed separately |
| Whether to adopt Fabric | Not a foregone conclusion; we will decide after comparing it with Databricks |
| Migration to Databricks | Not decided; Databricks is a candidate for comparative validation |
| Why the selection can be revisited | The medallion architecture can remain intact while components and connection settings are changed |
| Next step | Implement the Fabric workload on Databricks and compare practicality and operational overhead |
Judge the solution not only by whether it can support the migration, but by whether it can support AI several years from now.
1. The Existing Conditions That Shaped the Decision
Selecting a data platform is not simply a feature comparison. Most organizations have already accumulated cloud infrastructure, data assets, BI systems, security controls, and operational expertise. These assets constrain a migration, but they are also the starting point for the next platform. In our case, the existing environment included:
- Azure SQL Database, Azure Synapse Analytics, and Power BI
- A third-party machine learning platform
- Infrastructure and security designed around Azure
- Existing T-SQL, notebooks, and other code assets
- In-house Azure expertise
The question was not, “Which solution has the most features?” It was how to evolve our existing Azure data platform into the next generation of the platform.
The right answer depends on the organization’s cloud footprint, data assets, BI environment, engineering skills, security requirements, migration cost, and future AI strategy. An organization standardized on Google Cloud may have good reason to choose BigQuery. One that has already established its data warehouse and data-sharing model around Snowflake may reasonably continue with Snowflake.
In our case, Microsoft Fabric was a strong candidate because of its close fit with our Azure-based environment and its ability to bring Power BI and OneLake together on a single SaaS foundation. This reduced the apparent migration cost and complexity. We also discussed our current challenges with Microsoft technical support; unsurprisingly, Fabric was their recommendation. 🤗
2. Researching the Options Before Moving to Fabric
Fabric was not predetermined as the migration destination. Before starting the migration, we investigated Snowflake, Google Cloud, Azure Databricks, and Microsoft Fabric. We compared their fit with our existing assets, their potential for future Data & AI use cases, and the effort required for migration and operations.
| Platform | What we evaluated | Main concern at the time |
|---|---|---|
| Snowflake | Maturity as a data warehouse and data-sharing capabilities | Whether it needed to sit at the center of our Data & AI strategy |
| Google Cloud | BigQuery, AI integration, and alignment with GA4 for B2C use cases | Cost of establishing another cloud operating model |
| Databricks | Lakehouse, data engineering, and ML / AI | Learning and operating a new platform |
| Microsoft Fabric | Continuity with Azure, Power BI, and existing assets | Maturity as a relatively new service |
This comparison was not based on running every solution at the same production scale. We reviewed official documentation and technical material, and also used LLMs to help organize the information.
Because continuity with the existing environment, ease of migration, and current operational skills carried significant weight, we selected Fabric and began migration validation using F8 and F16 SKUs.* At that point, the choice was reasonable against the criteria we had defined.
SKU: The Fabric subscription tier and its associated capacity units (CUs).
3. What a Production-Oriented PoC Revealed
For the PoC, we selected part of a machine learning pipeline that had already been proven in production. Rather than simply reproducing the existing Synapse processes and MLflow configuration, we redesigned the T-SQL and notebooks to test whether Fabric could support a future Data & AI platform. The redesign focused on:
- Removing unnecessary processing and consolidating duplicate logic
- Clarifying the responsibility of each process
- CI/CD and change management
- Monitoring and reruns
- Reusing data safely from AI applications
As noted earlier, the test was not limited to whether Fabric could reproduce the existing processing. We evaluated whether development, operations, governance, and AI use could work together as a system that the organization could improve continuously.
The PoC did not reveal a single fatal flaw that would justify abandoning Fabric immediately. Instead, smaller gaps appeared across several evaluation areas. Taken together, they raised concerns about the operation of the platform as a whole.
| Evaluation area | What we tested in the PoC |
|---|---|
| Governance | Access control, data catalog, and governance spanning data and AI assets |
| Development and operations | External data integration, source control including notebooks, CI/CD, and operational monitoring |
| AI access to data | Text-to-SQL, unstructured data, secure access from AI agents, and AI-agent-assisted testing and quality improvement |
| AI lifecycle | Evaluation, monitoring, LLMOps, and continuous quality improvement |
Some items met our expectations, while others required additional design or workarounds. Each gap might be manageable in isolation, but accumulating workarounds would increase development and operational complexity and ultimately reduce business agility. These trade-offs became visible only after we built a realistic workload.
4. Is Fabric Still a Maturing Solution?
Microsoft Fabric incorporates the medallion architecture popularized by Databricks and unifies Power BI, Data Factory, Data Engineering, Data Science, and other capabilities around OneLake. Its ability to maximize synergy with Azure services and evolve an existing Microsoft environment into an integrated Data & AI Platform was—and remains—a major attraction.
At the same time, Fabric became generally available in November 2023, and new capabilities continue to be added. Because the platform is still evolving, we could not base an adoption decision on future potential alone. The PoC needed to establish whether the capabilities we required were stable and operable today, and whether Fabric was practical as an AI-ready data platform.
We therefore decided to evaluate Databricks again—not to dismiss Fabric, but to compare how well each option met our current production requirements while preserving the benefits of Azure integration.
5. Why We Revisited Databricks
The central question in our Databricks reassessment was whether governance across data and AI assets could be implemented through a consistent operating model.
In Fabric, Fabric IQ Ontology connects business concepts with data in OneLake and provides a semantic layer that people and AI agents can share. Microsoft Purview, by contrast, handles capabilities such as discovery, classification, lineage, protection, and auditing of data assets. Their purposes are different, but their touchpoints—catalogs, metadata, and governance—span multiple systems. This can make it difficult to determine which system is the source of truth and where each governance responsibility should reside. Ontology is also currently in preview and had not been announced when we performed the original solution comparison.
⚠️ Ontology: A formal representation of business concepts and the relationships between them that enables AI and other systems to interpret the meaning of data.
By comparison, Unity Catalog in Azure Databricks treats tables, models, services, and other resources as securable objects. It brings access control, lineage, auditing, and quality monitoring into a common governance layer for data and AI. Its attribute-based access control (ABAC) can apply row filters and column masks using governed tags.
The issue is not that Fabric lacks governance features. It is that Ontology and governance—both central to an AI-ready platform—have not yet converged into a single operating model. We saw this division of responsibility as a potential constraint on developing and operating AI agents after migration.
Before adopting Fabric for production, we need to determine whether the preview version of Ontology meets our use cases, whether it can be operated consistently with Purview, how future specification changes or alternative approaches would affect operations, and how much the design would depend on the roadmap.
This is why we brought Databricks back into the evaluation. The attraction was not a particular agent or LLM feature, but the ability to centralize permissions, lineage, and auditing for the data and AI assets used by AI in Unity Catalog. Ontology implementation still requires separate validation, but we valued the ability to place the governance baseline in one location.
6. What Made It Possible to Revisit the Decision
The main reason we could revisit the solution choice was that the overall design used a medallion architecture rather than depending on one service. The redesigned notebooks and other source code could also be incorporated into Databricks with relatively little change, keeping the effort required for a comparative PoC manageable.
- Medallion architecture: The Bronze, Silver, and Gold structure and its data flow can be carried over without redesigning the architecture
- Components: Fabric-specific capabilities need to be replaced with their Databricks counterparts
- Network: Destinations and authentication paths must be reconfigured, but the existing Azure resources can be reused, so the effort is relatively modest under our assumptions
- CI/CD: The pipelines were designed independently of Fabric, so their concepts and stages remain unchanged; only initial settings such as destinations, credentials, and deployment targets need to be reconfigured
Separating the long-lived architecture from replaceable components limits the impact of changing solutions. The fact that Azure Databricks runs in the same Azure environment also made it practical to reuse existing resources during the comparison.
7. How We Think About the Sunk Cost
As the project manager, I made the decision to reopen the solution selection. That did not make the time and effort invested in research, migration planning, and the implementation of the machine learning pipeline and CI/CD feel insignificant. Even so, the work was not simply a loss. It allowed us to turn the requirements for an AI-ready data platform into something concrete and to gain insights that desk research alone could not have produced.
My willingness to cut losses when necessary also played a part in revisiting the direction at this stage. But changing course was not our first response. Together with the project leads and team members, we compared ways to resolve the issues with the impact of evaluating another solution.
Ideally, we would obtain an assessment from someone with equally deep production experience across all four candidates—but people with that breadth of expertise are rare.
We did have an opportunity to speak with someone who had worked with both Databricks and Snowflake. Even so, this comparison is based primarily on official information, technical research, and the results of our Fabric PoC. Our current position is therefore not a final conclusion, but a hypothesis to test in the next stage.
8. Next Step
We have not decided to migrate to Databricks. In the next PoC, we plan to implement the same machine learning workload on Databricks and compare its practicality and operational overhead with Fabric under the same conditions. The result will inform—not predetermine—our choice of direction.
Conclusion
The PoC gave concrete shape to the two goals we had separated from the beginning: migration and modernization. Moving existing processing into a new environment is technically possible. Looking several years ahead, however, when AI agents become consumers of enterprise data, fitting the current specification is not enough. The platform must also provide a consistent way to manage ontology and governance.
Our work on Fabric was neither a detour nor a failure. Building the PoC exposed issues that were not visible at the outset. Because the architecture was organized around the medallion model, we can now compare Databricks without discarding the underlying design. The experience also showed us that modernization includes maintaining the freedom to revisit an investment when the long-term objective demands it.
We will draw our conclusion after running the same workload in the next PoC.