Hot off the success of FabCon in Las Vegas in March, the team behind the European Microsoft Fabric Community Conference in Stockholm pulled together a fantastic week of Fabric news, announcements, sessions, and networking for over 3000 people.
The sheer number of attendees at this first event, organised in just five months is a testament to the interest and the curiosity people and organisations have around Fabric. I presented on data sharing in Fabric and the audience participation, with questions and use cases really echoed this.
With so many of the Fabric product teams at the event presenting, and supporting with "Ask the Experts" sessions, it's easy to see the pull to get questions answered. It also means we get some huge announcements and leaps forward in the product itself. There was a lot of updates, new features, bits going generally available. You can read about most in the September news roundup.
I made an Advancing Fabric video summarising my top 10 announcements from FabCon Europe but if you missed it or prefer to read about them instead, here's the news I'm most excited about.
In the grand picture of announcements this is a relatively small one but for automation this is a huge step forward in securely interacting with Fabric's APIs and ties in nicely with my number two announcement below.
Expect this to apply to most APIs including your workspace management APIs, Capacity APIs, OneLake security and shortcut APIs and any CRUD Apis– Create, read update & delete across multiple items.
Another big one for automation, but in the infrastructure as code (IaC) space. There is now an official Terraform provider for Microsoft Fabric.
If you’re not familiar with Terraform, it’s an IaC tool for deploying and managing resources, and not just in Azure, for AWS, GCP and so much more. The investment from Microsoft here instead of purely in Microsoft’s alternative Bicep, is another step in the right direction for me with Fabric leaning into tools and technologies that don't force you into a purely Microsoft tech stack.
Many organisations already use Terraform for their cloud infrastructure so this will make integration and adoption of Fabric item automation that much easier.
It's currently in Public Preview so use in Production at your own risk.
Switching directions a little, this is one that Microsoft are describing as part of their modern data mesh architecture approach in Fabric. Sitting alongside domains, sub-domains, and workspaces in Fabric, its another tool to help govern and organise your data assets.
Essentially this is additional metadata you can define for your Fabric items, meaning if you have a workspace that has items spanning multiple projects or a lakehouse, semantic model, and reports specific to this year's reporting, you can now attach these tags to make it easier to organise and find the things you need.
Im interested in taking this for a spin to see how much value it actually adds and see if there is any integration with Purview for these tags to have a more seamless governance approach across the platform.
This is promised for the end of this year with a few new additions in the Real Time Intelligence experience now supported:
I am excited by all items coming under source control, but this definitely should have been in place already and this is it only just going generally available (Most items are still in public preview though). Driving adoption of good source control practices is difficult and having to work around gaps only creates more obstacles.
This is on top of the huge announcement that Fabric Runtime 1.3 is now GA which brings Spark 3.5 and Delta 3.2. It also brings the Native Execution Engine in public preview. Ok, so I'm bundling two into one here, but it is a huge deal.
The Native Execution Engine is Microsoft’s new completely rewritten (in C++) spark engine. A bit like Databricks' photon engine, you see the biggest improvements in performance of complex transformations and intensive queries. We'll definitely need to compare these side by side.
The icing on the cake for me on this one is that there’s no extra cost to using this and no code changes are needed.
Depending on your workloads, this might be a pretty big announcement for you. It's a great addition to the Data Warehousing experience. For me, The ability to have queries, results, and markdown commentary in one place is a great way to tell a story and layout the full process of data analysis. Less important in day-to-day data processing.
Ok so I've buried this one a bit down the list. Working with so many customers already using Databricks and Unity Catalog, this is a very big one for visibility and integration between Fabric and Databricks and I've been waiting to talk about this for a long time.
You can now bring your Unity Catalog tables into Fabric. This is essentially a shortcut in Fabric to those tables. No data movement happens despite the “mirrored” name, which is a tad frustrating.
It supports managed and unmanaged tables though there are some types not supported like Delta Sharing tables. I talked about this in my FabCon session about "shortcut inception" not being possible with External Data Sharing and it's the same here.
We've all been there. You create a data pipeline (in whichever flavour of Data Factory you have), and you add a single Copy Data activity to it and that's it. The dedicated copy job item is here as an alternative. It is a standalone item in your Fabric workspace that you can use for copying data more easily than creating a pipeline, if you don't intend on using any additional activities. It's in public preview
You also have the ability to do incremental loading too which means it will keep your data up to date.
There's not a lot to say here but this has been a long time coming. It's not just dark mode though, but the ability to customise the appearance of Power BI Desktop from dark, to light, or mirror your system default. No more seared eyeballs when you're creating pie charts at 1am.
Bringing it full circle back to my theme of automation, this is another big one for me. If you're building metadata driven pipelines or any form of scalable automation in Fabric, you need to consider the impact of running the same notebook several times as you pass each table through the process. High concurrency mode in pipelines means we are no longer waiting for additional sessions to start which can be significant if you're using custom pools.
That's just ten out of what felt like hundreds of announcements during and in the lead up to FabCon Europe and I have definitely glossed over a lot of the Copilot news here. Some equally big announcements that I didn't cover here include:
Fabric Data Pipeline support in On-premises Data Gateways is going GA.
There is a Fabric SKU calculator coming – currently in private preview.
Mirroring in Snowflake is now also GA
The REST APIs for OneLake shortcuts are now also GA
For network security enhancements we have more managed private endpoints, for Eventstream this time.
The European Microsoft Fabric Community Conference was a testament to the platform's growing influence and potential. Although we didn't see any groundbreaking announcements there was a lot of really important quality of life and functionality updates and that's exactly what I was looking for. Fabric needs to mature. We need the gaps filled and the bugs squashed to see the platform as a whole Enterprise ready and I'm eager to explore the integration between Databricks and Fabric more closely.