Explore the transformative power of data platforms in Part 4 on Industrial Data Platforms. Learn how achieving exponential growth is possible by breaking the data barrier with a unified approach.
Why 2 data platforms (operational and enterprise), instead of just one ? Modern stacks like Databricks or MS Fabric can handle it in my opinion. It seems to me that the key is to have one centralized platform for both OT and IT data and a strong collaboration between OT and IT professionals around de governance of it. I’m really interrested to better your vision.
Both Databricks and Fabric are not the storage layer, they can perfectly consume data from an OT-native platform as well as an IT-native platform.
If we take a look to the storage layer, I still see a lot of technical issues today when moving OT data at scale to cloud hyper scalers. Historians are extremely efficient in both data ingestion at scale as well as data consumption. When we compare bigger workloads between an historian and - let's take Azure ADX - we see very high storage and consumption costs. Also, these systems are typically just general purpose blog stores with limited to know build-in capabilities to work with sensor data.
There are many cloud native and on-prem OT data platforms available on the market. Some have limited functionality, some are way more advanced when it comes to contextualization.
From an organizational perspective, I also see a lot of challenges in most organizations. I personally haven't seen too many examples of data governance across IT and OT (if it exists, typically only a few use cases are considered)
However, I am _very_ interested in finding and speaking to those companies who have build integrated data platforms and introduced governance models. I'd love to see how these lighthouse companies pave the way forward for the rest.
I'd like to add another point to David's comment, specifically about tooling. There are excellent tools available for reporting and dashboarding, such as Tableau and PowerBI. However, when your focus shifts from analyzing relational data to time series data, you need different tools. These time series-focused tools capitalize on the strengths of time series databases and perform well at scale.
Transitioning everything to a platform like ADX would necessitate significant engineering effort and introduce risks related to maintaining the same levels of performance and stability at scale.
That said, I am confident that these challenges are surmountable, and we will eventually integrate everything into one platform
Yeah I don't say it is easy to merge the 2 platforms, I just say it is in my opinion technically possible in 2024 and probably a godd investment for the long term, especially if there is already a good collaboration culture between OT and IT in your organization (I'm lucky this is the case in mine :D)
This is what I challenge a bit, 5 years ago I would have agree with you, but technology evolve very quickly and tooling like Azure Data explorer (kusto engine) can compete with historian on time series analytics in term of performance. I really think that emergence of platformes like Cognite, or some AVEVA tooling can be leverage as a contextualization layer on top of enterprise data platforms and ensure we don't duplicate data in 2 platforms.
2 platforms = data silos = dependencies and complexity, and this is probaly why data governance cross IT/OT is complexe.
If you remove the perforance of historian for time series / real time analytics of the equation, could you elaborate on what is missing in modern data stack like Databricks / Fabric or Azure PaaS data service to replace an operational data platform ? (Yes there is probably some NERC/CIP / NIST compliancy parameters to take into account as well).
Why 2 data platforms (operational and enterprise), instead of just one ? Modern stacks like Databricks or MS Fabric can handle it in my opinion. It seems to me that the key is to have one centralized platform for both OT and IT data and a strong collaboration between OT and IT professionals around de governance of it. I’m really interrested to better your vision.
In the years to come, I definitely see both platforms merging (see also https://itotinsider.substack.com/i/138224053/crystal-ball-how-the-future-of-an-operational-data-platform-might-look-like )
Both Databricks and Fabric are not the storage layer, they can perfectly consume data from an OT-native platform as well as an IT-native platform.
If we take a look to the storage layer, I still see a lot of technical issues today when moving OT data at scale to cloud hyper scalers. Historians are extremely efficient in both data ingestion at scale as well as data consumption. When we compare bigger workloads between an historian and - let's take Azure ADX - we see very high storage and consumption costs. Also, these systems are typically just general purpose blog stores with limited to know build-in capabilities to work with sensor data.
There are many cloud native and on-prem OT data platforms available on the market. Some have limited functionality, some are way more advanced when it comes to contextualization.
From an organizational perspective, I also see a lot of challenges in most organizations. I personally haven't seen too many examples of data governance across IT and OT (if it exists, typically only a few use cases are considered)
However, I am _very_ interested in finding and speaking to those companies who have build integrated data platforms and introduced governance models. I'd love to see how these lighthouse companies pave the way forward for the rest.
I'd like to add another point to David's comment, specifically about tooling. There are excellent tools available for reporting and dashboarding, such as Tableau and PowerBI. However, when your focus shifts from analyzing relational data to time series data, you need different tools. These time series-focused tools capitalize on the strengths of time series databases and perform well at scale.
Transitioning everything to a platform like ADX would necessitate significant engineering effort and introduce risks related to maintaining the same levels of performance and stability at scale.
That said, I am confident that these challenges are surmountable, and we will eventually integrate everything into one platform
Yeah I don't say it is easy to merge the 2 platforms, I just say it is in my opinion technically possible in 2024 and probably a godd investment for the long term, especially if there is already a good collaboration culture between OT and IT in your organization (I'm lucky this is the case in mine :D)
This is what I challenge a bit, 5 years ago I would have agree with you, but technology evolve very quickly and tooling like Azure Data explorer (kusto engine) can compete with historian on time series analytics in term of performance. I really think that emergence of platformes like Cognite, or some AVEVA tooling can be leverage as a contextualization layer on top of enterprise data platforms and ensure we don't duplicate data in 2 platforms.
2 platforms = data silos = dependencies and complexity, and this is probaly why data governance cross IT/OT is complexe.
If you remove the perforance of historian for time series / real time analytics of the equation, could you elaborate on what is missing in modern data stack like Databricks / Fabric or Azure PaaS data service to replace an operational data platform ? (Yes there is probably some NERC/CIP / NIST compliancy parameters to take into account as well).