About the Incorta Intelligent Ingest Feature for Azure Synapse

Incorta’s Intelligent Ingest accelerates enterprise data to analytics by connecting with enterprise applications, some of which lack APIs. It defines the required relationships and logic for business users, and loads data into Azure Synapse via the Azure Data lake. This streamlines the integration between complex data sources and Microsoft Azure Synapse Analytics, accelerating data mart deployment and automating model design and source schema mapping with Incorta data applications.

Incorta Data Applications, aka Blueprints, present a new, faster, more effective method for data analysis and comprehension. They eliminate the need for traditional data modeling or ETL/ELT processes, yet preserve the granularity of raw data sets. By leveraging Incorta Blueprints, customers gain rapid access to enterprise data, and are able to reduce implementation time by as much as 85 percent.

How Incorta Intelligent Works:

The below Incorta components synergize with Intelligent Ingest to process the data from source systems to Azure Synapse. Let’s understand briefly about each object before we jump into the Incorta Intelligent Ingest.

The pipeline starts with the data connectors extracting the data from different source systems and applications, providing schema information and data structures.

The data loader handles data management, loading and scheduling of the schemas. It also manages metadata, compaction, and logging while loading the data. Data loads into paraquet files on Azure Data Lake Storage Gen2 (ADLS2).

The Schema is metadata about the properties of each physical source table and includes the source or location for the data, the table columns, data types, joins between tables, formula columns (computed and persisted columns), and runtime security filters..

The Business Schema is a logical grouping of one or more physical tables and a subset of physical columns. It is used to create a business semantic layer or business view, and provides end-users with an intuitive entry point into the data.

MVs persist the business views to the data lake. They can also implement complex transformation logic in Scala, R, SQL and Python. Materialized Views are written and debugged in a notebook interface and can read from multiple data sources and include arbitrarily complex logic including Machine Learning algorithms.

Capture best practices and pre-built content for accessing, organizing, and presenting data from popular business solutions. Blueprints contain data connection properties, physical schema descriptions, business views, materialized views, and sample queries. Blueprints slash the time and effort required to bring enterprise application data to Azure Synapse users.

Azure Synapse reads the prepared and user-friendly Parquet files from the shared data lake, making the data available for analytics users. Since the data lake consists of standard Parquet files, companies can optionally access the data from other 3rd party tools, such as Databricks to implement Machine Learning on Spark.