Big Data & AI: intelligent developments
Big Data represent the main trend in the IT field in recent years. Big Data refers to datasets with dimensions and characteristics such that they cannot be easily handled by traditional systems such as, for example, relational databases. This does not mean that the data management tools normally present in companies are incapable of handling big data, but rather that their use can be unproductive in terms of processing time and costs. To this end, traditional tools can be accompanied (never replaced) by new products such as distributed file systems, NoSQL databases, and distributed processing frameworks like Apache Spark and, the now dated, map-reduce. The choice of tools cannot be made superficially and must be based on the business requirements of the company (Polyglot Persistence pattern).
But what characteristics must data have to be considered “Big”? What business advantages can be derived from their processing? What are the main technologies for their storage, processing and management? This course aims to provide an overview of the state-of-the-art technologies and processes for managing Big Data in order to provide learners with practical tools to start introducing these tools into the company.
The topics are described through the presentation of real case studies and examples of how the main tools covered function.
Contents
What is Big Data; how big is Big Data? The main properties: volume, velocity, variety, value, veracity; classification of Big Data; how to identify Big Data; sources of Big Data; what opportunities for business? Big Data compared to traditional tools (RDBMS, DWH, BI, …); the Big Data management process.
Architectural patterns for Big Data storage; use of distributed file systems (e.g. HDFS); NoSQL databases and their classification; criteria for choosing a storage tool based on business requirements; the concept of Data Lake: what it is and how to implement it; how to transfer data to and from the Data Lake; storing data based on business needs; tools for data transfer.
Distributed processing; Map/Reduce and main implementation patterns; tools for Map/Reduce processing (YARN, TEZ, PIG, …); integration with traditional programming languages; Real-time analytics and complex event processing; Using Apache Spark; basic Data Mining concepts; advanced analytics and AI.
Extended tables (row, column, and temporal index); column families and super column families; nested tables; main functionalities; data modelling; architectural models and patterns; application fields; main columnar databases (Cassandra, HBase, ...); practical usage examples.
Use of native query tools and languages; mapping to SQL (Hive, Drill, Impala, ...); integration with RDBMS and traditional Business Intelligence tools.
What is Hadoop; main tools provided; main architectural patterns; main distributions compared.
Architecture, characteristics and functionality of MongoDB, Cassandra, CouchDB and others.
Integration architectures; usable tools.
Secure data storage; multitenancy management; access policies.
What is data governance; governance process; the metadata repository; roles and responsibilities; new professional figures (e.g. Data Scientist).
Prerequisites
Basic knowledge of databases, internet technologies, and distributed applications.
Classroom requirements
- Video projector with a minimum native resolution of 1024x768 (preferably higher)
- Unfiltered internet connection for the teacher's laptop
- Flip chart with markers of different colours (or equivalent tools)
NOTEThe lecturer will use their own laptop, on which all course examples are installed. If this is not possible, it is necessary to agree in advance on the preparation of a PC provided by the client.
Recipients