Course Outline
Introduction to Apache Iceberg
- Overview of Apache Iceberg capabilities
- Recap of fundamental concepts
In-Depth Analysis of Iceberg Architecture
- Detailed examination of the Iceberg table format
- Comprehensive architecture overview, focusing on metadata and file layout
- Internal mechanisms for schema and partition evolution
Advanced Installation and Configuration
- Configuring Iceberg for peak performance across different environments
- Integration with various data processing engines
- Advanced setup involving security, encryption, and access controls
- Setting up Iceberg within a distributed environment
Advanced Operations and Maintenance
- Managing large-scale Iceberg tables
- Implementing and managing complex schema modifications
- Handling partition evolution and hidden partitioning strategies
- Executing advanced CRUD operations involving schema and partition changes
Query Optimization Techniques
- Strategies for reducing query latency
- Utilizing partition pruning and file pruning
- Employing metadata caching and optimization strategies
- Implementing and testing query optimization techniques
Performance Tuning for Large Datasets
- Optimizing performance for large-scale datasets
- Leveraging Iceberg's built-in features for performance tuning
- Case studies on performance tuning in real-world scenarios
- Fine-tuning performance for extensive datasets
Advanced Data Migration and Integration
- Migrating complex data structures from other systems
- Integrating Iceberg with real-time data streams
- Migrating complex datasets and integrating real-time data streams
Reliability and Consistency
- Ensuring data consistency and integrity in distributed environments
- Implementing and managing transactional guarantees
- Managing failures and recovery mechanisms
- Implementing reliability and consistency features
Advanced Features and Customization
- Developing custom catalog implementations
- Extending Iceberg with custom features
- Implementing custom catalogs and extending Iceberg functionalities
Data Governance and Compliance
- Establishing data governance policies
- Ensuring compliance with data regulations
- Managing audit trails and data lineage
- Implementing governance and compliance features
Summary and Next Steps
Requirements
- Proficiency in core concepts, basic operations, and Iceberg table management
Target Audience
- Data engineers
- Data architects
- Data analysts
- Software developers
Testimonials (2)
A journey through the Spark world: a very intense course. DSL, spark sql, partitioning vs bucketing for me.
Georgiana Elisabeta
Course - Apache Spark Fundamentals
Hands on exercises. Class should have been 5 days, but the 3 days helped to clear up a lot of questions that I had from working with NiFi already
