Introduction At IBM, work is more than a job - it's a calling: To build.
To design.
To code.
To consult.
To think along with clients and sell.
To make markets.
To invent.
To collaborate.
Not just to do something better, but to attempt things you've never thought possible.
Are you ready to lead in this new era of technology and solve some of the world's most challenging problems? If so, lets talk.
Your Role and Responsibilities We are seeking highly skilled Senior Engineers to join our team working on the Apache Gluten project, a cutting-edge initiative aimed at enhancing the performance of JVM-based SQL engines like Apache Spark by offloading execution to native engines.
This role requires deep expertise in distributed systems, native code development, and familiarity with big data frameworks.Architecture & Development: Design, develop, and optimize the core components of the Apache Gluten project, focusing on the transformation of Spark SQL's physical plans to Substrait plans, and their execution using native libraries like Velox and Click House.
Lead efforts to offload performance-critical data processing tasks from JVM-based SQL engines to native libraries, improving overall query execution efficiency.
Work on integrating Gluten with various backends and frameworks, such as Apache Arrow for data format standardization, and extend support for additional backends and file formats.
Collaborate with other contributors, including those from leading tech companies like Intel and Kyligence, to enhance the project's functionality and ensure robust performance across various environments.