Data Engineer - Python AND Kafka AND (Hadoop OR HDFS OR Hive) AND Snowflake AND apache AND (iceberg

Responsibilities:
 
Engineer will be part of the datastore-migration Factory team that will be responsible to perform for the end-to-end datastore migration from on-prem DataLake to AWS hosted LakeHouse. This is a high visibility and crucial project for Goldman Sachs.
Responsibilities of the Engineer includes:
 
Pipeline Migration
Logic & Scheduling: Refactoring and migrating extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
Data Transfer: Executing the physical migration of underlying datasets while ensuring data integrity.
Stakeholder Engagement: Acting as a technical liaison to internal clients, facilitating handoff and sign-off conversations with data owners to ensure migrated assets meet business requirements.
 
Consumption Pattern Migration
Code Conversion: Translating and optimizing legacy SQL and Spark-based consumption patterns (raw and modeled) for compatibility with Snow...