Data Engineer

As a Data Engineer, your main goal is to build and maintain the systems that process and store data for the Events & Exhibitions ecosystem. You will take scattered, vendor‑specific data (from registration systems, apps, marketing tools) and transform it into a unified, AI‑ready dataset using a Medallion Architecture on Azure. Think of it as organizing raw data into a structured pipeline that’s ready for analysis and machine learning.

• Data Ingestion & API Integration (Bronze Layer)
  Build and manage robust ETL/ELT pipelines using Azure Data Factory to ingest data from 3rd‑party vendors (REST APIs, Webhooks, SFTP). Ensure raw data is landed securely in Azure Data Lake Gen2 (Bronze Layer) without data loss. Implement error‑handling and logging to monitor the health of real‑time and batch ingestion jobs.
• Transformation & Modeling (Silver & Gold Layers)
  Utilize PySpark (Azure Databricks/Sy...