Data Engineer - Databricks

Role Overview

We are seeking a skilled Data Engineer to join a pivotal transformation project aimed at developing a fully automated, SOX-compliant financial reporting system. This project integrates an Azure Databricks Lakehouse environment with Workday General Ledger and offers a chance to work closely with industry experts while gaining comprehensive experience with enterprise-level Databricks architectures.

Responsibilities

• Design and build robust ETL pipelines using Azure Databricks, PySpark, and Python optimized for production environments.
• Manage Delta Lake infrastructures and implement Unity Catalog governance frameworks.
• Create and maintain Bronze, Silver, and Gold Lakehouse data pipelines ensuring clear data lineage and auditability.
• Develop financial data transformation workflows, KPI-to-General Ledger mappings, and reconciliation procedures.
• Establish automated data quality frameworks covering validation, completeness verification, duplicate detection, and threshold control mechanisms.
• Develop REST and SOAP API integrations equipped with comprehensive error handling and monitoring solutions.
• Institute SOX-compliant controls, including audit trails, approval workflows, and governance protocols.
• Assist in testing phases, user acceptance testing (UAT), reconciliation tasks, and ongoing production oversight.
• Create and maintain operational dashboards and reporting layers tailored to Finance and business stakeholder needs.
• Produce detailed technical documentation, runbooks, and handover materials to ensure sustainable operations.

Required Qualifications and Experience

• Bachelor’s degree in Computer Science, Data Engineering, IT, Mathematics, or similar disciplines.
• Minimum of 3 years of professional experience in Data or Analytics Engineering roles.
• Proficiency with Azure Databricks, Databricks Workflows, Delta Lake, Unity Catalog, PySpark, Python, and SQL.

Additional Experience

• Familiarity with Azure Data Lake Storage Gen2 (ADLS), Azure Key Vault, and Azure Data Warehouse systems.
• Experience integrating APIs using REST and SOAP protocols.
• Strong grasp of Lakehouse architecture and layered data modeling (Bronze, Silver, Gold).
• Competence in constructing production-quality ETL workflows and comprehensive data quality monitoring frameworks.
• Excellence in technical documentation and effective communication.

Desired Professional Qualities

• Demonstrates ownership and accountability for timely project delivery.
• Effectively communicates across technical and business teams.
• Adapts rapidly to dynamic, fast-moving work environments.
• Exhibits meticulous attention to detail.
• Collaborates well within multidisciplinary, cross-functional groups.
• Understands compliance requirements, governance standards, and financial reporting imperatives.