직무 설명

Role Overview

This position is for a data migration specialist who will help modernize enterprise data platforms by moving workloads from Amazon Redshift and AWS Glue into Databricks. The role focuses on rebuilding and improving ETL/ELT processes, protecting data quality, and ensuring the migrated solution performs reliably at scale.

Key Responsibilities

Review existing data pipelines, ETL processes, and warehouse objects currently running in Amazon Redshift and AWS Glue.
Plan and execute migration approaches for transferring data and workloads to Databricks.
Build, tune, and support ETL/ELT pipelines using Databricks, Apache Spark, and Python.
Rework SQL scripts, stored procedures, and transformation logic so they function effectively in Databricks-based solutions.
Check data accuracy, completeness, and consistency before migration and after cutover.
Improve Databricks data models, partitioning strategy, and overall processing performance.
Apply Delta Lake practices for storage efficiency, version control, and optimization.
Work closely with architects and business stakeholders to ensure the solution aligns with requirements.
Investigate migration issues and resolve performance bottlenecks.
Prepare technical documentation, migration playbooks, and handover material.
Maintain compliance with security, governance, and regulatory standards during the migration process.

Requirements

A bachelor’s degree in Computer Science, Information Technology, Engineering, or a similar discipline.
At least 4 years of experience in data engineering or data platform migration work.
Practical expertise with Amazon Redshift, AWS Glue, Databricks, Apache Spark, Python, and SQL.
Working knowledge of cloud environments, ideally AWS.
Strong understanding of data warehousing principles and dimensional modeling.
Hands-on experience with Delta Lake and Lakehouse architecture.
Exposure to ETL/ELT frameworks and workflow orchestration tools.
Experience using Git and CI/CD pipelines.
Good analytical thinking and troubleshooting ability.
Strong written and verbal communication skills, including documentation.
Preferred exposure to Databricks Workflows, Unity Catalog, and Delta Live Tables.
Familiarity with Apache Airflow or similar orchestration tools is an advantage.
Experience with Infrastructure as Code tools such as Terraform or CloudFormation is a plus.
AWS and/or Databricks certifications are beneficial.
Background in large-scale enterprise data migrations is preferred.
Minimum overall experience required: 4 to 8 years in data engineering.
Relevant migration experience required: at least 2 years moving cloud data platforms to Databricks.

Technical Focus

Cloud platform: AWS
Data warehouse: Amazon Redshift
ETL tool: AWS Glue
Target platform: Databricks
Programming languages: Python and SQL
Processing frameworks: Apache Spark and PySpark
Storage layer: Delta Lake
Version control: Git
CI/CD tools: Azure DevOps, GitHub Actions, or Jenkins

Success Measures

Move Redshift databases and AWS Glue ETL jobs successfully into Databricks.
Deliver accurate reconciliations with very low data mismatch rates.
Improve performance and scalability of pipelines on the Databricks Lakehouse platform.
Provide complete migration documentation and support a smooth operational transition.
Finish the migration within the planned schedule while keeping business disruption to a minimum.

Employment Details

This is a full-time or contract opportunity depending on project needs.

The position is based in Hyderabad, India.

Additional Information

This opportunity is best suited to a data engineer who has worked on cloud data migrations and modernized enterprise platforms using Databricks.

Data Migration Developer

당신이 일하게 될 곳