AWS Databricks Specialist

Role Summary

We are looking for an experienced AWS Databricks Specialist with 5 to 7 years in IT and hands-on expertise in Databricks on AWS. The position involves designing, optimizing, and maintaining cloud data solutions leveraging advanced Databricks features.

Key Responsibilities

• Work closely with end users to identify challenges and propose effective Databricks-based solutions.
• Evaluate new Databricks functionalities, perform proof of concepts, and develop standard best practices documentation.
• Provide support for Spark-related issues, conduct code reviews, and offer technical recommendations.
• Lead and mentor junior team members through technical guidance and problem-solving assistance.
• Manage midsize teams ensuring delivery quality, deadlines, and adherence to project standards.
• Take accountability for assigned tasks within an Agile work environment, striving for continuous improvement.
• Create and maintain design and solution documentation aligned with specified requirements.
• Implement job cost optimization and performance tuning to maximize system efficiency.
• Support CICD pipelines and infrastructure automation efforts for seamless deployment.
• Participate fully in Agile ceremonies to enhance team collaboration and productivity.

Required Qualifications and Skills

• 5 to 7 years of IT experience with substantial hands-on use of Databricks on AWS.
• Proficiency with Databricks capabilities including Delta Lake and Unity Catalog for scalable data processing.
• Strong knowledge of AWS cloud services such as S3, Lambda, SNS, Step Functions, and Glue.
• Expertise in optimizing job costs to reduce runtime and improve performance.
• Experience with Python core libraries and PySpark for data engineering tasks.
• Good analytical and logical skills for writing SQL queries, stored procedures, and designing marts/views within RDBMS.
• Understanding of ETL processes and complete data pipeline architectures from source to data warehouse and reporting layers.
• Familiarity with CICD pipelines for code deployment utilizing tools like Terraform.
• Experience with version control systems including GitHub or equivalent platforms.
• Strong grasp of Agile methodologies and software development lifecycles.
• Excellent communication and presentation abilities for effective stakeholder collaboration.