Azure Databricks Architect

Role overview

Tata Consultancy Services is hiring an Azure Databricks Architect for a pan-India virtual engagement. The role calls for an experienced cloud data professional with 8 to 10 years of background in data architecture, Azure data services, and large-scale pipeline development.

Core expertise

The position requires strong working knowledge of data lake architecture along with hands-on experience across Azure services such as ADLS, Azure Data Factory, Azure Databricks, and Synapse. A solid grasp of lakehouse concepts, Databricks Delta, and Delta Live Tables is expected, along with practical exposure to SQL and PySpark/Python-based data processing.

Key technical work

• Build and maintain ELT pipelines using Azure Data Factory and Databricks, including Autoloader-based ingestion, notebook-driven scripting, and Synapse activities such as Copy and Data Flow tasks.
• Design metadata-driven pipelines with robust metadata management and dynamic processing logic.
• Work with Azure Data Lake Storage and Azure Serverless SQL Pool for scalable data storage and querying.
• Transform data using Spark and SQL while following cloud design patterns and established architecture best practices.
• Use Git or comparable version control tools to manage code safely and consistently.
• Investigate, troubleshoot, and resolve issues across ETL workflows and pipeline executions.
• Understand Azure Databricks and Azure Synapse capabilities, internals, and feature sets in depth.
• Support Azure DevOps-based CI/CD practices for streamlined delivery and deployment.
• Apply data profiling, validation, cleansing, and quality checks to improve reliability of data outputs.

Additional expectations

Applicants should also be comfortable working with cloud-native solution patterns, debugging complex data flow issues, and contributing to reliable, maintainable enterprise data platforms.