GCP Data Engineer - Apache Kafka & Event-Driven Systems (US Shift)

About the Role

Our client is transforming their Legacy platform into a modern event-driven system leveraging Google Cloud and Apache Kafka. This greenfield project involves designing and constructing a platform handling event ingestion from Legacy ECC systems and external carriers, with a Kafka streaming backbone processing approximately 10 million daily events. The platform includes a processing engine for rules and a compliant seven-year event archive stored across BigQuery and Cloud Storage.

As a core engineer, you will have full ownership and collaborate closely with a senior Legacy systems expert for domain knowledge, while you provide cloud, Kafka, and engineering expertise.

Key Responsibilities

• Designing and managing a Kafka cluster on Google Cloud, including configuration of topics, partitions, security, and retention policies.
• Developing an HTTPS event ingestion bridge using Cloud Run and Python to receive events from Legacy and external systems.
• Implementing a Kafka consumer-based rules engine in Python that processes events, manages state with Firestore, and sends triggers back to the Legacy system.
• Building Kafka Connect pipelines to archive all events in BigQuery and Cloud Storage with long-term compliance requirements.
• Setting up monitoring, alerts, dead-letter queues, and reconciliation processes ensuring zero data loss across the pipeline.

Required Qualifications and Experience

• A minimum of 5 years in software or data engineering focusing on backend or data platform development.
• At least 2 years hands-on experience with Apache Kafka across topics, partitions, consumer groups, offsets, and delivery guarantees.
• 2+ years working with Google Cloud Platform, particularly Cloud Run or GKE, IAM, VPC, and Cloud Monitoring.
• Proficient in Python for production development including REST APIs, robust error handling, and testing.
• Strong SQL skills with BigQuery, including working with partitioned and clustered tables.
• Experience in building resilient integrations featuring idempotency, retries, dead-letter queues, and ensuring at-least-once message processing.
• Ability to manage systems end-to-end covering design, development, deployment, monitoring, and documentation.

Preferred Skills

• Experience with Google Cloud Managed Service for Apache Kafka or Kafka Connect sink connectors.
• Familiarity with NoSQL databases such as Firestore or Bigtable.
• Knowledge of infrastructure-as-code tools like Terraform.
• Understanding of regulated or compliant data retention practices including WORM and audit trails.

Work Location & Schedule

• Position requires full-time office presence in Kalyan, India.
• The shift is fixed night hours from 6:30 PM to 3:30 AM IST, Monday through Friday, aligned with US business timings.
• A night-shift allowance will be provided.

Educational Eligibility

Applicants should have a B.Tech or B.E. degree in any specialization.