Advance Data Engineer

Job Summary
This job posting is for an existing, active vacancy and We are looking to hire Advanced Data Engineer, immediately who has experience in Databricks, Python, Pyspark, SQL & Apache Airflow..

Role Overview:

We are seeking an Advanced Data Engineer with strong expertise in the Databricks ecosystem to join our data engineering team. The ideal candidate will be responsible for designing, developing, and optimizing robust data pipelines and frameworks that support data analytics, machine learning, and reporting initiatives. You will play a key role in ensuring data governance, observability, and automation within a modern data stack.
In addition, the role requires strong skills in SQL, DBX (Databricks), PySpark, Data Engineering fundamentals, and experience with workflow orchestration tools such as Apache Airflow.

Job Responsibilities

• Understand, analyze, and contribute to the current Databricks architecture and design principles, ensuring scalability and performance.
• Develop and maintain efficient data processing scripts using Python and PySpark, ensuring clean, reusable, and scalable code.
• Demonstrate a deep understanding of datasets, including structure, lineage, semantics, and business context.
• Use GitHub for version control and collaborate effectively using GitHub Actions for automating workflows and CI/CD pipelines.
• Configure and maintain CI/CD pipelines in a DevOps environment for seamless code integration and deployment.
• Leverage AI coding assistants like GitHub Copilot and Databricks Assistant to improve development efficiency and code quality.
• Collaborate with cross-functional teams including data scientists, analysts, and platform engineers.
• Utilize advanced SQL for data transformation, analysis, and troubleshooting across large-scale datasets.
• Apply strong data engineering principles to design, optimize, and maintain scalable ETL/ELT processes.
• Build and manage data workflows using Apache Airflow or similar orchestration tools to ensure reliable automation and scheduling.
• Work extensively within the DBX (Databricks) environment to develop scalable pipelines and enforce best practices across the platform.

Required Qualifications

• 5+ years of experience in data engineering or related roles.
• Proficient in Python and PySpark, with a strong foundation in distributed data processing.
• Hands-on experience working with Databricks (DBX), including workspace administration and Unity Catalog integration.
• Strong understanding of data security and governance best practices.
• Proficiency in SQL, including complex queries, optimization, and performance tuning.
• Experience with monitoring tools such as Datadog for data system observability.
• Proficiency in Git/GitHub, including pull requests, branching strategies, and GitHub Actions.
• Experience with DevOps practices related to CI/CD, especially in data pipeline deployments.
• Familiarity with AI-powered coding tools such as GitHub Copilot and Databricks Assistant.
• Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.
• Experience in workflow orchestration, preferably with Apache Airflow.

Preferred Skillset

• Databricks or Azure certifications are a plus.
• Experience in cloud platforms (Azure) in a data engineering context.
• Familiarity with modern data stack tools and frameworks.
• Excellent communication and documentation skills.

The pay range for this role is $100,000 - $120,000* per annum including any bonuses or variable pay. Tech Mahindra also offers benefits like medical, vision, dental, life, disability insurance and paid time off (including holidays, parental leave, and sick leave, as required by law). Ask our recruiters for more details on our Benefits package. The exact offer terms will depend on the skill level, educational qualifications, experience and location of the candidate.

AI tools may assist in the recruitment process; however, all hiring decisions are made by the recruitment team based on a comprehensive evaluation of candidates.

"Tech Mahindra is an Equal Employment Opportunity employer. We promote and support a diverse workforce at all levels of the company. All qualified applicants will receive consideration for employment without regard to race, religion, color, sex, age, national origin, or disability. All applicants will be evaluated solely on the basis of their ability, competence, and performance of the essential functions of their positions with or without reasonable accommodations. Reasonable accommodations also are available in the hiring process for applicants with disabilities. Candidates can request a reasonable accommodation by contacting the company ADA Coordinator at [email protected]