Data Engineer

  • No. of positions: 1
  • Chennai
  • Last Active Date : 19 Jul, 2024

Budget

₹210,000 - ₹253,000 (Monthly)

Experience

6 to 15 Years

Work From

Onsite

Job Duration

6-12 Months

Required Skills

PythonSQLPysparkDatabricks CI/CDDevOps

Job Description

Note: Only vendors with an ISO:27001 certificate are eligible to apply and they have Datbrick certified candidates.

Job description:

We are looking for people who are self-motivated, responsive individuals who are passionate about data. You will build data solutions to address complex business questions. You’ll take data through its lifecycle, from the pipeline for data processing, data infrastructure, and create dataset data products.

Core responsibilities:

  • Monitor and optimize the performance of data pipelines and queries, addressing any bottlenecks and improving efficiency.

  • Design, develop, and maintain robust data pipelines using SQL, Python, and PySpark.

  • Partner with business team(s) to understand the business requirements, understand the impact to existing systems and design and implement new data provisioning pipeline process for Finance/ External reporting domains.

  • Monitor and troubleshoot operational or data issues in the data pipelines

  • Drive architectural plans and implementation for future data storage, reporting, and analytic solutions

  • Work closely with data analysts, data scientists, and other stakeholders to understand data needs and deliver high-quality data solutions.

  • Identify, troubleshoot, and resolve data-related issues, ensuring high availability and reliability of data systems.

Qualifications:

  • 5+ years of relevant work experience in data engineering or software engineering equivalent

  • 3+ years of experience in implementing big data processing technology: AWS / Azure / GCP, Apache Spark, Python

  • Experience writing and optimizing SQL queries in a business environment with large-scale, complex datasets

  • Hands-on experience with PySpark for big data processing, including data frame operations, Spark SQL, and Spark Streaming.

  • Proven experience with SQL, Python, and PySpark in a production environment.

  • Experience with big data technologies such as Hadoop, Spark, and Kafka are a plus.

  • Monitor and troubleshoot data pipeline issues to ensure seamless data flow.

  • working experience in a DevOps CI/CD environment

  • Data Brick Certificate is imortent t.o apply for this profile 

Submit Query icon