Data Engineer - Python
hyderabad
We are searching for an experienced Python Data Engineer with a strong focus on ETL processes and API development. This role is pivotal in designing and implementing efficient, scalable ETL pipelines and developing APIs to facilitate data access and manipulation. The ideal candidate will have a robust background in data engineering principles, ETL design, and web services development using frameworks like Flask or Django.
Key Responsibilities:
- Design, build, and maintain efficient, reliable, and scalable ETL pipelines using Python to extract, transform, and load data from various sources into data stores.
- Develop RESTful APIs using Flask or Django to enable seamless data access and manipulation by internal and external systems.
- Ensure optimal data processing architecture by implementing effective database solutions and data storage practices.
- Collaborate with the engineering team to integrate ETL processes and APIs into application ecosystems.
- Monitor, troubleshoot, and optimize data systems to ensure reliability and performance.
- Manage data security, backup, and recovery specifications to ensure data integrity and availability.
- Document ETL processes, API endpoints, and data models, maintaining clear and concise documentation for all developed systems.
- Keep abreast of new technologies and advocate for their adoption, where they can enhance the functionality and efficiency of data systems.
Qualifications:
- Bachelor's or master's degree in Computer Science, Engineering, Information Technology, or a related field.
- 3+ years of proven experience in data engineering with a strong emphasis on ETL pipeline design and API development.
- Expertise in Python, with extensive experience using data processing libraries (e.g., Pandas, NumPy) and ETL tools.
- Proficient in developing RESTful APIs with Flask or Django and familiar with API authentication and authorization mechanisms.
- Solid understanding of SQL and experience with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
- Experience with cloud services (e.g., AWS, Google Cloud, Azure) and their data services is highly desirable.
- Strong knowledge of data warehousing concepts, modeling, and schema design.
- Excellent problem-solving skills and the ability to manage complex projects with tight deadlines.
- Strong communication skills, capable of working collaboratively with technical and non-technical team members.