Design & Develop batch and streaming solutions using various Big Data technologies.
Have experience working with real-time or near-real-time ingestion.
Streamline the intake of the raw data into our Azure Data Lake.
Assessing technologies and approaches for ingestion, transformation, and storage.
Design and build frameworks for various common data engineering processes like ingestions, mastering, data cleansing, and data masking.
Prepare and present potential technical designs and solutions for Data pipelines and DataOps in Azure cloud environment.
Collaborate and partner with business domain leads, data scientists, product owners, enterprise architects and other functional leads to deliver world class data solutions.
Diagnose and address complex problems, including performance issues, scale, and drive to resolution to meet business initiatives.
Perform production support and deployment activities.
Build data pipelines from systems such as CRM, Ecommerce etc. with the emphasis on scalability and reliability.
Understand and translate business needs into data models to support long-term, scalable, and reliable solutions.
Create logical and physical data models using best practices to ensure high data quality and reduced redundancy
Continuously improve our data infrastructure and stay ahead of technology.
Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs.
8 – 10 years of experience in data engineering and designing scalable ETL pipelines
Advanced Proficiency in Hadoop, Scala, Spark, Kafka and SQL
6+ years of experience with cloud platforms like Azure, AWS, Google
Be skilled at architecting large and complex data pipelines.
Experience performing data analysis and data exploration.
Experience working with Cloud technologies, Azure, Databricks, ADLS, Spark, Cosmos DB, and other big data technologies.
Experience working in an agile delivery environment.
Experience working in a multi-developer environment, using version control.
Excellent knowledge in data structures and design patterns
Preferred experience with data integration tools
Work with cross-functional agile teams to drive projects through the full development cycle.
Help the team improve with the usage of data engineering best practices.
Write SQL for processing raw data, data validation and QA
Knowledge working with APIs to collect or ingest data
Strong Database knowledge, SQL & No-SQL preferred
Communication Skills Data Engineers are part of a team, working with database administrators, data analysts and management and need to be effective communicators.
Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly.
Experience integrating and building data platform in support of BI, Analytics, Data Science, and real-time applications.
Strong communication skills, with the ability to initiate and drive projects proactively and accurately
