Provide technical direction in the creation, implementation, and integration of complex big data solutions
Collaborate and translate business requirements into implementation design, and participate in the design, evaluation, and selection of the best software solutions to achieve business objectives
Design, develop, test, and deploy scalable and reliable data pipelines using technologies like Spark, Kafka, Hadoop, and other big data technologies
Extract, transform, and load data from various sources and formats, such as relational databases, APIs, JSON, XML, CSV, etc.
Perform data quality checks and validations, and handle data anomalies and issues
Collaborate with other developers, data engineers, data analysts, and business stakeholders to understand data requirements and deliver data solutions
Develop scalable systems that enable real-time validation of identified resources
Mentor junior developers and engineers on the team and champion industry best practices
Explore, evaluate, and conduct PoC to determine new technology solutions that would improve our ability to detect newer threats or add efficiency to current processes
Requirements & Skills:
7+ years of experience. Master’s or Bachelor’s degree in Computer Science, Engineering, Mathematics, or related field, or equivalent work experience
Can demonstrate past experience in developing and working with big data technologies, such as Spark, Kafka, Hadoop, Hive, etc.
Proficient in programming languages, such as Python, Java, Scala, or SQL
Strong analytical and problem-solving skills, and attention to detail
Good communication and teamwork skills, and the ability to work independently and proactively
Eager to learn new skills and technologies, and willing to take on challenges
