The ideal Big Data Engineer candidate that will work on collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.
Develop custom batch-oriented and real-time streaming data pipelines, migrating flows from ELT to ETL
Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, etc.
Act in a technical leadership capacity: Mentor junior engineers and new team members, and apply technical expertise to challenging programming and design problems
Resolve defects/bugs during QA testing, pre-production, production, and post-release patches
Conduct design and code reviews
Analyze and improve efficiency, scalability, and stability of various system resources
Contribute to the design and architecture of the project
Operate within Agile Development environment and apply the methodologies
Bachelor's degree in Computer Science or similar field of study, preferred
Demonstrate advanced knowledge of data architectures, data pipelines, real time processing, streaming, networking, and security
Experience developing ETL processing flows using MapReduce technologies like Spark and Hadoop
Experience developing with ingestion and clustering frameworks
Experience with building stream-processing systems, using solutions
Experience with various messaging systems, such as Kafka or RabbitMQ
Good knowledge of Big Data querying tools, such as Pig or Hive
Good understanding of Lambda Architecture, along with its advantages and drawbacks
Proficiency with MapReduce, HDFS
Experience with integration of data from multiple data sources
Ability to solve any ongoing issues with operating the cluster
The company is an equal opportunity employer and will consider all applications without regards to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.
OpenArc is a technology consulting firm providing industry-leading technical talent placement, software development, and technology strategy services to clients nationwide. Through a unique blending of people and software, OpenArc has a business practice that delivers amazing enterprise, mobile and consumer-facing apps and the best talent for contract, contract-to-hire and direct placements for clients and partners alike.
Staffed with the most-trusted recruiting experts, elite software developers, UI/UX designers and market experts, our team provides clients with the best resources, the right techniques and world-class support resulting in powerful measurable success.