Hello,
Please find the below requirement details
Position: Data Operations
Location: Syosset, NY
Duration: 5 Months to Hire
Client: 605 LLC
Vendor: Apex
Required Skills
Airflow, python, aws, and SQL
Job Description:
Data Operation Resource must have a broad and deep data skillset. In addition to design and build of modern analytics environments that comprise of raw data stores (data lakes) and cleansed data repositories by batch or streaming data pipelines. The Data Operations Engineer will be responsible for building, supporting and monitoring data pipelines and applications while ensuring the highest levels of performance, availability, security and compliance of data
Responsibilities include:
Support and monitor daily operation of data pipelines in the Data Operations space
Support and optimize desktop and cloud environments for data scientists, data engineers, and data analysts
Build data flows for data acquisition, modeling and aggregation using both batch and streaming paradigms
Consolidate/join datasets to create easily consumable, consistent, holistic information
Be responsible for Data Pipelines meeting their SLAs and the adoption of key practices including: Incident and Problem Ownership (standardized processes from the time an Incident is captured, through remediation, communication with stakeholders and resolution), System Reporting (weekly Incident reports and Systems health reports that cover ETL servers and databases with metrics on utilization, job inventories, system availability and other reports)
Monitor and Escalate (establish practices and communications around escalations from Level 1 support to Level 2 support)
Work on unique and interesting data challenges around architecting, building and managing pipelines that securely process hundreds of terabytes of data.
Work closely with analysts and statisticians to ensure the validity of our processes.
Our engineers are expected to wear a number of hats and have the opportunity to touch all parts of the stack. Our stack includes Apache Spark, Scala, Redshift and an ever-growing list of many other cool technologies.
WHO YOU ARE
Experience wrangling terabytes of big, complicated, imperfect data
Extensive experience designing and implementing ETL pipelines
Experience building and operationalizing large-scale enterprise data solutions, Data Lakes and applications using one or more of AWS data and analytics services
Experience with AWS products (Redshift, EC2, EMR, S3, IAM, RDS, Cloud Watch etc)
Bachelor's degree in Computer Science or a related field (or 4 additional years of relevant work experience)
A strong understanding of data structures, algorithms, and effective software design
Significant development experience with a major modern language (e.g. Java, Scala, Python, Ruby, C/C++, etc.)
Significant experience working with structured and unstructured data at scale and comfort with a variety of different stores (key-value, document, columnar, etc.) as well as traditional RDBMSs and data warehouses
Experience with or interest in AWS Glue, Redshift Spectrum and any other tools that enable data querying at scale
Experience with ETL job automation through Airflow pipelines
Exposure to visualization tools - Tableau, Google Analytics, JIRA, MS Project
Conduct code review with architecture team to ensure standards best practices are followed
Develop automated code deploys with GIT, AWS Lambda and AWS Batch services
Ability to multi-task and manage multiple environments
Provide on-call support and remote troubleshooting
Must work well in an agile, collaborative team environment
PREFERRED QUALIFICATIONS
Master's in Computer Science or a related field
Strong background with data-driven environment, monitoring and evaluation.
Disclaimer
This electronic mail (including any attachments) may contain information that is privileged, confidential, and/or otherwise protected from disclosure to anyone other than its intended recipient(s). Any dissemination or use of this electronic mail or its contents (including any attachments) by persons other than the intended recipient(s) is strictly prohibited. If you have received this message in error, please notify us immediately by reply e-mail or e-mail unsubscribe@imcsgroup.net so that we may correct our internal records. Please then delete the original message (including any attachments) in its entirety. Thank you