Responsibilities:
• Design, build and operationalize large-scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties – Spark/Python on Glue, DMS, S3, Athena, RDS-PostgreSQL, Airflow, Lambda, Code Commit, Code Pipeline, Code build, etc.
• Design and build production data pipelines from ingestion to consumption within a big data architecture, using DMS, Data Sync & Glue.
• Understand the existing applications(including on-premise Cloudera Datalake) and infrastructure architecture. • Analyse, re-architect and re-platform on-premise data warehouses to data platforms on AWS cloud using AWS or 3rd party services. • Design and implement data engineering, ingestion and curation functions on AWS cloud using AWS native or custom programming. • Perform detail assessments of current state data platforms and create an appropriate transition path to AWS cloud.
• Collaborate with development, infrastructure and data center teams to define Continuous Integration and Continuous Delivery processes by industry standards.
• Work on hybrid Datalake.
• Work closely with multiple stakeholders to ensure high standards are maintained.
Mandatory Skill-set
• Bachelors Degree in Computer Science, Information Technology or other relevant fields
• 5+ years of work experience with Data Ingestion, ETL, Data Modelling, Data Architecture to build Datalake. Proficient in designing, coding, and tuning big data processes using Pyspark.
• 3+ yrs of extensive experience in working on AWS platform using core services like AWS Athena, Glue Pyspark, RDS-PostgreSQL, S3 & Airflow(for orchestration).