C

Staff Platform Engineer - Big Data Platform Management

Centre for Strategic Infocomm Technologies
Full-time
On-site
Singapore, Singapore
IT
CSIT develops products to advance the national security interests of Singapore. We use our products in a wide range of operations, including but not limited to Counter-terrorism and Computer Network Defence. We are looking for talented Senior Big Data Platform Staff Engineer. Join the awesome CSIT family and use cutting-edge technologies to protect the nation. 

Role

    • We are seeking an experienced Big Data Platform Staff Engineer to join our team, responsible for managing and optimizing our Big Data infrastructure platform. The successful candidate will have a deep understanding of Big Data technologies, architectures, and systems, as well as experience with managing large-scale data processing and analytics environments. The role will involve designing, implementing, and maintaining the Big Data infrastructure platform, ensuring high availability, scalability, and performance.

Responsibilities

    • Design, implement, and manage the Big Data infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes.
    • Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation.
    • Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems.
    • Ensure the security and governance of the Big Data infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits.
    • Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data infrastructure platform.
    • Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software.

Requirements (Minimum Qualifications)

    • Bachelor's degree in Computer Science, Engineering, or a related field.
    • At least 8 years of experience in managing Big Data Platforms.  Proven experience with technologies like Spark, Kafka, Hadoop, Elastic Stack, Kudu, NiFi and Kafka.
    • Strong knowledge of Big Data architectures, including data lakes, data warehouses, and data pipelines.
    • Experience in troubleshooting and resolving issues of the Big Data stack, ranging from hardware, networking, operating system to software
    • Strong understanding of data storage systems, including HDFS, S3, and other object storage systems.
    • Experience with scripting languages, such as Python, Scala, or Java.
    • Experience in managing physical and virtual infrastructures.
    • Experience in supporting search workloads in multi-terabytes and billions of records.
    • Experience in distributed systems.
    • Proven experience in using DevOps tools and CI/CD pipelines effectively.
    • Experience in leading engineering teams.

Nice to Have

    • Certifications in Big Data, such as Hadoop Certified Developer or Spark Certified Developer.
    • Experience with cloud-based Big Data platforms, such as AWS EMR, Google Cloud Dataproc, or Azure HDInsight.
    • Experience with machine learning and deep learning frameworks, such as TensorFlow, PyTorch, or Scikit-learn.
    • Familiarity with containerization, such as Docker or Kubernetes.
    • Experience with agile development methodologies and version control systems, such as Git.
    • Experience with cloud computing platforms, such as AWS, Azure, or Google Cloud.
    • Experience with containerization, such as Docker or Kubernetes.

Why join us?

    • The work is purposeful and meaningful 
    • You will work with the best engineers 
    • We work with modern technologies and tech stacks 
    • We have excellent engineering culture and work-life balance 
    • We aspire to engineering and operational excellence 
    • We empower to innovate 
    • We grow together as a family 
As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.
Apply now
Share this job
Twitter Facebook Linkedin Email