Spark Jobs

I’m building a new web-based platform whose core value is clear, interactive data-visualization. To get there I need an experienced Nigerian software engineer who is equally at home designing back-end services and shaping the data that feeds them. Your first mandate will be to architect and code the full data pipeline—from ingestion through transformation to storage—so that the front-end team always has clean, query-ready information. You’ll also dive into exploratory and diagnostic analysis, helping us uncover trends that inform product decisions and power the visual stories we show users. The stack is flexible, but you should be comfortable with modern data-engineering tools (think Python or Scala for ETL, SQL-based warehouses, stream processing frameworks, ...

Core Focus: Roughly 50% hardening and governing cloud data pipelines, and 50% building business-facing AI/ML Proof of Concepts (POCs). Support MLOps infrastructure to operationalize, monitor, and scale models built by data science teams. Key Responsibilities Pipeline & Feature Engineering: Design, automate, and scale robust data pipelines for AI/ML workloads. Build and manage feature stores to support model training and inference. Data Integration: Ingest and transform structured and unstructured data sources using AWS Glue, Snowflake, and Informatica Data Management Cloud (IDMC). MLOps & Automation: Implement DevOps/MLOps practices and CI/CD pipelines (via GitHub Actions). Monitor pipeline performance, troubleshoot production issues, and mitigate model drift. Governance-b...