Data Engineer (Middle/Senior)

4+ years of experience

EU or UK

Remote, Full-time

About the Company:
Our partner provides Databricks-focused consulting, supporting one of the largest companies in Europe that builds end-to-end AI solutions. Their clients span banking, insurance, and other enterprise sectors, where they implement scalable AI-driven data platforms and pipelines.

About the Role:
We are looking for a Middle to Senior Data Engineer who can productionalize and optimize 2 existing SARIMA-based anomaly detection pipelines that process mainframe logs. The focus is purely on engineering, not model development — improving performance, scalability, and cost efficiency.

Key Responsibilities:

Phase I – Data Engineering & Optimization:

Productionalize 2 existing pipelines (Python + PySpark + Delta).
Optimize performance and serverless cluster costs on Databricks.
Implement Databricks Asset Bundles and CI/CD with Jenkins.
Deliver a cost analysis and proposal for scaling up to 700+ pipelines.

Phase II – MLOps & Scale-Up:

Build parameterized pipeline templates for mass rollout.
Integrate MLOps best practices (model versioning, retraining, monitoring).
Ensure scalable orchestration and efficient use of compute.

Requirements:

Strong Python and PySpark skills (Databricks + Delta Lake).
Experience with CI/CD (Jenkins) and Databricks Asset Bundles.
Background in data pipeline optimization, partitioning, and cost tuning.
Knowledge of MLOps workflows and time-series models (SARIMA familiarity is enough).
DevOps mindset with experience on Azure/AWS and serverless environments.
EU-based candidates only (must).
Excellent communication skills with the ability to explain concepts clearly at both logical and conceptual levels.
Fluent in spoken and written English.