Freelance Data Engineer - Senior opdracht

Data Engineer - Senior

The Data Engineer creates the data infrastructure in a simple and efficient matter by design, build and integrate data from various sources. The Data Engineer builds data pipelines from existing data sources and other (on-prem) systems into the Big Data platform; Creates consistent e2e BI solutions with a data lifecycle starting with collecting the needed data until visualizing them; Creates AI/ML models in BigData use cases and helping business to leverage the results; User story creation with good understanding of business processes, product and customer journey; Rates user stories for customer value and to challenge the product owner on prioritization of user stories (e.g., stability versus functionality); User story refinement: Is able to translate user stories into technical stories; Development: Able to design, develop and optimize data storage and data processing solutions and proficient in in one or more program languages like python and SQL to ensure sufficient quality level ,including test criteria; Deployment: Is able to create and execute script based deployments, including rollbacks. Is able to maintain/setup automated deployment using the VF standard toolset; Operations: Is able to setup automated monitoring and proactive alerting, is able to provide parameters to size the underlying application infrastructure.

The Data Kong Team is responsible for building and maintaining the Asset Analytics Wind data pipelines on Databricks, delivering trusted and scalable data products up to the Silver layer and enabling Value Delivery Teams through platform support, infrastructure management, governance, and technical expertise. We are looking for a highly experienced, structured, and self-driven Senior Data Engineer / Databricks Platform Engineer who can independently own complex technical challenges, support the wider team, and contribute across both data engineering and platform engineering domains.

Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake. Build and support Bronze and Silver layer data products following Medallion architecture principles. Develop and optimize batch and streaming ingestion pipelines. Monitor, troubleshoot, and improve Databricks performance, Spark jobs. Support Value Delivery Teams with platform onboarding, troubleshooting, best practices, and operational support. Implement and maintain Databricks and supporting Azure infrastructure using Terraform. Manage security, access control, and governance using RBAC, Unity Catalog, Service Principals, and Managed Identities. Develop and maintain CI/CD pipelines for automated deployment of data and infrastructure components. Ensure platform reliability, scalability, cost optimization, and operational excellence.

The team is present in the Amsterdam office on Tuesdays and Thursdays and we expect the candidate to also be avaiable in Amsterdam on those days.

De eisen
Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake. Ability to troubleshoot complex production issues across, data, infrastructure, identity and platform layers. Good knowledge of distributed data processing, including Spark optimization, partitioning, joins, AQE, data skew handling, and Delta optimization. Experience with Structured Streaming and event-driven architectures. Knowledge of Databricks Asset Bundles. Experience in Terraform and Infrastructure-as-Code. Experience with Azure DevOps/GitHub, CI/CD pipelines, and deployment automation. Good understanding of RBAC, Unity Catalog, Azure Entra ID, Key Vault, and data governance. Experience working with Azure Data Lake Storage (ADLS Gen2) and cloud-native data platforms. HBO/WO. Proven trackrecord in the data engineering area. English is the main language we use for communication.

De wensen
Experience with Databricks Declarative Pipelines (Delta Live Tables), including Streaming Tables, Materialized Views, data quality expectations, and pipeline orchestration. Experience working with OT, IoT, telemetry, or high-volume time-series data Experience in the energysector is a pre

Do you want to be part of the clean energy transition and make a real impact through data? As a Data Engineer at Vattenfall, you’ll be part of an agile SCRUM team working with advanced analytics tools across various systems to solve complex issues, collaborating closely with Vattenfall Wind stakeholders. What you’ll do: Your focus will be on designing, building, and maintaining data products that enable real-time and historical wind turbine data sharing with internal and external stakeholders. You will work with Kubernetes, Python, and Message Queuing Technology (MQTT) to create scalable data pipelines and integrate live and high-frequency, and scientific data from our wind turbine fleet.

What you bring
Good communication skills for explaining complex systems to non-technical stakeholders. You know your strengths and weaknesses. English is the main language we use for communication since the members of our team have different nationalities.

De eisen
You know Python inside out, including DevOps best practices like automated testing, CI/CD and Git. Experience building, deploying, and managing containerized applications using Docker and Kubernetes. Experienced with Cloud services. Experience with Databricks and building reliable data pipelines using PySpark, SQL, and Delta Lake. Experience with real-time messaging technologies, such as MQTT or other message queuing systems, for streaming and processing live data. Experience building and consuming APIs for data sharing and integration. You are familiar with the Agile way of working and methodology. Your command of English is strong. HBO/WO. We expect the candidates to have a proven trackrecord in the data engineering area.

De wensen
Cloud services: preferably Azure. As a bonus; familiarity with Apache NiFi or similar data integration and data flow tools.