Senior Data Engineer

Job Details

Senior Data Engineer
Location: UK Remote/Hybrid
Division: Technology

About Verian

Verian works with clients around the world, providing evidence, insights and advisory services to inspire the next generation of public policy and programmes. We operate at the intersection of data, technology, and human understanding. 

With permanent offices in 17 countries and a team of 800 specialist consultants and researchers, we advise government departments, regulators, charities, and public sector bodies to make better decisions and improve people’s lives.

About the Role

Our team is looking for a Senior Data Engineer to join our Data Platform team and help deliver our product strategy for managing and analysing social research data. You will work primarily with Azure and Databricks, supported by technologies like Power BI, to design, build, and maintain scalable, reliable, and high-quality data pipelines.

As a key member of the team, you’ll collaborate closely with product managers, lead engineers, and cross-functional teams to turn business requirements into technical solutions that drive real-world impact.

Key Responsibilities

•    Design, develop and maintain data pipelines using Azure Data Factory (ADF), Databricks, Synapse, Python, R, Spark, Pandas and SQL.
•    Manage and optimize Azure resources including Azure Portal, Blob Storage, and Data Lake.
•    Implement and manage data workflows, ensuring seamless data orchestration and integration.
•    Develop and maintain ETL processes, including delta loading and change data capture (CDC) mechanisms.
•    Implement pipelines with Azure DevOps, Terraform and/or CI/CD.
•    Ensure data security, quality, and governance across all, including access controls and retention policies.
•    Collaborate with stakeholders to translate business requirements into technical specifications.
•    Utilize Power BI for data visualization and reporting.

Skills and Experience

•    Proven experience implementing data platforms and products.
•    Strong experience with Cloud technologies, particularly Azure.
•    Proficiency in Python, R, Spark, Pandas, SQL.
•    Experience working with Delta Live Tables.
•    Familiarity with Databricks medallion architecture.
•    Experience with Agile and DevOps methodologies.
•    Knowledge of Unity Catalog.
•    Experience with CI/CD pipelines and Terraform.

Why Join Verian

Joining Verian means becoming part of a dynamic and collaborative community that thrives on innovation and intellectual curiosity.

•    Play a highly impactful role in building a data platform from scratch, right at the heart of our digital transformation.
•    Contribute to the creation of innovative data products leveraging AI and Machine Learning.
•    Work on cutting-edge projects that influence public policy and drive meaningful social change.
•    Enjoy flexible working arrangements that support work–life balance.
•    Collaborate with a passionate, and supportive team in an environment where wellbeing truly matters.

How to Apply

If you’re passionate about turning data into insights that impact society, we want to hear from you! Apply online today, and if your profile matches, we’ll invite you for a first exploratory interview.

We encourage applications from all sections of society and provide reasonable accommodations for candidates with disabilities throughout the application and interview process.

Learn more about us at veriangroup.com