Senior Data Engineer - Madrid, España - Daniel J. Edelman Holdings
Descripción
Responsibilities:
- Design, build, and maintain scalable and robust data pipelines to support analytics and machine learning models, ensuring high data quality and reliability for both batch & realtime use cases.
- Design, maintain, optimize data models and data structures in tooling such as Snowflake and Databricks.
- Leverage Databricks for big data processing, ensuring efficient management of Spark jobs and seamless integration with other data services.
- Utilize PySpark and/or Ray to build and scale distributed computing tasks, enhancing the performance of machine learning model training and inference processes.
- Monitor, troubleshoot, and resolve issues within data pipelines and infrastructure, implementing best practices for data engineering and continuous improvement.
- Diagrammatically document data engineering workflows.
- Collaborate with other Data Engineers, Product Owners, Software Developers and Machine Learning Engineers to implement new product features by understanding their needs and delivery timeously.
Qualifications:
- Minimum of 3 years experience deploying enterprise level scalable data engineering solutions.
- Strong examples of independently developed data pipelines endtoend, from problem formulation, raw data, to implementation, optimization, and result.
- Proven track record of building and managing scalable cloudbased infrastructure on AWS (incl. S3, Dynamo DB, EMR).
- Proven track record of implementing and managing of AI model lifecycle in a production environment.
- Experience using Apache Airflow (or equivalent), Snowflake, Lucenebased search engines.
- Experience with Databricks (Delta format, Unity Catalog).
- Advanced SQL and Python knowledge with associated coding experience.
- Strong Experience with DevOps practices for continuous integration and continuous delivery (CI/CD).
- Experience wrangling structured & unstructured file formats (Parquet, CSV, JSON).
- Understanding and implementation of best practices within ETL end ELT processes.
- Data Quality best practice implementation using Great Expectations.
- Realtime data processing experience using Apache Kafka Experience (or equivalent) will be advantageous.
- Work independently with mínimal supervision.
- Takes initiative and is actionfocused.
- Mentor and share knowledge with junior team members.
- Collaborative with a strong ability to work in crossfunctional teams.
- Excellent communication skills with the ability to communicate with stakeholders across varying interest groups.
- Fluency in spoken and written English.
Edelman Data & Intelligence (DXI) is a global, multidisciplinary research, analytics and data consultancy with a distinctly human mission.
We use data and intelligence to help businesses and organizations build trusting relationships with people: making communications more authentic, engagement more exciting and connections more meaningful.
DXI brings together and integrates the necessary people-based PR, communications, social, research and exogenous data, as well as the technology infrastructure to create, collect, store and manage first-party data and identity resolution.
DXI is comprised of over 350 research specialists, business scientists, data engineers, behavioral and machine-learning experts, and data strategy consultants based in 15 markets around the world.
Más ofertas de trabajo de Daniel J. Edelman Holdings
-
Research Manager
Madrid, España - hace 1 semana
-
Product Director, Data Strategy
Madrid, España - hace 1 semana
-
Senior Product Director, Ux
Madrid, España - hace 1 semana
-
Senior Backend Engineer
Madrid, España - hace 1 semana
-
Research Manager
Madrid, España - hace 1 semana
-
Front-end Engineer
Madrid, España - hace 4 días