Data Engineer - (DataBricks)

As a Senior Data Engineer, you will be a key member of our data team responsible for designing, building, and maintaining the data infrastructure and pipelines that drive our data-driven, decision-making processes. You will collaborate with cross-functional teams to ensure the availability, reliability, and accessibility of our data assets, enabling our organization to extract actionable insights and deliver high-impact solutions.

What you will be doing:

• you will work in the capacity of a Data Engineer focusing on the data deliverables for Business and Studio.
  
• Designing and building data models to support Data Science, Business Intelligence, and downstream data sets.
  
• Building APIs and data products to better integrate data throughout our systems and processes.
  
• Monitoring the data pipelines and communicating any issues to leadership and partners.
  
• Working closely with business partners and developers to ensure proper requirements are documented and agreed to for our different initiatives.
  
• Partnering with developers and leadership to coordinate cross-function and cross-team efforts.
  

Requirements

• Proven experience with Data Engineer/Data Quality Engineer in a fast-paced environment
  
• An expert developer using SQL and SQL-like query languages
  
• Have deep expertise in Python and have experience organizing Python-based projects
  
• Vast Experience with Databricks
  
• Vast Experience with Spark for Big Data
  
• Vast experience with ETL and ELT
  
• Experience with Snowflake
  
• Experience with Cloud AWS and/or Azure
  

Would be Nice to have:

• Have experience defining and crafting automated unit and integration testing frameworks for data projects
  
• Experience with Data BuildTool - DBT
  
• Experience with Orchestrators such as Airflow
  
• Have a strong understanding of different data modeling methodologies (Kimball, Inmon, Data Vault)
  
• Previous Experience as Business Intelligence and Business Analyst
  
• Experience with IaC's such as Terraform or Cloud Formation
  
• Experience with CI/CD automation - using Jenkins
  
• Release Management - GitHub
  
• Run PoCs with new tools
  
• experience with Batch and Streaming data pipelines
  
• Experience working with data ingestion from 3rd party vendors using APIs
  
• Work on integration of data platforms into observability tools
  
• Excellent collaborator by tailoring your communication for different audiences and ensuring effective communication between developers, partners, and leadership.