Data Engineer - Databricks

Job Title: Data Engineer - Databricks

Location: US (Remote) – Must be based in the US
Experience: 6+ years
Immediate Joiners Only

Job Description:

We are seeking a skilled and motivated Data Engineer to join our team and work on Databricks-based solutions. This role will involve collaborating with stakeholders to understand data requirements and help in designing and developing scalable, secure, and efficient data solutions using Databricks, AWS, and other cloud technologies. You will be responsible for engineering complex data architectures and pipelines, ensuring seamless integration, optimal performance, and continuous improvement in data systems.

Key Responsibilities:

• Collaborate with stakeholders to gather data requirements and design Databricks-based solutions aligned with business goals.
• Engineer and document scalable, secure, and high-performance data architectures and models.
• Provide expert guidance to cross-functional teams including data engineers, analysts, and scientists, ensuring adherence to best practices.
• Develop and manage complex data pipelines, ETL workflows, and data integration strategies within Databricks.
• Optimize and monitor Databricks clusters, workloads, and queries for performance and cost efficiency.
• Implement strict security controls and ensure compliance with data privacy standards within Databricks solutions.
• Create and maintain comprehensive technical documentation including architecture diagrams, data flow representations, and guides.
• Act as the primary technical point of contact, providing updates to stakeholders and ensuring smooth communication of project progress.
• Collaborate closely with cross-functional teams to ensure end-to-end data solutions are met with project objectives.
• Stay up-to-date with trends in data engineering and Databricks, recommending and implementing new methodologies and tools.

Required Skills & Experience:

• Minimum of 5+ years of hands-on experience in data engineering and analytics.
• Proficiency in Databricks, with a focus on Spark-based data processing and analytics.
• Strong understanding of data engineering concepts, including data modeling, ETL processes, and data pipeline development.
• Advanced skills in SQL for querying, transforming, and analyzing data.
• Expertise in Python programming for automation and data manipulation within Databricks.
• Strong experience with Apache Airflow for orchestrating and scheduling ETL workflows.
• Familiarity with big data technologies such as Spark, Hadoop, and related frameworks.
• Strong understanding of data security and compliance best practices.
• Proficient in data modeling principles and data warehousing concepts.
• Experience using version control systems like Git.
• Excellent communication and collaboration skills, with the ability to work with cross-functional teams.