Data AI Engineer

About the job

Key Responsibilities: Data Engineering and Integration: • Design, develop, and optimize scalable data pipelines and ETL processes using Palantir Foundry for data integration, transformation, and creating data models to support analytics and business use cases. • Build and manage APIs and microservices using Python and Java to integrate data across systems and manage data processing and application logic.Data Modeling and Architecture: • Design and implement robust, scalable, and efficient data models in Palantir Foundry. • Collaborate with data architects to define data governance, data lineage, and data quality standards. • Develop and maintain reusable pipelines and templates for data transformation and enrichment.Azure Cloud Expertise: • Utilize Azure Synapse, Azure Data Lake, and Azure Storage for data storage and processing. • Implement secure and efficient data workflows on Azure, ensuring compliance with organizational and regulatory policies. • Monitor and troubleshoot Azure data pipelines for performance optimization.Programming and Automation: • Write clean, maintainable, and efficient code in Python and Java for data processing, automation, and integration tasks. • Develop scripts to automate repetitive tasks and improve overall system efficiency. • Hands on Experience on Snowflake in writing procedures / functions, Snowpipe, Data Pipelines, Data Transformation. Collaboration and Stakeholder Engagement: • Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights. • Provide technical support and training on Palantir Foundry to internal teams. • Participate in Agile development processes and collaborate with DevOps teams to ensure seamless deployment. Performance Monitoring and Optimization: • Monitor data pipelines and applications for reliability, scalability, and performance. • Implement best practices for error handling, logging, and alerting to ensure system stability. Required Skills and Qualifications: • Technical Expertise: • Proficiency in Palantir Foundry for data integration, modeling, and pipeline development. • Strong programming skills in Python and Java. • Hands-on experience with Azure Data Engineering tools (e.g., Azure Data Factory, Databricks, Synapse, Data Lake, etc.). • Solid understanding of data structures, algorithms, and software engineering principles. • Data Engineering Skills: • Experience in building and optimizing ETL/ELT pipelines for large-scale data processing. • Proficiency in SQL for data querying and transformation. • Familiarity with data governance, data lineage, and data security practices. • Cloud Expertise: • Strong knowledge of Azure cloud services and infrastructure for data engineering. • Experience with CI/CD pipelines, containerization (e.g., Docker), and orchestration tools (e.g., Kubernetes). • Problem Solving and Collaboration: • Excellent problem-solving skills and ability to troubleshoot complex issues in data systems. • Strong communication skills to collaborate effectively with technical and non-technical stakeholders. Preferred Qualifications: • Certification in Palantir or Microsoft Azure Data Engineering or related cloud certifications. • Experience in working in Agile/Scrum environments.

Requirements

• Data Engineering
• Azure
• Python
• Java
• Palantir Foundry

Preferred Technologies

• Data Engineering
• Azure
• Python
• Java
• Palantir Foundry