Data EngineerRole OverviewThe Data Engineer is responsible for designing, building, and maintaining scalable, cloud�native data pipelines and data infrastructure that support analytics, reporting, businessintelligence, and real-time data processing. This role ensures that data is accessible,reliable, secure, and optimized for performance across enterprise platforms. The DataEngineer collaborates with business stakeholders, analysts, data scientists, andapplication teams to deliver high-quality data solutions using modern cloud, big data, andAPI-driven technologies.Key Responsibilities• Design, develop, and maintain scalable ETL/ELT pipelines for batch and real-timedata processing.• Build and optimize data models, Delta Tables, and Lakehouse architectures tosupport analytics and reporting.• Develop and integrate RESTful APIs and data services to facilitate seamless dataexchange across enterprise systems.• Implement real-time and high-frequency data ingestion frameworks using streamingtechnologies and event-driven architectures.• Design and manage cloud-native data solutions leveraging Azure services includingAzure Data Factory, Azure Databricks, ADLS, Event Hubs, and Synapse Analytics.• Develop and optimize Databricks Spark applications for large-scale datatransformation and processing.• Ensure data quality, governance, security, and compliance across data platforms.• Collaborate with data scientists, analysts, application teams, and businessstakeholders to deliver scalable data solutions.• Troubleshoot, monitor, and optimize pipeline performance and data platformreliability.• Support DataOps and CI/CD practices for data pipeline deployment andautomation.Required Skills & Qualifications• Strong proficiency in SQL and relational databases such as Oracle, SQL Server, andMySQL.• Strong programming skills in Python, PySpark, PL/SQL, Java, or Scala.• Hands-on experience with Azure Cloud technologies:o Azure Data Factory (ADF)o Azure Databrickso Azure Data Lake Storage (ADLS)o Azure Synapse Analyticso Azure Event Hubso Azure Functionso Azure API Managemento Azure DevOps• Experience with Databricks Lakehouse architecture, Delta Lake, and Delta Tables.• Expertise in API development, API integration, RESTful services, and microservicesarchitecture.• Experience processing high-volume and high-frequency data with low-latencyrequirements.• Strong knowledge of real-time data ingestion and streaming technologies such asKafka, Azure Event Hubs, or Kinesis.• Experience with Spark, Hadoop, and distributed data processing frameworks.• Hands-on experience with OpenShift, Kubernetes, Docker, and containerizeddeployments.• Experience with workflow orchestration tools such as Apache Airflow and AzureData Factory.• Understanding of data governance, data security, and compliance best practices.Preferred Qualifications• Experience with Delta Live Tables (DLT), Auto Loader, and Change Data Capture(CDC).• Knowledge of DataOps, CI/CD, and Infrastructure as Code (IaC).• Familiarity with event-driven architectures and real-time analytics platforms.• Azure Data Engineer (DP-203) and Databricks certificationsmandatory skills: Python Azure SQL REMOTEADVANCED ENGLISHOriginally posted on Himalayas