Sidhu  ·  Data Engineer  ·  8+ yrs

Senior
Hyderabad8+ years experienceremote
Available within 48 hrs

About Sidhu

Data Engineer with 8+ years of experience specializing in Azure Cloud data engineering, with additional on-prem expertise. Experienced in developing robust data engineering pipelines and solutions, utilizing a Scrum-based methodology. Possesses strong technical and analytical skills, consistently delivering efficient data processing and integration solutions. Skilled in leveraging technologies such as Azure Databricks, Azure Data Factory, Snowflake, and PySpark for scalable data architectures.

  • Successfully managed end-to-end ETL pipeline implementations, achieving significant efficiency gains.
  • Proficient in developing custom PySpark scripts for data transformation and analytics.
  • Secured sensitive data and environment variables using Azure Key Vault.
  • Utilized Azure Databricks for machine learning and predictive modeling to improve business outcomes.
  • Experienced in modernizing ETL processes and implementing CI/CD for data solutions.
  • Demonstrated technical leadership in coordinating project execution and resolving technical issues.

Core expertise

Azure
Technology
8/10
Azure Databricks
Technology
8/10
AD
Azure Durable Functions
Technology
8/10
SQ
SQL
Technology
8/10
PySpark
Technology
8/10

Additional skills(31)

Azure DatabricksPySparkAzure DevOpsPythonSnowflakeAzure Data FactoryAzure Durable FunctionsSQLADLSETL/ELT processes

Why hire Sidhu?

8+ years Azure Data Engineer + Tech Lead with multi-cloud ETL ownership

SSIS → Azure & Snowflake ETL modernization

20% data processing efficiency increase via Azure Databricks

Multi-DB: Azure Cosmos DB + Azure Postgres SQL + Snowflake + Teradata + Oracle

Azure Synapse Analytics + ADLS + Azure Functions + Logic App

CI/CD for Snowflake in Azure DevOps — demonstrated through delivery on production projects in the candidate's experience

Datastage ETL + CA Erwin modeler legacy + Datastage performance tuning

Custom Snowflake roles + privileges + integrations

Managed overall implementation strategies for Azure data engineering projects

Achieved 20% data processing efficiency increase via end-to-end Databricks ETL pipelines

Developed custom PySpark scripts for raw data → analytics/reporting transformation

Led ETL modernization from SSIS to Azure & Snowflake with CI/CD

Secured environment variables + sensitive data via Azure Key Vault across projects

Utilized Azure Databricks for ML + predictive modeling for forecasting

Project highlights(4)

Project 1 – Tech Lead - Azure

  • Led a significant Azure-based data engineering project, focusing on environment setup, ETL pipeline implementation, and integration with various Azure services.
  • Utilized Azure Functions, APIM, and Azure Databricks to support business use cases.
  • Managed the overall implementation strategy and environment setup in Azure Services (Azure Functions, APIM, and Azure Databricks).
  • Spearheaded end-to-end implementation of ETL pipelines using Azure Databricks, enhancing data processing efficiency by 20%.
  • Developed custom PySpark scripts in Azure Databricks for raw data transformation into formats suitable for analytics and reporting.
  • Implemented REST APIs on Azure Functions for integrating business use cases, including Live Ramp data extraction and CRM contact updates.
  • Configured an Azure DevOps GIT repository for Azure Functions to ensure efficient change tracking and collaboration.
Azure Data FactoryAzure DatabricksAzure Durable FunctionsSQLPySparkADLSETL/ELT processesData modeling and schema designData integration and transformationData governance and securityAzure DevOpsPythonBig Data technologies (Spark)Azure Cosmos DBAzure FunctionsLogic AppKey VaultsDataverse Web APIAPIM (Azure API Management)REST API

Key outcomes:

  • Managed overall implementation strategy and environment setup for Azure services

  • Increased data processing efficiency by 20% through ETL pipeline implementation

  • Developed custom PySpark scripts for data transformation

  • Secured sensitive information using Azure Key Vault

  • Utilized Azure Databricks for machine learning and predictive modeling, improving business outcomes

Project 2 – Data Engineer

  • Focused on data processing within an Azure environment, involving Databricks, PySpark, and Azure Data Lake Gen2.
  • Emphasized data validation, transformation, and performance optimization.
  • Created Databricks notebooks for data processing and implemented validation and transformation logic using PySpark DataFrame and SQL API.
  • Developed an Azure Key Vault backed Secret Scope in Azure Databricks to integrate Azure Key Vault securely.
  • Implemented SQL connectivity with PySpark and designed and created SQL tables.
  • Worked on performance optimization techniques in PySpark and was responsible for estimating cluster size, monitoring, and troubleshooting of the Azure Databricks cluster.
Azure DatabricksSQLPySparkSparkSQLADLSData modeling and schema designData integration and transformationBig Data technologies (Spark)Key Vaults

Key outcomes:

  • Implemented validation and transformation logic using PySpark DataFrame and SQL API

  • Created an Azure Key Vault backed Secret Scope for secure integration

  • Worked on performance optimization techniques in PySpark

  • Responsible for estimating cluster size, monitoring and troubleshooting Azure Databricks cluster

Project 3 – Tech Lead/Senior Developer - Azure & Snowflake

  • Served as a Tech Lead/Senior Developer on an Azure and Snowflake project, focusing on ETL modernization, data warehousing, and CI/CD implementation.
  • Played a primary liaison role, coordinating with clients and managing technical execution.
  • Acted as the primary liaison, coordinating with the client for the overall technical execution of the project and managing work stream leads.
  • Led the modernization of ETL from SSIS to Azure & Snowflake, including logical and ETL orchestration in Azure Data Factory.
  • Utilized Azure Key Vault for securely storing and accessing secrets in Azure Data Factory.
  • Involved in the creation of custom roles in Snowflake, assigned specific privileges, and implemented CI/CD for Snowflake in Azure DevOps.
  • Created Snowflake stored procedures using SQL language and managed external stages for the staging layer.
Azure Data FactoryAzure DatabricksSQLSnowflakeADLSETL/ELT processesData modeling and schema designData integration and transformationAzure DevOpsData warehousingAzure Cosmos DBAzure Postgres SQLData LakeAzure FunctionsLogic AppKey VaultsAzure Synapse AnalyticsSSIS

Key outcomes:

  • Coordinated with client for overall technical project execution

  • Led modernization of ETL from SSIS to Azure & Snowflake

  • Implemented CI/CD of Snowflake in Azure DevOps

  • Involved in creation of custom roles and integrations in Snowflake

Project 4 – ETL Developer/Senior Developer – Oracle

  • Engaged as an ETL Developer/Senior Developer on an Oracle-based project, specializing in data extraction, transformation, and loading using Teradata and Datastage.
  • Focused on data modeling, integration, validation, and performance tuning.
  • Extensively worked on creating ETL programs using Datastage for data extraction, transformations, and loading.
  • Designed mainframe jobs using mainframe source files as an ETL developer.
  • Performed data integration and validation tests for data warehousing tasks.
  • Developed data models using CA Erwin modeler to perform data loading and describe appropriate relationships, interacting with data architects.
  • Extensively worked on Datastage performance tuning to address source, target, and mapping level bottlenecks.
SQLETL/ELT processesData modeling and schema designData integration and transformationData warehousingTeradataDatastage ETL toolErwin toolOracle SQL

Key outcomes:

  • Extensively worked on creating ETL programs for data extraction, transformations, and loading

  • Performed data integration and validation tests for data warehousing tasks

  • Developed data models using CA Erwin modeler

  • Extensively worked on Datastage performance tuning to address bottlenecks

8+ years of industry experience

SaaS / B2B1 project
  • •Project— Tech Lead - AzureAzure Data Factory · Azure Databricks · Azure Durable Functions · SQL +16

Ready to work with Sidhu?

Onboard within 48 hours. No long hiring cycles, no recruiter middleman.

At a Glance

LocationHyderabad
Experience8+ years
Work moderemote
Direct hirePossible
Start within48 hours
From$2,012/ month

Single contract. Billed in USD.

Typically responds within 4 business hours.

5-day replacement guarantee
48-hour onboarding, single invoice
Direct chat — no recruiter middleman

Top Skills

Azure
8/10
Azure Databricks
8/10
Azure Durable Functions
8/10
SQL
8/10
PySpark
8/10
Seniority signals
Owns production deploysSystem owner
VerifiedVetted by Witarist
Technical skills assessed & verified
Background & identity checked
English communication verified
Ready to onboard in 48 hours

Not sure if this is the right fit?

Tell us your requirements and we'll match you with the best candidates.

Sidhu

Azure Data Engineer