Manoranjan · Senior Spark Data Engineer · 6+ yrs
Mid-LevelAbout Manoranjan
Manoranjan is a Sr. Data Engineer with 6+ years of experience in software development, specializing in Big Data.
6+ years of commercial experience in
Skills(15)
Why hire Manoranjan?
Designed and implemented scalable event processing systems on AWS, ensuring data reliability.
Improved data processing performance across multiple ETL pipelines through optimization techniques.
Orchestrated complex data workflows using Airflow and PySpark for marketing purposes.
Contributed to CI/CD pipeline setups using Jenkins, streamlining development lifecycles.
Improved data processing performance by 30% across multiple ETL pipelines through various optimization techniques.
Successfully built an ETL pipeline for data synchronization between two applications for marketing purposes.
Enabled targeted marketing campaigns by developing a robust ETL pipeline for user preferences.
Identified distributor anomalies, leading to improved decision-making and reduced potential losses.
Project highlights(4)
Event Integration System ETL Pipeline – Data engineer
Overview: The project aimed to fetch event data (attendees, events, sessions, speakers) from an application called Rain Focus. Responsibilities: Interacted with clients to gather requirements, provide work updates, and manage development/testing with team members. Prepared Technical Solution Documents outlining project implementation. Improved data processing performance using various optimization techniques.
Key outcomes:
Successfully built an ETL pipeline for data synchronization between two applications for marketing purposes.
Improved data processing performance through optimization techniques.
User Preferences ETL Pipeline – Data engineer
Overview: This project involved capturing product users' communication preferences. Responsibilities: Interacted with clients for work updates, development, and testing with other team members. Processed events published by Event Bridge using AWS Lambda and stored them in RDS. Enriched this data further using PySpark scripts, orchestrated as Airflow jobs.
Key outcomes:
Developed a robust ETL pipeline for user preferences, enabling targeted marketing campaigns.
Ensured data reliability and scalability using SQS for large event processing.
Shipment Data Analysis – Data engineer
Overview: The project aimed to store and analyze shipment data in ADLS. Responsibilities: Utilized Azure Databricks, PySpark, Blob storage account, and Data Factory v2.0 as the Big Data Platform. Developed business logic for data wrangling using PySpark.
Key outcomes:
Successfully analyzed shipment data to identify and report anomalies, mitigating potential losses.
Red Flag - Distributor anomalies – Data engineer
- The project focused on identifying distributor anomalies for a retail company across various scenarios.
- The goal was to flag these anomalies as 'red flags' to support better decision-making at a top level.
- Understood all technical requirements for the project.
- Utilized Azure Databricks, PySpark, Blob storage account, and Data Factory v2.0 as the Big Data Platform.
- Used Pandas to read Excel files and transformed them into PySpark data frames in Databricks notebooks.
- Played an important role in Data Wrangling.
- Responsible for wrangling data and sending it to Blob storage.
- Developed business logic for data wrangling using PySpark.
- Implemented Data Factory pipelines to transfer data from blob storage into ADLS.
Key outcomes:
Enabled a retail company to identify distributor anomalies, leading to improved decision-making.
Successfully implemented data processing and transfer workflows for anomaly detection.
Industry experience
Logistics & Supply Chain
Reported in resumeAt a Glance
Single contract. No agency markup confusion.
Typically responds within 4 business hours.
Vetted by WitaristNot sure if this is the right fit?
Tell us your requirements and we'll match you with the best candidates.
Manoranjan
Sr.Data Engineer