Shubham Gagrani · Senior AWS Spark Data Engineer · 5+ yrs
Mid-LevelAbout Shubham
Shubham Gagrani is a seasoned Data Engineer with over 5.5 years of experience in developing and optimizing data pipelines. He has a proven track record of leveraging technologies such as Python, Apache Spark, and AWS to deliver scalable solutions. His expertise extends to cloud infrastructure management and AI/ML model development, making him a versatile asset for any data-driven organization.
Core expertise
Additional skills(24)
Flask
Bootstrap
MySQL
Pandas
SeleniumWhy hire Shubham?
Designed and maintained diverse data pipelines handling large amounts of data.
Increased pipeline speeds by up to 10x and reduced errors by 20 percent.
Successfully set up and managed AWS services, implementing CI/CD pipelines.
Increased pipeline speeds by up to 10x.
Reduced errors by 20 percent in data extraction and storage processes.
Attracted 235 registered users to a SaaS web application.
Project highlights(5)
Fine-Tuned Stable Diffusion Model – AI Developer
Overview: Fine-tuned a Stable Diffusion model to generate artwork in the traditional Japanese Ukiyo-e style. Responsibilities: Developed an AI Model by fine-tuning a Stable Diffusion model on Japanese Ukiyo-e style images.
Key outcomes:
Optimized training using 33 instance images and 1,000 regularization images.
SaaS Sound Generation Application – Developer
Overview: A web application to generate sound samples from text prompts, deployed on Render. Responsibilities: Developed and deployed a SaaS application using Flask, Bootstrap, MySQL, and replicate.ai for AI model inference.
Key outcomes:
Successfully attracted and managed 235 registered users.
Ensured reliable performance and security through secure payment processing.
E-commerce Data Pipeline – Data Engineer
Overview: Zid is a cutting-edge e-commerce platform designed for SaaS applications. Responsibilities: Developed data pipelines using the medallion architecture for analytics. Maintained MySQL databases, automated tasks, and monitored AWS Glue jobs.
Key outcomes:
Ensured smooth operation of MySQL databases through maintenance and performance optimization.
Implemented automation strategies to streamline repetitive tasks.
Media Data Aggregator – Data Engineer / ML Engineer
Overview: Velocity Media Data Aggregator is an innovative project for extracting, integrating, and presenting public information about places in South Africa. Responsibilities: Developed and maintained data extraction scripts using Apify and stored data in AWS S3.
Key outcomes:
Developed and executed comprehensive testing plans to ensure data accuracy.
Successfully set up and managed AWS services, implementing CI/CD pipelines.
Real Estate Ads Aggregator – Data Engineer
Overview: KI Immo is a real estate ads aggregator website that collects and normalizes ads from diverse sources. Responsibilities: Extracted data from 15+ diverse sources using SQL connections, Selenium, Airflow, Spark jobs, Kafka, and APIs.
Key outcomes:
Increased pipeline speeds by up to 10x by transitioning to a different data extraction framework.
Reduced errors by 20 percent by applying a data quality check mechanism.
5+ years of industry experience
- •E-commerce Data Pipeline— Data EngineerPython · MySQL · AWS Glue · Terraform +6
- •Media Data Aggregator— Data Engineer / ML EngineerPython · AWS S3 · AWS Lambda · Pandas +6
- •Real Estate Ads Aggregator— Data EngineerPython · SQL · Selenium · Airflow +5
- •SaaS Sound Generation Application— DeveloperPython · Flask · Bootstrap · MySQL +1
- •Fine-Tuned Stable Diffusion Model— AI DeveloperPython · Hugging Face
- •Real Estate Ads Aggregator— Data EngineerPython · SQL · Selenium · Airflow +5
At a Glance
Single contract. Billed in USD.
Typically responds within 4 business hours.
Top Skills
Vetted by WitaristNot sure if this is the right fit?
Tell us your requirements and we'll match you with the best candidates.
Shubham Gagrani
Python Developer