Kiran Patil · Data Engineer · 5+ yrs
Mid-LevelAbout Kiran
Kiran Patil is an experienced Big Data Engineer with 5 years of experience in data engineering, specializing in AWS cloud solutions and Hadoop ecosystem. The candidate has a strong background in Spark development, data migration, and warehousing. He has a proven ability to design and implement robust data pipelines, perform data cleansing, ingestion, and transformation using modern big data technologies.
- Successfully delivered AWS-based data warehousing solutions leveraging S3, Glue, Lambda, and Redshift.
- Developed and optimized Spark programs for data processing and analysis.
- Managed Hadoop ecosystems including HDFS, Hive, and Sqoop for efficient data handling.
- Implemented data validation and quality checks for data integrity.
- Proficient in various file formats like Parquet and ORC for optimized storage.
Skills(21)
SPARK
PYTHON
MYSQL
DATABRICKS
HADOOP
YARN
HIVEWhy hire Kiran?
5 years Big Data Engineer with AWS + Hadoop expertise.
Hands-on with AWS S3 + Glue + Lambda + Redshift + EMR + IAM.
PySpark scripting for ETL pipelines.
Hadoop ecosystem — HDFS + Hive + Sqoop + YARN + MapReduce.
Performance — Hive partitioning + bucketing with Parquet/ORC formats.
Successfully implemented AWS-based data warehousing solutions for data ingestion + transformation.
Developed and optimized Spark programs for data cleansing + ingestion using PySpark.
Managed and utilized Hadoop ecosystem (HDFS, Hive, Sqoop, YARN) for robust data handling.
Designed and created Hive tables with partitioning + bucketing for efficient data management.
Performed data validation, quality checks, and processing using aggregation, joins, filters.
Project highlights(2)
Data Migration Into Redshift – Data Engineer
- This project focused on creating a scalable and cost-effective AWS-based data warehousing solution.
- It involved establishing data pipelines for data ingestion and transformation into Redshift.
- Wrote PySpark/Python scripts to read data from MySQL, create data frames, and write them to AWS S3 landing buckets.
- Utilized AWS Glue jobs for reading from S3, performing data cleaning, and transferring to a final S3 bucket.
- Developed AWS Lambda functions to automatically trigger Glue jobs upon S3 uploads.
- Created tables and loaded cleaned data into the AWS Redshift cluster.
Key outcomes:
Successfully implemented an AWS-based data warehousing solution for data migration.
Established automated data pipelines for ingestion, cleaning, and transformation using AWS services.
Ensured data validation during ingestion to AWS S3 landing buckets.
Data Warehousing – Big Data Developer
- This project involved ingesting customer data from a mainframe system into a Data Warehouse.
- It utilized the Hadoop ecosystem for data storage and processing.
- Used Sqoop to efficiently transfer data from databases to HDFS.
- Created and managed Hive tables, performing data loading, analysis, validation, and quality checks.
- Implemented partitions and bucketing in Hive to handle structured data efficiently.
- Worked with various file formats like Parquet and ORC and performed data processing tasks such as aggregation, joins, and filters.
Key outcomes:
Successfully ingested customer data from mainframe systems into a Data Warehouse.
Implemented data validation and quality checks to ensure data integrity.
Optimized Hive table performance through partitioning and bucketing.
5+ years of industry experience
- •Data Migration Into Redshift— Data EngineerSPARK · PYSPARK · PYTHON · AWS (S3) +8
At a Glance
Single contract. Billed in USD.
Typically responds within 4 business hours.
Vetted by WitaristNot sure if this is the right fit?
Tell us your requirements and we'll match you with the best candidates.
Kiran Patil
Big data engineer