Gaugran  ·  Senior Multi-Cloud Data Engineer  ·  5+ yrs

Overview: This project designed and implemented a robust end-to-end data pipeline to extract data from Google Analytics and transfer it to a cloud-based analytics platform for business insights. Responsibilities: Designed and implemented the end-to-end data pipeline using AWS Glue, Databricks, and Apache Airflow for orchestration. Leveraged Databricks for extracting large datasets and integrated with AWS S3 for cloud storage. Utilized Apache Airflow to automate scheduling and execution, optimizing resource usage. Applied PySpark for processing and transforming large-scale datasets, enabling real-time and batch capabilities. Implemented a multi-stage transformation process in Snowflake to deliver actionable insights.

Overview: This project involved developing and deploying a data pipeline monitoring system for Product 360. Responsibilities: Developed and deployed the data pipeline monitoring system using Databricks for processing and Snowflake for warehousing. Ingested and processed large-scale datasets from Azure Data Lake Storage (ADLS) and Azure Blob Storage using Databricks. Automated the pipeline with Apache Airflow to ensure scalable and reliable data flow to Snowflake. Implemented a custom data observability system to track data health, detecting anomalies and changes in patterns.

Overview: This project focused on designing and implementing a customer data integration platform to consolidate data from multiple sources (CRM, e-commerce platforms) into a single source of truth. Responsibilities: Designed and implemented the platform to consolidate customer data from CRM and e-commerce platforms. Leveraged AWS Glue to automate the ETL process into AWS S3, ensuring data consistency. Employed Databricks and PySpark for large-scale processing and complex transformations.

Overview: This project engineered a fully automated data extraction and transformation pipeline for health care systems. Responsibilities: Engineered a fully automated data pipeline leveraging AWS WorkMail, AWS S3, and AWS Lambda. Integrated AWS Lambda with AWS WorkMail to automate Excel file ingestion into an S3 bucket. Triggered Lambda functions for data cleansing and transformation.

• This project built an advanced real estate price prediction model with high accuracy using machine learning algorithms.
• It aimed to provide stakeholders with easily interpretable predicted real estate prices.

• Built the prediction model using linear regression, SVM, and decision trees.
• Conducted extensive data analysis and feature engineering using Pandas, Numpy, and Seaborn for preprocessing.
• Utilized Sklearn and TensorFlow to train, test, and validate various ML models.
• Visualized key data insights and model results using Matplotlib and Seaborn.
• Developed an optimization pipeline for model selection and hyperparameter tuning, achieving 89% prediction accuracy.