ETL Architecture

 Request and Guidelines Provided

• Client: Private Equity Secondaries firm

• Create a data warehouse in Google Cloud Platform (GCP) by migrating the existing data jobs and workflows from Talend and Deal Cloud to GCP

• Support in addressing ad-hoc business requests and resolving data-related issues

Final Deliverable and Value-add

• Created a fully migrated and optimized data infrastructure on Google Cloud Platform (GCP), including all migrated data jobs from Talend and Deal Cloud

• Implemented robust monitoring mechanisms for the workflows, ensuring proactive identification and resolution of issues

• Refactored old SQL queries and optimized existing ETL pipelines to reduce latency and execution time

• Assisted the client by providing them with customized solutions tailored to meet the business requirements while enhancing operational reliability and efficiency

Methodology

• Developed a comprehensive migration strategy to transition data jobs from Talend to GCP and onboarded Salesforce to migrate jobs from Deal Cloud by leveraging Cloud Functions and Workflow

• Established monitoring mechanisms to track job progress and identify any issues or delays, along with data validation checks to ensure data integrity

• Integrated with external API endpoints such as Preqin, CapIQ, and Burgiss to acquire holding-level details by developing a custom data retrieval process for further analysis and dashboarding

• Tools/Technology used: Python, BigQuery, Google Cloud Function, Google Workflows

Snapshots of the Output