Architected a custom API integration and high-volume data pipeline to migrate over 1TB of historical file data from Salesforce to Google Shared Drive, eliminating the need for expensive third-party tools and reducing additional Salesforce storage costs by 100%.
Tools: APEX (Classes, Queueable, Batch, Test), Named Credentials, Salesforce REST API (HttpRequest and HttpResponse), Custom Fields, Data loader
As our organization transitioned away from Salesforce for active transaction management, we were left with over 1TB of historical static file data that was required for industry-standard audit compliance. Maintaining this data within Salesforce was becoming financially unsustainable, incurring high monthly surcharges for "extra storage" when the company already possessed underutilized space within Google Drive. After evaluating third-party migration tools priced between $5,000 and $10,000, the business required a more cost-effective, custom-built solution to securely offload this data while maintaining a searchable, organized archive.
I engineered a bespoke API integration using APEX Queueable jobs to automate the high-volume transfer of files to a Google Shared Drive. By building this "in-house," I provided the brokerage with a zero-cost alternative to third-party apps that offered granular control over the migration process. The solution was designed to be "failure-tolerant," incorporating custom status tracking on the ContentVersion object to distinguish between successfully archived files and those requiring manual troubleshooting. This architecture allowed for the systematic deletion of migrated files from Salesforce, successfully reclaiming storage and halting all related overage fees.
The migration engine operates via a scheduled background job that identifies transaction records and their associated file attachments. The process begins by interfacing with the Google Drive API, handling OAuth, to programmatically generate folders using a strict naming convention to ensure easy cross-referencing between both systems. Using the Blob data type, the Apex logic retrieves the multi-part file content and recreates the file within the corresponding Google Drive folder, regardless of the file extension. To manage the 1TB volume without hitting governor limits, I utilized Batch and Queueable Apex to process the data in manageable segments. Upon a successful transfer, a custom field on the Salesforce content version record is marked as "Transferred," while any errors trigger a "Failed" status for audit purposes. Once the migration was validated, the "Transferred" files were purged from Salesforce, effectively shifting our historical archive to Google Drive.