How Salesforce Apex Can Query Databricks Directly Without Syncing Billions of Rows
A development team faced the challenge of letting Salesforce users look up records from a Databricks dataset containing over one billion rows, without copying that data into Salesforce storage. Syncing the full dataset was impractical due to high storage costs, monthly refresh overhead, and data drift issues. The solution involved querying Databricks on demand via its REST API directly from Salesforce Apex, using the SQL Statement Execution endpoint. Integration required configuring Named Credentials and External Credentials in Salesforce, with a key step being enabling the 'Allow Formulas in HTTP Header' option to correctly resolve the Bearer token. Because the Databricks API is asynchronous, the implementation also needed polling logic to handle cases where the SQL warehouse is cold and returns a pending status.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in