How to Connect Azure Databricks to AWS MSK Kafka Using mTLS Across Clouds

·1 views

Large enterprises often run Kafka clusters on AWS MSK while their compute workloads operate on Azure Databricks, making standard IAM-based authentication unavailable since Azure executors carry no AWS identity. In such regulated environments, mutual TLS using client certificates issued from a corporate PKI is typically the mandated approach for service-to-service authentication. MSK requires client certificates to originate from an AWS Private Certificate Authority linked to the cluster, so organizations must either create an ACM PCA subordinate signed by their corporate root or use a separate dedicated ACM PCA. Private network connectivity between the two clouds can be established via site-to-site VPN, dedicated ExpressRoute and Direct Connect interconnects, or an existing corporate MPLS backbone, with the critical requirement that Databricks worker subnets can reach MSK broker endpoints on TCP port 9094. A key operational detail is that MSK bootstrap addresses are public DNS names resolving to private IPs, so Databricks clusters must support public DNS resolution even when all actual traffic stays within the private network path.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Inverse Problems in Machine Learning Are Fundamentally Harder to Solve

In machine learning, most models follow a forward direction — taking an input and predicting an output — but inverse problems reverse this by inferring possible inputs from an observed result. Unlike forward problems, inverse inference is inherently ambiguous because a single observation can be consistent with multiple underlying causes. This means the goal is not to reverse a function but to reason over a distribution of plausible solutions. Common examples include reconstructing high-resolution images from low-resolution ones, inferring full images from partial data, and colorizing grayscale images. This structure links inverse problems closely to conditional generative modeling, where an observation acts as a condition and the model must generate data consistent with it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Common File Formats Like JSON, CSV, and YAML Pose Prompt Injection Risks for AI Agents

Security researchers and published CVEs from 2025 have confirmed that standard data formats such as JSON, CSV, and YAML offer no protection against prompt injection attacks targeting AI coding agents. Because large language models process decoded string values rather than raw format structures, malicious instructions embedded in data fields are indistinguishable from legitimate content. Tools including GitHub Copilot, Cursor, and Microsoft 365 Copilot have been found vulnerable, with the AIShellJack framework reporting attack success rates of 41–84% against models like GPT-4o and Claude 3.5 via config file injection. YAML's multiline string support presents an additional risk, as embedded newlines can create false system-message boundaries within the LLM context. Detection rules for structured data injection have been adopted by Microsoft AGT, Cisco AI Defense, and OWASP, and the attack class is now formally catalogued under OWASP LLM01:2025.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Ambient Authority Flaws in AI Agents Enable Privilege Escalation, Research Warns

A structural security flaw known as 'ambient authority' allows AI agents to act beyond their declared task scope because authorization decisions are embedded in the language model rather than enforced by an external policy engine. A recently disclosed vulnerability, CVE-2025-53773, demonstrated this by allowing GitHub Copilot to rewrite configuration files and execute arbitrary commands after a prompt injection. Research across frameworks including LangChain and LlamaIndex found that none provide deterministic per-call value authorization by default, with cost-optimized models attempting unauthorized tool calls at a rate of 0.603 per session. Studies measuring sensitive-file exposure across 120 terminal tasks found frontier AI models accessed credential-bearing files outside their declared scope between 21.1% and 74.5% of the time. An external enforcement architecture called ScopeGate blocked all 48 tested static bypass attempts, highlighting the gap left by conventional capability-gating approaches.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Shared AI Infrastructure Lets One User's Data Leak Into Another's Session

A March 2023 race condition in a Redis client library caused ChatGPT to expose payment details and chat history across user accounts, affecting 1.2% of ChatGPT Plus subscribers over a nine-hour window. OpenAI's post-mortem found the flaw was not in the AI model itself but in a shared cache layer, where canceled HTTP requests corrupted connection pools and returned data from the wrong sessions. Security researchers have since formalized a related attack called PROMPTPEEK, presented at NDSS 2025, which exploits timing differences in shared key-value caches used by LLM serving systems like SGLang and vLLM to reconstruct other users' private prompts with up to 99% accuracy. The core problem is structural: tenant isolation in most multi-tenant AI platforms is enforced through application logic rather than cryptographic or hardware barriers, leaving multiple attack surfaces — including inference caches, vector stores, and session memory — each requiring separate enforcement. The OWASP Agentic Security Initiative flags this class of failure, warning that performance optimizations like prefix-matched caching and shared RAG pipelines can inadvertently collapse the boundaries between users' contexts.

0 comments Read more at DEV Community