SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

AWS LLM Inference Compared: Bedrock, SageMaker, and Self-Hosted EKS Costs Broken Down

0
·2 views

A developer who has deployed all three AWS inference options for clients has published a cost comparison of Amazon Bedrock, SageMaker Endpoints, and self-hosted EKS for running large language models in production. Bedrock charges per token with no infrastructure management, SageMaker uses dedicated instances billed hourly, and EKS with GPU spot instances offers the lowest per-token cost but the highest operational complexity. According to the analysis, Bedrock is most economical at low volumes, but self-hosting becomes significantly cheaper beyond roughly 10,000 to 20,000 requests per day. At 500,000 daily requests, Bedrock costs an estimated $18,000 per month compared to around $2,022 for self-hosted EKS. The author recommends a hybrid approach for most production systems, using Bedrock for low-volume, high-quality tasks and self-hosted vLLM on EKS for high-volume, cost-sensitive workloads like classification and embeddings.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How one dev team built a single AI tutor routing 60+ features across five subjects

A development team built a multi-subject AI tutoring platform covering general academics, Indian competitive exams like JEE and NEET, language learning up to CEFR C1, career coaching, and an arena mode. Rather than letting each feature manage its own model calls, the team centralized all AI-backed features into a single registry with defined pricing, model chains, and fallback logic. A cost audit revealed that 'thinking tokens' were the biggest expense, prompting a full architectural redesign of the routing layer. The team also discovered a critical flaw where mastery scores never updated because quiz triggers relied on undocumented user-typed keywords, which was fixed by adding explicit UI controls and a shared progress engine. Key engineering lessons included treating response caching as an allowlist, issuing automatic refunds when cheaper fallback models respond, and ensuring errors surface visibly to users rather than silently rendering blank screens.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds AI Memory Agent With Intentional Forgetting, Benchmarks It Against Naive Systems

A developer built an AI memory agent called Synapse for the Global AI Hackathon Series on Qwen Cloud, designed to selectively forget outdated information rather than storing everything indefinitely. Unlike conventional approaches that treat all data equally in a vector database, Synapse assigns each memory a decaying salience score based on importance, recall frequency, and elapsed time. The system uses two different decay rates — around 72 hours for episodic details and 30 days for stable facts like preferences — so it forgets casual conversation while retaining meaningful information. A background process periodically consolidates repeated mentions into single memories and detects contradictions, automatically retiring stale facts when newer ones supersede them. The developer benchmarked the system against a naive vector-store approach using real Qwen Cloud API calls throughout, with no mocked responses, to validate the claims in practice.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Engineering team cuts RAG pipeline latency 40% using Bayesian search and hybrid retrieval

A development team rebuilt their retrieval-augmented generation (RAG) pipeline from scratch after standard production deployments proved inadequate for legal, API, and support-ticket use cases. The team replaced fixed 512-token chunking with document-type-specific strategies, including recursive, semantic, and agentic chunking, achieving recall@10 scores between 91% and 97%. They combined vector search with BM25 and cross-encoder reranking using Reciprocal Rank Fusion, which improved relevance correlation from roughly 0.75 to 0.92. A Bayesian-informed search optimization was also applied, ultimately reducing overall query latency by 40%. The team published their configurations and Python code to help other engineers replicate the tunable retrieval pipeline.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How One Team Built Automated LLM Evaluation That Catches 92% of Hallucinations

A development team discovered critical gaps in their AI quality process after deploying a RAG-based customer support assistant that served hallucinated responses to over 500 users before the issue was detected. The failures included fabricated billing policies and rate-limit figures pulled from a competitor's documentation, exposing the team's reliance on informal, manual testing. In response, they built a production-grade evaluation pipeline featuring a suite of automated judges assessing faithfulness, instruction-following, schema validity, safety, and domain accuracy. The system integrates directly into CI/CD workflows, blocking code merges that degrade output quality and enabling regression detection in real time. The team reports the pipeline now catches 92% of hallucinations before deployment, replacing subjective human review with structured, metric-driven evaluation.

0 comments Read more at DEV Community