DeepLearning.AI and AWS Course Covers Source Systems, Batch and Streaming Ingestion, and Pipelines

·1 views

A learner completed the second course in the DeepLearning.AI and AWS Data Engineering Professional Certificate on October 3, 2026, passing the final exam after hands-on AWS labs. The course covered how source systems generate and update data, how to connect to them, and how to troubleshoot common connection problems. Participants compared batch and streaming ingestion patterns and the ETL and ELT approaches, and built one pipeline of each type on AWS. The material also addressed orchestrating pipelines with infrastructure and pipelines as code, using Apache Airflow, along with monitoring tools for data systems and data quality. The author notes that choosing batch or streaming depends on the use case, while ETL versus ELT depends on whether transformation happens before or after loading.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How RAG Lets AI Answer Questions From Large Document Sets Without Retraining

The article explains Retrieval-Augmented Generation, a method that lets AI answer questions from long documents without being retrained. Using an HR example, it describes a company wanting to know its vacation carry-over rules buried in a 600-page handbook. RAG works by first retrieving relevant passages and then having the language model answer based on that evidence. The system is split into an indexing pipeline that ingests, chunks, and embeds documents, and an answering pipeline that retrieves, checks authorization, and generates cited responses. The author stresses that retrieval must be reliable, since similarity alone does not guarantee a correct passage.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SF Compute Builds Exchange for Physical Delivery of GPU Compute Capacity

SF Compute, founded by Evan Conrad, operates a marketplace where buyers and sellers trade future GPU compute time, functioning somewhat like a commodities exchange. The company's central challenge is ensuring that compute bought actually matches the specified hardware, network, and performance requirements, since GPU capacity is not a standardized product. Conrad explained that the firm deliberately stays neutral by not owning large amounts of compute, arguing that hyperscalers and GPU makers cannot run such a market without undermining trust. The interview also describes a price curve with discounts for near-term capacity and identifies large arbitrage gaps between purchase costs and inference revenue. Conrad argued that physical delivery must be established before cash-settled derivatives are introduced, to avoid manipulation and basis risk.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Pass the AWS SAA-C03 Solutions Architect Exam: A Practical Study Guide

The AWS Certified Solutions Architect – Associate exam is widely used in the cloud industry as a baseline credential, with certified professionals reportedly earning between $90K and $170K in the US. The author notes that more than half of first-time candidates fail, often because they study ineffectively rather than because the material is too difficult. The exam is framed as a judgment test built around scenario questions with several plausible answers, split across four domains with security weighted at 30 percent. The article outlines a six-week study plan covering core services, recurring architecture patterns such as decoupling, caching, disaster recovery, and database selection, followed by timed full-length practice exams. It also recommends practice tests that mirror the exam's scenario style, as well as aiming for consistent scores above 80 percent before booking.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Overlapping AI Subscriptions Quietly Drain Household Budgets, Study-Style Analysis Finds

A DEV Community article argues that many people pay for several AI tools whose functions largely overlap, without realizing the total cost. It cites figures suggesting people underestimate monthly subscription spending by more than 2.5 times, with the average adult now billed for over eight subscriptions. The piece outlines sample AI stacks, such as ChatGPT Plus, Claude Pro, and Perplexity Pro, that can total roughly $720 a year or more. It attributes the waste to overlapping features, auto-converting free trials, and reluctance to cancel for fear of needing a tool later. The author recommends listing all charges, applying a resubscribe test, and keeping one general assistant plus one specialist.

0 comments Read more at DEV Community