SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Dev uses source_hash to safely skip duplicate pipeline runs in data platform

0
·2 views

A developer building a mini manufacturing data platform implemented a source_hash-based idempotency mechanism to prevent duplicate processing of the same input files. The system computes a content hash of each input CSV and uses a composite key of dataset_id, business_date, and source_hash to identify whether a successful run already exists. If a prior successful run matches all three fields, the new execution is marked as skipped rather than reprocessed, eliminating duplicate gold metrics. The approach also tracks reuse_count in the run catalog to preserve an audit trail of skipped executions. The developer notes this is a synthetic, non-production slice and flags that handling corrected files for the same business date via atomic partition overwrite remains an open problem for a future Spark/Iceberg-based implementation.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer Proposes Cryptographic Two-Channel Architecture to Counter Prompt Injection

A developer writing on DEV Community argues that prompt injection in large language models is best understood as a control/data boundary problem, drawing a structural parallel to SQL injection. The proposed solution separates untrusted content from trusted instructions using two distinct channels: a data plane that receives and cryptographically signs external content as inert, and a control plane that exclusively handles tool execution and action selection. Untrusted text can only influence application behavior if a verified signature confirms it passed through the data plane, with all verification handled in application code rather than relying on the model's judgment. The architecture uses Ed25519 asymmetric cryptography so the control plane can verify data-plane provenance without being able to forge it. The author acknowledges the approach does not fully solve prompt injection, since most current LLM APIs still merge all inputs into a single token stream at inference time.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds RAG App Using LangChain, OpenAI, and Pinecone to Query Private Docs

A developer has built a Retrieval-Augmented Generation (RAG) application that allows large language models to answer questions based on private documents they were not trained on. The app uses LangChain to orchestrate the pipeline, OpenAI to generate embeddings and answers, and Pinecone as a vector database to store and search document chunks by meaning. During ingestion, the source text is split into 31 chunks, converted into numerical vectors, and stored in a Pinecone index configured with 512 dimensions. At query time, the most relevant chunks are retrieved and passed to the language model, so it answers from document context rather than relying solely on memorized training data. The project is structured into two separate Python scripts — one for document ingestion and one for retrieval and answer generation.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How 'Bust-Out' Fraudsters Game Credit Systems by Mimicking Model Borrowers

Bust-out fraud is a form of first-party credit fraud where an account holder deliberately builds a spotless repayment record over months to earn repeated credit limit increases, then draws down the entire line and vanishes. The scheme unfolds in three stages: careful cultivation, a gradual utilisation ramp-up, and a final bust where all available credit is withdrawn and payments cease. The core challenge for risk analysts is that bust-out accounts and genuinely distressed borrowers look nearly identical on a single monthly statement, making standard metrics like utilisation or days-past-due unreliable for telling them apart. Key distinguishing signals lie in the trajectory rather than any snapshot — bust-out accounts show rapid utilisation growth from a recent full-payment streak, higher cash-advance activity, and a history of limit increases, while distressed borrowers show slower climbs with no such clean prior record. A data science project using a synthetic account-month panel demonstrates that detecting bust-out fraud requires tracking the shape of behaviour across time, not just current account standings.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

DevOps Engineer Documents Five Cascading Failures in Legacy CRA-Django Containerization

A developer recently undertook what appeared to be a routine task of containerizing a legacy Create React App frontend and Django backend for production deployment. The project's inherited Dockerfile was built for a Next.js app, requiring a full swap to an nginx static file server since CRA outputs a plain build directory, not a standalone Node server. Further issues included a mismatched package manager command, where an npm-style rebuild call was used in a Yarn Classic environment, and native modules compiled against Node 24 while the runtime stage used Node 22, risking silent ABI crashes in production. Upgrading to Node 17 or later also broke the build due to OpenSSL 3 deprecating legacy MD4 hashing used by Webpack 4 in react-scripts 4, fixed only by setting NODE_OPTIONS=--openssl-legacy-provider. A secondary breakage from postcss-safe-parser's undeclared package exports further illustrated how outdated transitive dependencies can fail silently in CI but crash in production.

0 comments Read more at DEV Community