SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Python retrieval pipelines for support content should enforce crawl boundaries

0
·1 views

The article argues that a retrieval system for a customer-support knowledge base should treat crawl boundaries as a data contract and an incident boundary rather than a single allowlist. It recommends fetching only approved source classes, preserving document and section citations, and quarantining material with unclear ownership or freshness before indexing. The author notes that robots.txt is guidance rather than authorization, so applications still need their own scope and access checks. Each chunk should carry its source URI, revision, section path, visibility class, and fetch time so citations point to the exact text behind an answer. The piece includes a Python example that checks host, path, publication state, and visibility before content is chunked.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Saudi refinery designs sovereign AI to turn engineers' operating knowledge into shared plant insight

A proposed sovereign AI architecture would let a Saudi refining operator run open-weight models on its own in-country hardware, reading only its internal systems of record. The system would link data from the historian, shift logs, work orders, change records, incident reports and turnaround findings into one governed model of the plant. Named agents would draft trip analyses, shift handovers and turnaround finding dispositions, which shift engineers would confirm or approve. Confirmed causes and decisions would be retained to improve later analyses, and a command centre would let engineers query the system with cited answers. The design responds to retirements of senior engineers and the heavy workload of plant turnarounds.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Fixing RC2-40-CBC Errors When Opening PFX Files in OpenSSL 3

Opening an older PFX archive with OpenSSL 3 can fail with an RC2-40-CBC unsupported-algorithm error, because RC2 is served by the legacy provider. The author suggests running openssl pkcs12 with the -legacy flag and -info -noout to inspect the archive's structure and encryption, while keeping integrity verification on. If the consuming application supports modern encryption, the identity can be re-exported into a new archive using OpenSSL 3 defaults, with the leaf certificate, needed intermediates, and an encrypted PEM private key extracted in a private directory. The original archive should be kept for rollback. The new file should then be verified by opening it with the default provider, comparing the leaf's SHA-256 fingerprint, and testing it in the target application. The author notes that repacking changes only the archive encryption, not the certificates or their signatures.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Drawing React line charts with plain SVG and no charting library

A developer shows that a basic line chart can be built in about 60 lines of React using only SVG, since SVG coordinates can be drawn directly. The post explains three core ideas: a scale that maps values to pixel positions, nice tick selection that produces readable axis labels, and building lines as SVG path strings. The example handles missing values by splitting the line into separate runs, so a null becomes a gap rather than a zero. It also adds accessibility by pairing the SVG with a hidden data table for screen readers. The author then compares how earlier oks-ui Chart versions mishandled such cases, showing that version 1.1.2 drew nulls as zero and sent the line outside the chart.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

LoRA matches full fine-tuning on most tasks, but full training wins on large new-knowledge workloads

Nodus reports that LoRA performs about as well as full fine-tuning on typical instruction and reasoning datasets and in reinforcement learning, provided the adapter covers all layers and uses a higher learning rate. Two studies, Biderman et al. (2024) and Thinking Machines' LoRA Without Regret, show that LoRA trails full fine-tuning once data volume exceeds the adapter's capacity, as in continued pretraining on billions of tokens. The authors attribute the gap to full fine-tuning's ability to learn weight changes of far higher rank. Memory costs also differ sharply: full fine-tuning needs roughly 16 bytes per parameter, about 7.7 times the footprint of LoRA on Qwen3-8B and Qwen3-32B by the article's estimates. By that math, a LoRA run on Qwen3-8B fits on a single 48 GB GPU, while full fine-tuning requires multiple GPUs.

0 comments Read more at DEV Community
Python retrieval pipelines for support content should enforce crawl boundaries · ShortSingh