SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Three Ethical Rules Every Developer Should Set Before Scraping Instagram

0
·1 views

A developer with hands-on experience building Instagram data pipelines argues that most scraping projects fail due to decisions made on day one, not from technical errors later. The author outlines three self-imposed boundaries: avoiding any data behind a login wall, treating publicly visible data as legally restricted under GDPR-style rules, and respecting platform rate limits as a binding constraint rather than an obstacle to engineer around. On privacy, the author recommends documenting a clear collection purpose before gathering any data, enforcing a 90-day retention limit in code, and excluding personal accounts in favour of openly operating business profiles. On rate limits, the prescribed approach favours sequential requests, generous delays, and a hard daily cap set well below enforced thresholds. The author contends that these constraints proved more durable than authenticated alternatives, with their pipeline surviving two platform changes that disabled colleagues' login-dependent collectors.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer launches free in-browser CS learning platform with AI tutor and memory visualizers

A developer has released CodePulse PRO, a free computer science learning platform that runs entirely in the browser with no sign-up or payment required. The platform uses WebAssembly to execute Python, JavaScript, and SQL code locally, eliminating the need for local environment setup. It includes an AST and memory state visualizer that animates stack and heap allocation, pointer references, and execution frames in real time. An AI voice tutor offers concept explanations and progressive hints, while a mock interview simulator replicates FAANG-style whiteboard coding sessions with live feedback. The platform covers over 49 course tracks spanning topics from Harvard CS50 and MIT algorithms to distributed systems, DevOps, and enterprise Java.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How to Handle Email API Rate Limits in Password Reset Flows Without Duplicate Sends

When an email provider returns a 429 (rate limit) response during a password reset flow, treating it as backpressure rather than a failure prevents duplicate messages and broken user experiences. The recommended approach separates three distinct clocks: the reset token's security lifetime, the user-facing cooldown period, and the provider-facing retry delay. A durable delivery job should be created within the same database transaction as the reset token, ensuring no valid reset state exists without a corresponding delivery task. Workers, not web processes or users, should handle retry scheduling by honoring the provider's Retry-After header or applying capped exponential backoff with jitter. Public endpoints should always return a neutral response regardless of account existence or provider status, and sensitive token strings must never be used as idempotency keys or appear in structured logs.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Testing LLMs as AI Coordinators on Consumer Hardware Reveals Key Memory and Reliability Limits

A developer experimenting with local multi-agent AI systems found that Gemma 4 26B, while effective as a worker model, failed to reliably orchestrate multi-step tasks across several agents. A tool-calling specialist, llama3-groq-tool-use:8b, performed worse by fabricating an entire collaboration session, including invented tools and teammates, rather than executing real calls. A larger model, gpt-oss:120b at roughly 77 GB, reasoned correctly but could not coexist in RAM alongside the worker models on an 80 GB machine. The experiments highlighted two distinct failure modes: benchmark-passing models that hallucinate progress in agentic settings, and capable models that are simply too large for consumer hardware constraints. The findings point toward a practical rule that coordinator and worker model sizes must be balanced to fit within available system memory.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Chinese AI APIs Offer Comparable Performance at Fraction of US Model Costs in 2026

A developer's cost analysis comparing US and Chinese AI APIs in 2026 found that Chinese models from labs like DeepSeek, Qwen, Kimi, and GLM have largely closed the performance gap with American counterparts. On benchmark tests measuring broad reasoning, top US models such as GPT-4o and Claude 3.5 Sonnet scored between 88.7 and 89.0, while leading Chinese models scored between 85.5 and 87.5. The price difference, however, is stark: GPT-4o output costs $10.00 per million tokens, while DeepSeek V4 Flash charges just $0.25 — roughly 40 times cheaper. The analysis noted that Chinese model APIs are accessible to international developers without requiring local phone numbers or Chinese-language documentation. For high-volume applications, the findings suggest switching to Chinese models could significantly reduce API costs without a major drop in output quality.

0 comments Read more at DEV Community
Three Ethical Rules Every Developer Should Set Before Scraping Instagram · ShortSingh