SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Browser automation or direct network calls: choosing how to scrape sites without public APIs

0
·1 views

Sites that expose only a web interface can be automated in two ways: driving a real browser or calling the backend endpoints the frontend uses. Browser tools such as Playwright, Puppeteer, and Selenium click, type, and read the DOM, which makes them suited to full user flows like SSO logins or 2FA prompts. However, they break when UI labels or layouts change, even if the underlying action still works. Direct network calls avoid that fragility but depend on request shape, authentication, and response schemas, which can change without warning. The article recommends choosing based on which failure modes a team can best tolerate, and validating status codes and schemas in network-based integrations.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

ROC-AUC Can Mislead on Rare Events; Precision-Recall Metrics Reveal False Alarms

A tutorial explains that a classifier scoring a high ROC-AUC can still flood reviewers with irrelevant alerts when the target event is rare. Using a hypothetical wildfire social-media monitor, it notes that positives make up only about 1% of posts in the example. ROC-AUC measures how well positives are ranked above negatives, while precision and recall focus on the usefulness of flagged posts. The article argues that for imbalanced problems such as fraud, security, maintenance, and crisis monitoring, precision-recall evaluation can give a more meaningful picture of performance. It promises a reproducible Python demonstration comparing both metrics on the same classifier.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Payments Firm Weighs Wiz-io Alternatives for Cloud Attack Surface Management

An organization in the payments industry is reassessing its choice of Wiz-io for cloud attack surface management, arguing that the tool was favored largely through personal engineering relationships rather than a rigorous evaluation. The piece contends that payment platforms need external attack surface coverage, real-time threat intelligence integration, and remediation that causes no downtime, since outages can quickly lead to financial losses and regulatory penalties. It proposes evaluating alternatives such as Palo Alto, Check Point, and CrowdStrike against criteria including continuous external asset mapping, correlation of threat data with internal vulnerabilities, and disruption-free patching methods. The author argues that inadequate tooling would leave misconfigured assets and exposed APIs open to attack and create maintenance windows that adversaries could exploit.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Multi-Agent AI Systems Often Fail From Poor Coordination, Not Weak Agents

A DEV Community article argues that AI agents working together can perform worse than a single agent when they lack coordination, with duplicated searches, conflicting rewrites, and looping calls driving up costs. The author compares this to disorganized college group projects, where overlapping and contradictory edits undermine the final result. The proposed fix is not removing agents but adding simple handoff rules, such as requiring agents to claim tasks before starting and allowing only one agent to write each result. A minimal coordination contract in Python is offered as an example, using a dictionary to track task ownership without machine learning or vector databases. The author recommends rules that let agents disagree by creating new revision tasks rather than overwriting others' work.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Browser-based Chinese tone trainer scores pitch contours without uploading audio

A developer built a free, browser-only tool that helps learners of Chinese tones by recording a single syllable and plotting its pitch curve against the target tone. The tool runs entirely on the user's device, so no audio is sent to a server, and it requires no app download or account. To get usable recordings, the developer turned off the browser's default microphone processing, since automatic gain control and noise suppression distort pitch information. Pitch is extracted with the YIN algorithm, which avoids the octave errors common in simpler autocorrelation methods. The write-up also covers how the contours are normalized and scored, including extending the scoring from single syllables to whole words and sentences.

0 comments Read more at DEV Community
Browser automation or direct network calls: choosing how to scrape sites without public APIs · ShortSingh