Python Script Finds Duplicate and Near-Duplicate Pages in a Sitemap

·3 views

A tutorial explains how to detect duplicate pages across a website using only Python's standard library. Duplicates often arise unintentionally from CMS URL changes, tracking parameters, print views, or template-based pages. The script reads a sitemap, fetches each page, strips the shared template, and compares content by identical HTML, identical main content, and similarity scored with difflib. Matching pages are grouped into clusters with canonical checks and a suggested keeper URL, and results are exported to CSV with a console summary and a CI-friendly exit code. The author notes that Google does not treat duplicate content as a penalty in itself, but it can complicate search tracking and user experience.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Documents DevOps Learning Journey Through Hands-On Building and Troubleshooting

A developer has started publishing a public log of their DevOps and cloud engineering studies on DEV Community. So far they have experimented with Python and FastAPI, Docker, Kubernetes, GitHub Actions, and Jenkins, building a small employee API and CI pipelines. Their next phase covers Linux and networking, AWS, Terraform, monitoring with Prometheus and Grafana, and incident response practices. They plan to use the Floci local AWS sandbox to practise without incurring cloud costs, and will post weekly updates that include their errors and fixes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Preventing double bookings in a Telegram appointment bot using Python and SQLite

A developer describes how they built a small appointment bot for barbershops and salons that avoids double bookings without external services, relying only on SQLite. Rather than maintaining a table of slot availability that can go stale, the design stores only bookings and computes free slots on request based on working hours and service duration. An overlap check ensures a slot is free only if it does not conflict with any existing booking, and a final re-check inside a single transaction guards against races when the booking is written. Reminders are prevented from repeating through per-reminder flags, and the booking logic takes the current time as a parameter so it can be tested against an in-memory database. The author also mentions selling a complete bot template for $19 as a commercial offering.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Who Should Pay for Client Website Software: Client, Agency, or Free Tools?

A discussion among freelancers, agency owners and designers asks who should buy the paid software that websites depend on, such as premium plugins, page builder licences and per-seat editor access. One approach has clients buy every licence in their own name, which keeps costs visible and makes leaving an agency simple. Another has agencies hold multi-site licences and bundle the cost into care plans, which is simpler for clients and often cheaper overall. A third camp favours free or open-source tools, or platforms without per-person editing fees, to avoid recurring licence management. Each model carries trade-offs, including renewal risks when agency relationships end and the added setup or maintenance effort that free tools can require.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Walkthrough: How One Jira Ticket Moves Through an Org of AI Agents

A single ticket, PLAT-412 on rate limiting the public /export API, is followed from a quarterly board goal to merged production code. The author describes a layered design in which agents run inside software-enforced controls, including a policy engine for tool calls, a metered model gateway, and a kill switch that does not depend on agent cooperation. Each organizational seat, from CEO to coder to reviewer, is defined by its inputs, decisions, outputs, escalation rules, and performance metrics, which allows humans and agents to be swapped in the same role. The coder agent works through intake, understanding, clarification, estimation, planning, implementation, self-checking, and handoff, with a rule that every acceptance criterion must map to a concrete test. Hard stop rules such as repeated-error escalation, budget thresholds, and out-of-plan file changes are enforced by the orchestrator rather than by prompts.

0 comments Read more at DEV Community