SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Developer Proposes 'Library of Websites' to Catalog the Entire Internet

0
·1 views

A developer raised a discussion on DEV Community questioning whether a structured, browsable database of all active websites on the internet currently exists. Unlike search engines that rank results by popularity, the proposed platform would focus on discovery, allowing users to explore websites regardless of their traffic or prominence. The concept would require website owners to install a verification snippet, similar to Google Search Console, to register their sites in the database. The developer acknowledged key open questions around technical feasibility, owner participation, and whether voluntary registration is the right approach. The post invites community feedback on whether such a platform already exists and how it could realistically be built.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How Trino, Spark, and DuckDB each query the same Apache Iceberg table

Apache Iceberg allows multiple query engines to read the same table stored in object storage without duplicating data, with each engine differing only in how it accesses the shared metadata. Trino connects via a catalog and offers clean, straightforward SQL for interactive queries, making it well-suited for shared lakehouse environments. Spark requires additional session configuration with Iceberg extensions but is the preferred choice when queries are part of larger data pipelines involving transforms or batch writes. DuckDB provides the fastest path for local, read-only inspection by scanning Iceberg metadata files directly, though it can also attach a REST catalog for broader catalog-backed workflows. Understanding how all three engines interact with the same underlying table is essential for teams building and operating real lakehouse architectures.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

UUID v7 outperforms v4 as a database primary key due to time-ordered sorting

A developer exploring UUID internals found that version 7 UUIDs, standardised in RFC 9562 in 2024, offer a meaningful advantage over the widely used v4 format for database primary keys. Both types are 128-bit identifiers that allow distributed systems to generate unique IDs locally without any central coordination or risk of collision. While v4 UUIDs fill 122 bits with cryptographic randomness, v7 embeds a 48-bit Unix millisecond timestamp at the most significant end, making IDs sort chronologically as plain strings. This time-ordered structure prevents the random B-tree index scattering caused by v4 keys, which triggers page splits and cache inefficiency at scale. No browser built-in currently generates v7, but the logic can be hand-rolled by writing the timestamp byte-by-byte to avoid JavaScript's 32-bit bitwise operator limitation.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developers Build AI Cyber-Attack Detective Tool at Hackathon After All-Night Coding Sprint

A three-member team — Mehraan, Aqib, and Ubaid — built KoshurLock Holmes, an AI-powered cybersecurity investigation tool, during a WeMakeDevs hackathon. The tool addresses a core problem in post-breach forensics: evidence from VPN logs, badge readers, email gateways, and other sources is scattered across systems, forcing analysts to manually connect the dots over days or weeks. KoshurLock Holmes parses uploaded evidence files, extracts entities and relationships using Cognee, and constructs a unified knowledge graph so that the same individual appearing across multiple logs resolves to a single node. Users can query the system in plain English, receive cited multi-hop reasoning, and even instruct it to discard false planted clues, causing flawed conclusions to collapse. The project nearly stalled at 4 AM on the final day when the team hit Groq's free-tier token limit with the submission deadline hours away, but they pushed through to complete it by morning.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How to Build a Bot-Resistant AI Browser Agent Using Playwright, Gemini, and Bright Data

A technical guide published on DEV Community walks developers through building an AI-powered browser automation agent using Node.js, Playwright, Google Gemini, and Bright Data. The tutorial addresses a core challenge in modern web automation: most AI browser agents fail not due to poor reasoning but because websites detect and block non-human traffic through fingerprinting, IP reputation checks, and behavioral analysis. To bypass these defenses without building custom anti-detection infrastructure, the guide delegates browser identity and anti-bot handling to Bright Data while the AI layer focuses on reasoning and task execution. The result is a functional agent called SIKKI Agent, capable of browsing real websites, extracting product data, analyzing content, and generating reports. The entire stack relies on open tools — Node.js, Playwright, Bright Data, and Gemini — with no proprietary AI frameworks required.

0 comments Read more at DEV Community
Developer Proposes 'Library of Websites' to Catalog the Entire Internet · ShortSingh