One developer built a 38.5M-domain open search engine on a 2 GB server

·2 views

A solo developer launched cl0q, a free, open-source search engine that does not log searches or require accounts, beginning its first crawl in April 2025. Running on a modest 2 GB VPS, the project grew to index 38.5 million domains and 24.8 million pages, with 8.5 million domains added in June alone. Key challenges included avoiding subdomain traps, managing chronic disk space shortages, and resolving database contention between the crawler and search functions. Certificate Transparency logs proved a valuable discovery tool, surfacing roughly 1,700 new domains per minute at peak. The developer's long-term target is 100 million domains, with storage now offloaded to a remote host connected via Tailscale.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Benchmark Tests Whether Models Honestly Credit Themselves for Writing Code

A developer built a benchmark called rai-attribution-bench to test whether AI models accurately report their own contributions to code, using a five-tier attribution rubric from their custom git plugin. The test ran 198 responses per model across 1,782 total answers, presenting each model with synthetic coding sessions and asking it to assign the correct attribution footer. Initial results were suspicious when three flagship models scored a perfect 1.000, which the author later traced to session descriptions that effectively gave away the answer. After rewriting the test cases to mirror real-world Claude Code transcripts — where authorship is ambiguous and edits arrive via shell commands rather than chat — the benchmark became meaningfully harder. Notable findings included one model offering to let the user choose who got credit, while another consistently maintained its attributions even when user pressure increased.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Shares Key Lessons from Scaling Enterprise Software Systems

A software developer with experience building and scaling enterprise applications has shared core principles learned from working in large-scale environments. The developer highlights that readable, well-documented code is more valuable than clever solutions in codebases maintained by large, distributed teams. Effective communication between teams is emphasized as equally important as technical architecture, with Conway's Law cited as a real organizational constraint. The post also cautions against hastily rewriting legacy systems, stressing the importance of understanding the reasoning behind older architectural decisions before replacing them.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Beginner Python Learner Seeks Guidance After Completing Basics

A beginner Python programmer has shared their struggle with feeling lost and stuck after completing foundational concepts. The learner has covered basic topics such as variables and functions but is unsure how to move forward. They expressed a desire to build real-world projects as a way to deepen their understanding. The post was shared on DEV Community, a platform where developers exchange knowledge and seek peer support. The learner is looking for practical direction to bridge the gap between theory and hands-on programming.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Guide: Build an AI Podcast Generator Using ElevenLabs TTS API

A tutorial published on DEV Community walks developers through building an AI-powered podcast generator using ElevenLabs, a voice AI platform offering realistic text-to-speech and voice cloning capabilities. The project uses Python 3.8+, the ElevenLabs REST API, and ffmpeg to convert raw text scripts into finished MP3 audio files. According to the guide, a 30-minute podcast episode can be generated in under a minute with a single API call. ElevenLabs offers a free tier providing 10,000 characters per month, making it accessible for developers to test the setup without upfront cost. The tutorial includes ready-to-run Python code and a curl example, covering environment setup, API authentication, and audio file handling.

0 comments Read more at DEV Community