A single robots.txt line can block your entire site from Google search

·1 views

A misconfigured robots.txt file with 'Disallow: /' instructs all web crawlers to ignore an entire website, a mistake often made when copying settings from staging to production environments. Other common issues include a missing robots.txt file, no sitemap reference, and blocking CSS or JavaScript assets that search engines need to render pages. A broken or empty sitemap.xml can similarly reduce crawl efficiency without any obvious warning signs. A developer has released a free, open-source tool called OGCheck that lets users audit their robots.txt and sitemap.xml via simple curl commands. The tool, built with Python's standard library, also checks whether Open Graph social preview images load correctly.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Launches 47 Browser-Based Dev Tools With No Signup or Data Tracking

A developer has built KRUMB.DEV, a free collection of 47 developer utilities that run entirely in the browser without sending user data to any server. The platform covers tools for formatting, encoding, generating, testing, converting, and network diagnostics, including JSON formatters, JWT decoders, regex testers, and DNS lookups. No account creation is required, and sensitive inputs such as passwords, JWTs, and source code never leave the user's device, except for tools like DNS and SSL checks that inherently require a network call. Built with Next.js, TypeScript, and Tailwind CSS, the project is open source under the MIT license and available on Vercel. The developer states the roadmap will focus on utility rather than monetization, with more tools planned based on common developer needs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Dynamic Role and Permission System in Laravel Without Hardcoding

A technical guide published on DEV Community outlines how to implement a database-driven Role-Based Access Control (RBAC) system in Laravel for web applications. The approach replaces hardcoded role checks in controllers with a flexible architecture built around three core entities: Users, Roles, and Permissions, linked through many-to-many database relationships. Developers are walked through generating Laravel migrations, models, and pivot tables to store roles and permissions dynamically. A reusable PHP trait called HasPermissions is introduced to keep the User model clean while enabling role and permission lookups. The system integrates with Laravel's Gate mechanism, allowing permission checks to be managed from an admin panel without requiring code redeployment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Python Beginner's Guide Part 1: Installation, Variables, and Basic Programs

A beginner-focused tutorial series on Python has launched its first part, covering the foundational skills needed to start programming. The guide walks readers through installing Python, verifying the setup, and writing their first 'Hello, World!' program. It introduces core concepts such as variables, data types, user input, arithmetic operators, and code comments. Python is highlighted as a versatile language used across data science, web development, AI, automation, and cybersecurity. Part 2 of the series is set to cover conditional statements, loops, functions, lists, and dictionaries.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How One Developer Built a Production App Solo Using AI Across the Full Stack

A software developer spent 18 months building Fincher, a social app for discussing movies and TV shows, entirely alone while holding a full-time job, using AI tools across every stage of product development. The project, which had its first commit in January 2025, is now production-ready and approaching public launch. The developer argues that while AI has compressed the time needed to ship software, it has not reduced the engineering disciplines required — strategy, UX, architecture, backend, and infrastructure still demand the same rigor. A key warning is raised against two common failure modes: using AI only for code completion, or trusting it as an oracle, both of which cause critical steps like research and validation to be skipped. The methodology concludes that AI changed how long each development loop takes, not what obligations a developer must fulfill.

0 comments Read more at DEV Community