SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

robots.txt matching is a literal prefix, and one trailing slash would have left our dashboard crawlable

0
·6 views

Our robots.txt is generated from one array: export const CRAWLER_DISALLOW = [ '/dashboard', '/play', '/api', '/auth', '/subscribe', '/webhook', '/too-many-requests', ] as const Not one of those has a trailing slash, and that is the first thing I would check in anybody else's robots.txt. Disallow is a literal prefix match against the path. Disallow: /dashboard/ blocks /dashboard/settings and /dashboard/billing, and leaves the bare /dashboard perfectly crawlable, because /dashboard does not begin with /dashboard/. In our case that is not a hypothetical: /dashboard is linked from the footer of th

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How to See Any Company's LinkedIn Ads with Python (Ad Text, Impressions and Targeting)

If you sell to businesses, your competitors are almost certainly advertising on LinkedIn: webinars, whitepapers, demo offers, "thought leader" posts from their executives. LinkedIn lets you see all of it. The LinkedIn Ad Library lists every ad that ran on LinkedIn in the past year, and for ads shown in the EU it even adds run dates, impressions, the country split and the targeting the advertiser chose (the EU's Digital Services Act requires it). The catch: it's a search page for humans. No export, no API, 24 ads at a time, and a name search that's fuzzy.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Our Permissions-Policy denies geolocation, and the pricing page still knows what country you are in

curl -sI https://pub-trivia.app/ permissions-policy: camera=(), microphone=(), geolocation=() referrer-policy: origin-when-cross-origin strict-transport-security: max-age=63072000; includeSubDomains; preload x-content-type-options: nosniff x-dns-prefetch-control: on x-frame-options: SAMEORIGIN No x-powered-by. Six headers, declared once in next.config.mjs for /:path*, and each one is a decision rather than a default. The two most interesting are the ones that look like they contradict the product. The pricing page shows you a price in your own currency. It knows roughly where you are.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Getting to Know Yourself: Understanding self in Ruby

self is one of those Ruby keywords that looks easy until you meet it in real code. You see it in a setter: self.name = normalized_name Then in a class method: def self.from_cents(...) Then somewhere inside a DSL or instance_eval, and suddenly the question isn't really: What does self mean? The better question is: Who is self right here? That small change makes Ruby's object model much easier to follow. Consider a model-like object: class Customer attr_accessor :name, :email def initialize(name:, email:) @name = name @email = email end def normalize self.name = name.strip.downcase self.email =

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Not Every File Needs a Server: Building File Tools with a Local-First Approach

Not Every File Needs a Server: Building File Tools with a Local-First Approach When we think about online file tools, the architecture often looks Choose a file ↓ Upload it to a server ↓ Process it ↓ Download the result It works. But while building file utilities, I kept coming back to a simple Does every file really need to leave the user's device? For many common operations, the answer is no. Modern browsers are capable of doing much more than simply uploading At the same time, trying to make everything client-side introduces its That led me toward a fairly simple principle: Process locally

0 comments Read more at DEV Community