feiCode SEO Optimization: Enhancing robots.txt + Search Engine Indexing

feiCode
, ,
1

Issue

feicode.com has never been indexed by Google. Upon investigation, three issues were identified:

1. Missing Sitemap Directive in robots.txt

Forgejo provides a complete sitemap (25 pages covering all repositories + 1 user page), but the robots.txt file does not inform search engines of its existence.

2. Overly Restrictive robots.txt

The original version only permits crawling of the homepage and the /explore page, effectively blocking nearly all valuable content.

3. Failure to Register with Search Engine Console Tools

feicode.com has never undergone search engine verification nor proactively submitted its sitemap.

Completed

Rewrite of robots.txt

The nginx configuration’s location block for robots.txt was changed from hard-coded return 200 to an alias pointing to a static file—improving maintainability.

Key changes:

  • Added Sitemap directive pointing to sitemap.xml.
  • Reduced Crawl-delay from 10 to 2.
  • Allowed crawling of repository homepages and /explore.
  • Blocked low-value pages: blame, commit, raw, diff, patch, archive, and activity.
  • Blocked functional pages: login, registration, settings, notifications, and search.
  • Blocked AI crawlers: GPTBot, CCBot, Amazonbot, Bytespider, etc.

Pitfalls of Custom robots.txt in Forgejo

In Forgejo v11.x, there are three layers of robots.txt:

  1. Built-in (dynamically generated via hardcoded Go code).
  2. Custom file (theoretically overrides the built-in version, but in practice, does not take effect in v11).
  3. nginx-level interception (highest priority).

Final solution: Use nginx’s alias to serve a static robots.txt file—simple and reliable.

Pending (Assigned to modiqi)

Google Search Console Registration

  1. Add feicode.com as a property.
  2. Verify ownership via DNS TXT record (domain DNS is managed on Cloudflare).
  3. Submit the sitemap.
  4. Request indexing of the homepage.

Optional Next Steps

  • Baidu Search Resource Platform
  • Bing Webmaster Tools
  • Regular monitoring of indexing coverage