The web is no longer just a collection of pages for humans and search engines. A second layer is forming: one built for AI agents, large language models, structured data systems, and machine-to-machine discovery.
That does not mean websites are dead. It does not mean every business needs to rebuild its site in markdown. And it definitely does not mean “AI SEO” is a magic shortcut.
It means the web is becoming more layered. The winners will be the businesses that understand which layers matter now, which ones are experimental, and which ones are distractions.
Key Takeaways
- Crawlable HTML is still the foundation of SEO.
- Structured data still helps machines understand entities, products, services, and relationships.
- LLMs.txt is more like a signpost than a ranking tool.
- Open Knowledge Format, or OKF, is useful for packaging structured knowledge, especially internal documentation and data definitions.
- Agentic Resource Discovery, or ARD, is about helping AI agents discover capabilities, tools, APIs, and services.
- Most local and service businesses should fix their core website before chasing machine-only formats.
What Is the “Second Layer” of the Web?
The second layer of the web is machine-readable infrastructure that sits on top of traditional webpages.
The first web layer is familiar: HTML pages, internal links, navigation, sitemaps, robots.txt, schema markup, product feeds, blog posts, and service pages.
The emerging second layer adds formats and protocols designed for AI systems: LLMs.txt, Open Knowledge Format, Agentic Resource Discovery, MCP, WebMCP, AI catalogs, machine-readable documentation, and agent-facing metadata.
Search Engine Journal described this shift as the web developing a parallel machine-readable infrastructure. That framing is right. These systems are not replacing the website. They are adding another way for machines to interpret, navigate, and interact with digital information.
HTML Is Still the Foundation
The most important point is also the least exciting: normal HTML still matters most.
Search engines and AI systems still need to crawl pages, understand internal links, interpret headings, and evaluate content structure. Your navigation, footer, headings, schema, images, and internal links all help machines understand what your site is and how its pages relate.
Flattening everything into markdown may sound efficient for AI, but it strips away context. A webpage is not just words. It is structure, hierarchy, emphasis, placement, relationships, and intent.
For most businesses, the right move is not “convert the site to markdown.” The right move is to make the HTML clean, make the pages crawlable, use clear headings, improve internal linking, add schema where it helps, make the content useful, and keep important content visible on page load.
Boring still works.
What LLMs.txt Actually Does
LLMs.txt is often described as a robots.txt-style file for AI systems, but that comparison can be misleading.
Robots.txt tells crawlers where they should or should not go. A sitemap tells search engines which URLs exist. LLMs.txt is closer to a guide for an AI agent that is already looking at your site.
It can point an AI system toward important pages, documentation, rules, or resources. That may help once an agent is already aware of your site. It is not likely to make your business suddenly discoverable in AI search by itself.
Think of LLMs.txt as a signpost, not a growth strategy.
What OKF Is For
Open Knowledge Format, or OKF, is an open specification from Google Cloud for representing curated knowledge as markdown files with YAML frontmatter.
That sounds technical because it is. OKF was not created primarily for a plumber, restaurant, med spa, law firm, or local contractor. It was designed for structured knowledge: data definitions, table schemas, metrics, runbooks, APIs, internal documentation, and other knowledge that agents may need to consume reliably.
In plain English, OKF helps turn messy internal knowledge into a portable library that both humans and AI agents can read.
That makes sense for developer documentation, API references, product specs, data catalogs, internal knowledge bases, technical runbooks, complex ecommerce product data, and enterprise documentation.
For a basic marketing website, OKF is probably not the next urgent move. For companies with structured knowledge, technical documentation, or agent-facing tools, it is worth watching and testing.
What ARD Is For
Agentic Resource Discovery, or ARD, is different from OKF.
OKF is about packaging knowledge. ARD is about discovering capabilities.
Google Developers describes ARD as an open specification for finding and verifying tools, skills, and agents across the web. It uses catalogs and registries. A business can publish an ai-catalog.json file on its own domain, describing capabilities such as MCP servers, OpenAPI tools, agents, or services. Registries can then crawl and index those catalogs so agents can discover and verify them.
That matters if your company offers something an AI agent might use directly: an API, booking system, quoting engine, product database, support tool, inventory feed, transaction workflow, or specialized agent.
For most small business websites, ARD is not urgent today. But the direction is clear: AI systems will not only read the web. They will increasingly act through it.
The Mistake Businesses Should Avoid
The mistake is treating all of this as one big bucket called “AI SEO.”
It is not one thing.
Each layer does a different job. HTML helps crawlers and users understand your site. Schema helps define entities and relationships. Sitemaps help expose URLs. LLMs.txt helps guide AI systems already visiting your site. OKF packages knowledge for agents. ARD exposes capabilities for agent discovery. Product feeds help commerce platforms understand inventory and offers.
If you treat these as interchangeable, you will waste time. Worse, you may create duplicate versions of your content that conflict with each other.
A machine-readable layer is only useful if it supports the real business. It should not become a parallel website that nobody maintains.
What Most Businesses Should Do First
Before chasing OKF, ARD, or any new AI-search format, most businesses should fix the basics.
- Make sure your most important pages are indexable.
- Improve page speed and mobile usability.
- Clarify your services, locations, and offers.
- Use strong internal linking.
- Add schema where it fits.
- Publish content that answers real customer questions.
- Keep your Google Business Profile updated.
- Build proof: reviews, case studies, examples, photos, and testimonials.
- Track impressions and clicks in Google Search Console.
- Refresh important pages every few months.
That foundation will help traditional SEO, local SEO, and AI-driven discovery.
When the New Machine Layer Matters
The machine layer matters most when your business has structured knowledge or machine-actionable services.
If you are a local service business, focus on HTML, local SEO, reviews, service pages, and helpful content.
If you are an ecommerce business, pay close attention to product feeds, schema, inventory data, and clean product pages.
If you are a SaaS company or technical platform, start thinking about documentation, API references, OKF, ARD, and agent-facing discovery.
If you are building tools agents can use, ARD may become important quickly.
The point is not to ignore the second layer. The point is to adopt it in the right order.
The Bottom Line
The web is becoming more machine-readable, but the fundamentals have not disappeared.
Businesses still need clear websites, useful content, technical health, authority, and trust. AI systems may change how discovery works, but they still need reliable inputs. A weak website does not become strong because it has an LLMs.txt file. A confusing offer does not become clear because it is packaged in markdown.
The best strategy is layered: build a strong human website first, make it easy for search engines to understand, add structured data where it helps, then test AI-facing formats when they match your business model.
The second layer of the web is real. But it is not a replacement for the first one.
Sources:
Search Engine Journal: The Web Is Growing A Second Layer
Google Cloud: Open Knowledge Format
Google Developers Blog: Agentic Resource Discovery