Pages machines can read

Text in the HTML

The safest place for any fact is plain text in the page's HTML as the server sends it. Some AI crawlers fetch the raw HTML and do not run JavaScript the way a browser does. Search engines like Google do render JavaScript, but rendering costs them time, and not every crawler bothers. A fact that appears only after scripts run may never be seen by the systems an assistant relies on.

Many website builders produce server-rendered HTML, and their text is fine. Problems arise with pages built as single-page applications, with content loaded from a separate service after the page loads, and with third-party widgets that inject text into the page.

The view-source test

An owner can check this without technical help. Open the page in a desktop browser, right-click and choose “View page source” (not “Inspect”, which shows the rendered page). Use the browser's find function to search for a sentence from the page, a price, or the opening hours. If the text is in the source, a crawler that fetches the page can read it. If the source is mostly scripts and the sentence is missing, the content depends on JavaScript.

PDFs and images of text

PDF menus, price sheets and brochures are common, and crawlers vary widely in how well they read them. A scanned PDF, which is a picture of a page, holds no text at all unless it has been processed with character recognition. Images of text (a price list exported as a graphic, a schedule designed in a social media tool, a certificate photographed on the wall) are the same problem in another form. Bay Area AEO's restaurant page describes turning PDF and image menus into crawlable pages as a core part of the work for restaurants; the same applies to any trade that publishes lists. Keep the PDF for people who want it, and put the content on an HTML page too.

Alt text for the images that remain

Images that carry meaning (a photo of the storefront, a finished kitchen, a dish) should have alt text that says what they show in plain words: “Exterior of the bakery on Fourth Street in San Rafael with the green awning”. Alt text helps people using screen readers, and it gives a crawler a sentence where it would otherwise have nothing. Avoid stuffing it with keywords; describe the image.

Booking widgets, maps and embeds

Third-party widgets (booking calendars, review carousels, embedded maps, social feeds, chat boxes) are usually loaded from another site inside a frame or by script. A crawler reading the business's page sees little or nothing of their content. That is fine for features that are inherently interactive, as long as the facts they contain are also stated in text. A class schedule inside a booking widget should have a plain-text summary on the same page. A map embed should sit beside the written address and directions. A review carousel should be supplemented by a few quoted reviews in text, with permission.

Tabs, accordions and “read more”

Hiding text behind tabs or accordions is fine for readers on a phone. What matters is whether the hidden text is in the HTML when the page loads. In most well-built themes it is, and the accordion simply hides it with CSS. In some, the text is fetched only when the reader clicks. Use the view-source test on a page with an FAQ accordion: if the answers are in the source, the design is fine.

Speed and reliability

Crawlers have limited patience. A page that takes many seconds to respond, or times out under load, may be skipped or read partially. Reasonable speed is also good for customers, most of whom arrive on phones. Large uncompressed images, many third-party scripts and slow hosting are the usual causes. Free tools such as Google's PageSpeed Insights show the main problems and how to fix them.

Mobile layout

Google indexes the mobile version of pages, and AI Overviews draw on that index. If the mobile layout drops content that the desktop layout shows (a common shortcut in older themes), that content may be missing from what Google's systems read. Check the main service pages on a phone and make sure every fact on the desktop version is there too.

Headings and semantic HTML

Crawlers use a page's structure to understand it. One main heading naming the page's subject, section headings that name what each section answers, paragraphs as paragraphs, lists as lists, tables as tables: these let a machine find the passage that answers a question and lift it cleanly. A page built from styled boxes with no real headings, or with text placed inside images and buttons, is harder to take apart.

Duplicate and thin pages

Machines also read what a site says across pages. Many near-identical pages, such as city pages with only the name changed, dilute the signal and can lead crawlers to skip pages they judge to be repeats. Old pages that contradict current ones (last year's prices, a former location, a discontinued service) can be read as current. Remove or redirect pages that are no longer true, and consolidate near-copies into one strong page.

Pop-ups, cookie banners and gates

Some pages put a barrier in front of their content: a full-screen newsletter pop-up, an age gate on a winery or bar site, a location prompt, a cookie banner that covers the page until it is dismissed. People click past them. Crawlers may not. If the content is in the HTML behind the overlay, most crawlers still read it; if the overlay replaces the content until a click, they see only the overlay. Keep overlays light, and check with the view-source test that the page's real text is present without any interaction.

One address per page

The same page can often be reached at several addresses: with and without “www”, with and without a trailing slash, with tracking parameters added by a newsletter tool. Machines may treat these as separate pages with identical content. Choose one form, redirect the others to it, and set a canonical tag on every page pointing to its preferred address. Use the same form in the listings and the structured data. It is a small, technical piece of housekeeping, and it keeps every signal pointing at one copy of each page.

Language and translation

Many Bay Area businesses serve customers in more than one language. If the site has pages in Spanish, Chinese, Vietnamese, Tagalog or another language, make sure each is a real HTML page with the language declared, and that the facts on it match the English pages. A translated page with last year's hours is a stale page in two languages.

Before a redesign goes live

Redesigns are where readable sites become unreadable ones. A new theme may load text by script, move the menu into a PDF, drop the FAQ answers into click-to-load panels, or ship with a robots.txt that blocks everything. Before launch, run the view-source test on the new versions of the five most important pages, compare every fact with the master record, and confirm that the old addresses redirect to the new ones. Then run the same checks again a week after launch, when the caches have cleared and the crawlers have started to return.

An owner's checklist

• Every important fact appears in the page source as text
• Menus, price lists and schedules exist as HTML pages, with any PDF as an extra
• Meaningful images have descriptive alt text
• Facts inside widgets are also stated in text on the page
• FAQ answers are in the HTML before anyone clicks
• Main pages load in a few seconds on a phone
• The mobile layout shows everything the desktop layout shows
• Old pages that are no longer true have been removed or redirected

Related chapters27. AI crawler access · 22. Menus, event schedules and offer lists · 29. Site structure and internal links · 17. Service pages an assistant can quote