We have known for a long time that Google can crawl web pages up to the first 15MB but now Google updated some of its help ...
It could cause you a lot of problems.
Google updated two of its help documents to clarify how much Googlebot can crawl.
Google Search Advocate John Mueller pushed back on the idea of serving raw Markdown files to LLM crawlers, raising technical concerns on Reddit and calling the concept “a stupid idea” on Bluesky.