Rendered at 08:23:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
terrycody 2 hours ago [-]
This data is useless, total pages online are meaningless, you have to count on the total alive domains with at least some pages, also other very useful data are how many domains with alive traffic, Top 1 million, 10 millions domains...
Some websites can find these data, include ahrefs.com
vivzkestrel 5 hours ago [-]
- very stupid question here
- let us say i want to index all 4 billion pages to generate vector embeddings and make a new search engine
- what does my AWS infra look like?
- how much storage on s3?
- what size of an ec2 instance?
SyneRyder 2 hours ago [-]
From my own attempt at a small personal search engine of the web, which doesn't yet use vector embeddings, each page averages to 10kB of indexed storage.
That would make the final index size of 4 Billion pages about 40 Terabytes. Those charts seem to suggest that's just the size of the Bing index though, and that Google is actually 10x larger at 40 Billion pages. So that would be 400 Terabytes.
My little engine doesn't index the full HTML. If I did, each page averages to 170KB in size, and your index storage just grew 17x.
On a tiny scale - single digit millions - you can get ridiculously far with just SQLite.
vivzkestrel 14 minutes ago [-]
- thank you for sharing that
- another stupid question: how do you about loading data from millions of pages simultaneously
- here is my silly thought process for this:
- get multiprocessing library in python
- combine that with asyncio and aiohttp
- send a whole bunch of requests and save raw html to file storage?
- some big questions...
- how often do you scan the same website
- what headers do you need to add in order to make it not look like some bot or should you actually reveal that you are a search engine bot
- do you need rotating proxies? is something like brightdata or residential proxies used or am I overthinking this?
- I am thinking of taking a small subset of 400 billion pages (like maybe just every blog ever listed on HN) and vector embedding all the text
- what do you think the cloud infra side specifically on AWS (since I am highly familiar with it) would look like?
xacky 2 days ago [-]
99% of the web is just scraper fodder now.
shevy-java 10 hours ago [-]
It indeed has changed. I wonder if we can change it back.
Someone ask Alan Kay how to fix today's web.
ece 4 days ago [-]
So what's with the increased variance since 2021? Better tracking of pages maybe...
shevy-java 10 hours ago [-]
> Since January 15, 2025 we cannot receive any data from Google. They make it very difficult to automatically retrieve data from the search engine
Google became truly Evil. The sooner this evil company is gone the better.
Meanwhile I think we need a new variant of the world wide web. The old one is too easily broken, e. g. by Google and its pathetic attempt to install a private version here. Also aided by AI slop. Plans are inter-connected here.
wackget 4 days ago [-]
> "size of the World Wide Web (The Internet)"
The WWW is not the internet.
Insanity 4 days ago [-]
Yah but the average person (layman) reading this won't know or care much, they'll understand what the website tries to show.
Some websites can find these data, include ahrefs.com
- let us say i want to index all 4 billion pages to generate vector embeddings and make a new search engine
- what does my AWS infra look like?
- how much storage on s3?
- what size of an ec2 instance?
That would make the final index size of 4 Billion pages about 40 Terabytes. Those charts seem to suggest that's just the size of the Bing index though, and that Google is actually 10x larger at 40 Billion pages. So that would be 400 Terabytes.
My little engine doesn't index the full HTML. If I did, each page averages to 170KB in size, and your index storage just grew 17x.
On a tiny scale - single digit millions - you can get ridiculously far with just SQLite.
- another stupid question: how do you about loading data from millions of pages simultaneously
- here is my silly thought process for this:
- get multiprocessing library in python
- combine that with asyncio and aiohttp
- send a whole bunch of requests and save raw html to file storage?
- some big questions...
- how often do you scan the same website
- what headers do you need to add in order to make it not look like some bot or should you actually reveal that you are a search engine bot
- do you need rotating proxies? is something like brightdata or residential proxies used or am I overthinking this?
- I am thinking of taking a small subset of 400 billion pages (like maybe just every blog ever listed on HN) and vector embedding all the text
- what do you think the cloud infra side specifically on AWS (since I am highly familiar with it) would look like?
Someone ask Alan Kay how to fix today's web.
Google became truly Evil. The sooner this evil company is gone the better.
Meanwhile I think we need a new variant of the world wide web. The old one is too easily broken, e. g. by Google and its pathetic attempt to install a private version here. Also aided by AI slop. Plans are inter-connected here.
The WWW is not the internet.