spin up YaCY, see for yourself. It turns into a full time project, because there’s always ranking tweaks and and indexing startpoint changes to be made.
It’s relatively easy to put together a crawler for specific purposes, but a general web search is hard, and requires a lot of storage.
spin up YaCY, see for yourself. It turns into a full time project, because there’s always ranking tweaks and and indexing startpoint changes to be made.
It’s relatively easy to put together a crawler for specific purposes, but a general web search is hard, and requires a lot of storage.