Yeah, that’s a problem. So even if you want to search your local newspaper, you won’t be able to because of bot protection. Your crawler will be blocked. You also can’t put the whole of Reddit or even Lemmy in your search index.
I do think this is one of the places where yacy has it’s strength since yacy is p2p you can retrieve results from other nodes that have indexed the page thus large websites can be collectively indexed by many nodes.
I also don’t remember having problems with the crawler getting blocked but I also didn’t do a lot of crawls I can see this being a problem though.
Yeah, that’s a problem. So even if you want to search your local newspaper, you won’t be able to because of bot protection. Your crawler will be blocked. You also can’t put the whole of Reddit or even Lemmy in your search index.
I do think this is one of the places where yacy has it’s strength since yacy is p2p you can retrieve results from other nodes that have indexed the page thus large websites can be collectively indexed by many nodes.
I also don’t remember having problems with the crawler getting blocked but I also didn’t do a lot of crawls I can see this being a problem though.