Submitted by nastratin t3_124looq in technology
seweso t1_jdztnqd wrote
Reply to comment by theassassintherapist in AI Is Exposing Who Really Has Power in Silicon Valley by nastratin
There is literally a robots.txt which can prevent robots from crawling websites, and then people complain robots crawled websites.... go figure.
Living-blech t1_je15if9 wrote
You can tell a spider program to ignore that file.
Viewing a single comment thread. View all comments