Viewing a single comment thread. View all comments

dmart89 OP t1_j9nkm2u wrote

Reply to comment by noxiousmomentum in [D] Python library to collect structured datasets across the internet by dmart89

Fair. Thanks for your thoughts. I personally find constructing scrapers and parsing data annoyingly tedious, but it's probably just me (:

2

ch9ki7 t1_j9nw6hu wrote

building and maintaining scrapers is tedious! I would also like some better solution. the idea is not bad, just maybe difficult to solve.

3

dmart89 OP t1_j9olr3r wrote

Possibly, yes, I would need to check. I recently built parsing services for tiktok, and it was super annoying to deal with.

1

ch9ki7 t1_j9oqe44 wrote

maybe something like scraperapi but with some kind of Dsl one could send as post payload.

but als a Problem is that you often need a scraped result as input for another request

1

step21 t1_j9nwh4u wrote

Also, some of it might give you legal trouble if you f e make a public crawler for linkedin

2

dmart89 OP t1_j9olf7e wrote

There was a court ruling a year or two ago that concluded that scraping public linkedin profiles is legal :) LN obviously still doesn't want you to scrape their data, so building scrapers for it is extra tedious because you need to navigate their blocking.

2

KPTN25 t1_j9qy2xi wrote

> court ruling a year or two ago that concluded that scraping public linkedin profiles is legal

Forgot about this. I may be dating myself with problems of the past.

Still imagine they're doing their best to make it really hard to do, though.

1