User Controls
Posts by gadzooks
-
2019-03-11 at 5:47 AM UTC in 90's Thread
-
2019-03-11 at 5:42 AM UTC in 90's Thread
-
2019-03-11 at 5:35 AM UTC in 90's Thread
-
2019-03-11 at 5:34 AM UTC in 90's Thread
-
2019-03-11 at 5:23 AM UTC in 90's Thread
-
2019-03-11 at 5:18 AM UTC in 90's Thread
-
2019-03-11 at 5:16 AM UTC in 90's Thread
-
2019-03-11 at 5:13 AM UTC in License plate
-
2019-03-11 at 5:12 AM UTC in 90's Thread
-
2019-03-11 at 5:10 AM UTC in THis sit =e is dead
Originally posted by -SpectraL Obviously it would have to only output publicly available data for you to piss all over, but that's probably cleaner than fucking parsing the markup and trying to extract the son-of-a-bitch that way.
I don't even mind parsing markup.
I actually kinda like doing it from time to time.
I've spent quite a bit of time knee deep in the NiS HTML/CSS when making my GreaseMonkey script for blocking infinityshock.
The general structure of NiS templates is pretty familiar to me. -
2019-03-11 at 5:07 AM UTC in THis sit =e is deadI don't know django right now.
I could prolly figure it out if I was a bit more sober and clear headed and had a few hours, maybe even days, to go over the docs.
But right now, my script is doing it's thing, and not too inefficiently from what I can tell.
It's actually just about to reach 30% done. -
2019-03-11 at 5:04 AM UTC in THis sit =e is dead
Originally posted by -SpectraL ie:
Yeah, go for it, you fucker. If you want to write a management command(django's mechanism for scripts that don't happen as part of the request/response cycle), but I'll bet you're too dumb for it, to pull it straight from the DB and dump that god damned garbage into some CSV files or something I wouldn't mind running it and just sending your idiotic ass the output instead of you having to scrape everything using that shovel nose of yours. Obviously it would have to only output publicly available data for you to piss all over, but that's probably cleaner than fucking parsing the markup and trying to extract the son-of-a-bitch that way.
Wut?
Nigga I'm drinking scotch, snorting k, and smoking crack right now.
I'm pretty much that right now.
Except my script I wrote while (mostly) sober, so it works. -
2019-03-11 at 4:57 AM UTC in THis sit =e is dead
Originally posted by Lanny Yeah, go for it. If you want to write a management command (django's mechanism for scripts that don't happen as part of the request/response cycle) to pull it straight from the DB and dump it into some CSV files or something I wouldn't mind running it and just sending you the output instead of you having to scrape everything. Obviously it would have to only output publicly available data but that's probably cleaner than parsing the markup and trying to extract content that way.
I appreciate the offer.
I've got a simple script just downloading each HTML response for every iterative thread request (i.e: "thread 1".html, "thread 2.html", etc) up until i == 10,000 for now). That would be close to a third of the entire content. I started running it with a half second interval between requests.
I have some very simple exception handling (the bare essentials), and so my PyCharm console is showing me that it's currently at thread number 511xx. And it's also catching a very vaguely defined exception (on my part) and telling me that every few dozen threads there is a thread not saving... I imagine some threads have been deleted over the course of the years for one reason or another.
But it keeps on truckin'. It's looking like it should actually be totally done the entire site before the night ends, which is pretty good. I'm just brute-saving full HTML files for each thread, and I'll use Beautiful Soup to parse the files locally in some fashion and make some kind of database.
Panny was on my ass the other day about a word cloud I promised I'd make him, so it kinda lit the fire under my ass to just archive all the publicly viewable posts in one fell swoop so that I can analyze and process the data locally.
The only other thing I might want to run a separate script for is extracting member post info stats (basically just username, reg date, post count, and thanks given and thanks received) for each user.
I don't think it will necessitate downloading entire posts all over again. I'll optimize it as best I can. -
2019-03-11 at 4:45 AM UTC in 90's Thread
-
2019-03-11 at 4:40 AM UTC in 90's Thread
-
2019-03-11 at 4:36 AM UTC in 90's Thread
-
2019-03-11 at 4:30 AM UTC in 90's Thread
-
2019-03-11 at 4:27 AM UTC in 90's Thread
-
2019-03-11 at 4:27 AM UTC in SO. FUCKING. HIGH.
-
2019-03-11 at 4:26 AM UTC in 90's Thread
