User Controls
Posts by rabbitweed
-
2020-09-15 at 3:08 AM UTC in Any pythoners in here know how I can speed up this code?Never heard of grequests (I'm straight just like aldra). But I checked out the github, which says:
Note: You should probably use requests-threads or requests-futures instead.
So there we go.
https://github.com/requests/requests-threads
each request is a new thread. -
2020-09-15 at 3:05 AM UTC in Any pythoners in here know how I can speed up this code?
Originally posted by aldra I don't code in python because I'm not a homosexual but you were probably right the first time, this is going to be the bottleneck:
not so much slow code but it'll queue up each page get, you'll want to use threading to run several of them concurrently if at all possible
I completely missed that bit, but you're totally right. That's the worst bottleneck in the whole loop. -
2020-09-15 at 2:23 AM UTC in Any pythoners in here know how I can speed up this code?And most importantly... benchmark now, then benchmark after every optimisation you add. Just because you feel something is faster doesn't mean it actually is.
-
2020-09-15 at 2:22 AM UTC in Any pythoners in here know how I can speed up this code?It's hard to tell without being able to run it here myself, but these should be pretty universal
# Get all articles for the day
for date in pandas.date_range('2011-01-01', '2020-09-01', tz='Europe/London').tolist():
There's no need to transform the date range to a list. A date range is already iterable, and you're just allocating more memory for no reason.
articles += f"title: {content['title']}\narticle: {' '.join(content['article'])}\n"
Never ever use += on an python string. They are immutable so you're reallocating memory every loop. Use join, like you mentioned before. Some info here:
https://waymoot.org/home/python_string/
It was 30 times slower to use += then to use join in this benchmark
with open(f"{os.getcwd()}/scrape/data/dailymail.txt", 'a+') as file:
file.write(articles)
The easiest win is here. You're opening and closing a file every loop, just to write to it. Open it before the loop starts, and close it when the loop ends instead.
If you use threads like sophie suggested, I'd have a queue of articles. One thread adds articles to the queue, another thread takes them from the queue and writes them to the file. -
2020-09-15 at 1:45 AM UTC in Fona © 9-14-2020I have to agree with G here. Lying is great, definitely gets results if you don't over do it.
-
2020-09-15 at 12:43 AM UTC in Can we get a VIP/sane people's forum?
-
2020-09-14 at 11:54 PM UTC in Fona © 9-14-2020
Originally posted by G No brag he asked a question, I answered.
Perfection is a matter of perspective, change yours.
You do nothing here but talk about how great you are. Just like speedy.
It's transparent. Secure people can admit their fuckups, or stuff that isn't so great.
I see through you old man. -
2020-09-14 at 11:50 PM UTC in Fona © 9-14-2020
Originally posted by G Lucrative investments,fruitful income,savings,vibrant sex life,classic whips,salon quality hair & skin.
The more you brag about your life the more we all believe absolutely nothing is wrong and you're perfect.
This is an old school online message board you tard. Not a facebook feed. -
2020-09-14 at 11:41 PM UTC in Fona © 9-14-2020
-
2020-09-14 at 12:10 PM UTC in Fona © 9-14-2020
-
2020-09-14 at 11:40 AM UTC in I'm your favorite space niggaWhat's even the best way of getting acid these days? Get someone to mail it to you on one of them darknet stores?
I'd rather not actually meet with drug dealers IRL. -
2020-09-14 at 11:38 AM UTC in Wokesters FLIP OUT at covid study that breaks down by race
-
2020-09-14 at 10:10 AM UTC in Debtor's Prisons
-
2020-09-14 at 10:07 AM UTC in How does this make you feel, bigots?Grylls is so fucking stupid he couldn't even figure out it was a location, let alone that it was a Greek word.
-
2020-09-14 at 10:02 AM UTC in How does this make you feel, bigots?Durrr where is Thessalonika durrr
-
2020-09-14 at 9:53 AM UTC in Fona © 9-14-2020I wish you nothing but success and happiness.
I also hope I never have to live near you because smokers are gross. -
2020-09-14 at 9:45 AM UTC in I'm your favorite space nigga
Originally posted by MORALLY SUPERIOR BEING 2020 IV: Intravenous Soyposting OP literally made this thread to shit on non-black people.
You know me, I'm not politically correct. I've never seen bummy shitting on white people, except all the drugged up losers here who call him 'nigger' (even though he probably earns more than someone like grylls, US pay > UK pay).
He's pretty chill. He should stop having sex with single mothers or reacting to six letter words, but none of us are perfect. -
2020-09-14 at 9:35 AM UTC in My Girlfriend wants to see TaiwanYou cannot travel to Taiwan with a PRC passport.
I've been to Taiwan. China doesn't run it at all.
It's an interesting place. The land is beautiful, the towns built on it are hideous looking, save for the odd brightly coloured temple. Public transport is amazing, crime is low, driving is pretty bad, everyone's pretty friendly, people are very relaxed but constantly doing dangerous shit like 3 men riding a scooter without helmets. It's first world but their hearts are still in the third world. It's great and crappy at the same time. I recommend it. -
2020-09-14 at 7:45 AM UTC in I'm your favorite space nigga
-
2020-09-14 at 7:25 AM UTC in I'm your favorite space nigga
