Unusual SSL Error when opening page with Mechanize.

2017-04-30 at 2:51 AM UTC

#1

Sophie Pedophile Tech Support

So, as perhaps was evident by the content of my other thread i am writing a sort of CLI for HoneyDB. I am using Mechanize to retrieve Threat Feed and everything seems to be working fine except for something in the way mechanize or urllib2 or whatever mechanize depends on, handles SSL.

Running this code:


import mechanize
from blessings import Terminal

t = Terminal()

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('user-agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'),
('X-HoneyDb-ApiId', DB_API_ID), ('X-HoneyDb-ApiKey', DB_API_KEY), ('accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')]

	try:
		response = br.open("https://riskdiscovery.com/honeydb/api/twitter-threat-feed")
	except Exception as e:
		print "\n[" + t.red("!") + "]Critical, could not open HoneyDB"
		print "\n[" + t.green("+") + "]The following status code was recieved: "
		print e
[/code]


Gives me this error:

[code]
<urlopen error Errno 1 _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error>
[/code]

And some nonesense about `response` being referenced before assignment but that's only because opening the page with mechanize gives me this error.

Someone on Stackoverflow mentioned something about the error being in the SSL lib and posted a monkey patch but that didn't seem to work either.

Any help would be appreciated.

[i]Post last edited by Sophie at 2017-05-01T01:05:41.187644+00:00[/i]

2017-04-30 at 6:21 PM UTC

#2

Sophie Pedophile Tech Support

Someone help me get this work around to do what i want


import socket
import httplib
import ssl

def connect(self):
	sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address)
	if self._tunnel_host:
		self.sock = sock
		self._tunnel()

	self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ssl_version=ssl.PROTOCOL_TLSv1)

httplib.HTTPSConnection.connect = connect

The problem appears to be that the SSL lib is out dated. But fuck me if i have to translate my entire program to Python 3.

2017-05-01 at 12:07 AM UTC

#3

Lanny Bird of Courage

It's openssl that's old right? Assuming it dynamically linked have you tried upgrading it?

2017-05-01 at 12:37 AM UTC

#4

The Self Taught Man Black Hole

This is way above my pay grade boys.

2017-05-01 at 1 AM UTC

#5

Sophie Pedophile Tech Support

Originally posted by Lanny It's openssl that's old right? Assuming it dynamically linked have you tried upgrading it?


Sophie@Ubuntu$ sudo pip install --upgrade ssl
Downloading/unpacking ssl
  Downloading ssl-1.16.tar.gz
  Running setup.py (path:/tmp/pip_build_root/ssl/setup.py) egg_info for package ssl
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip_build_root/ssl/setup.py", line 12, in <module>
        + "or earlier.")
    ValueError: This extension should not be used with Python 2.6 or later (already built in), and has not been tested with Python 2.3.4 or earlier.
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip_build_root/ssl/setup.py", line 12, in <module>

2017-05-01 at 1:04 AM UTC

#6

Sophie Pedophile Tech Support

also wtf is with the formatting in OP

2017-05-01 at 9:24 AM UTC

#7

Sophie Pedophile Tech Support

Also honestly i haven't a clue whether the fault is in OpenSSL or not. All i know is i get this error when trying to request the page i want. You over estimate my Comp-Fu Lan Lan. From what i hear the problem is that the implementation of the SSL lib in Python 2.7 is dinky with regards to newer versions of SSL.

2017-05-03 at 4:31 AM UTC

#8

Sophie Pedophile Tech Support

I just realized i could just invoke `curl` with os.system and avoid Python's handling of SSL entirely but that seems like a cop out. Opinions?

2017-05-03 at 7:20 AM UTC

#9

Lanny Bird of Courage

That's not a bad plan. There's also pycurl which uses C interop to libcurl which saves you the one or two process overhead of os.system. You also don't have to fuck around with passing strings to the invoked process via argv.

2017-05-03 at 7:41 AM UTC

#10

Sophie Pedophile Tech Support

Originally posted by Lanny That's not a bad plan. There's also pycurl which uses C interop to libcurl which saves you the one or two process overhead of os.system. You also don't have to fuck around with passing strings to the invoked process via argv.

Oh yeah i forgot PyCurl was a thing. Let me read up on the docs for that, i think i will go with that. Now i feel embarrassed about posting shitty source to GH, what will my kouhais and senpais think of me.

2017-05-03 at 9:04 AM UTC

#11

Sophie Pedophile Tech Support

Ok so, guess what. My PyCurl implementation did not work with OpenSSL either but i rebuilt from source so that it does. But now i am having an issue with the way PyCurl objects work. they are NoneType objects so thjey are not iterable. If i write out the data as is i just get a long ass string. I can't just split a string in a file so i wite the data out to a file then read it back in but i can't split on a FileType object either. So i load my FileType object into a list but guess what a list object does not have an attribute "split".


import os
import blessings
import time
import pycurl

c.setopt(pycurl.URL, "https://riskdiscovery.com/honeydb/api/twitter-threat-feed")
feed = c.perform()
			
os.system("clear")
print "\n\n[" + t.green("+") + "]Retrieved Threat Feed, formatting..."
time.sleep(1)
			
outfile_one = open('feed.txt', 'wb')
c.setopt(c.WRITEDATA, outfile_one)
c.perform()
outfile_one.close()
			
infile = open('feed.txt', 'r')
#infile.read()
			
str_obj = []
			
for item in infile:
    str_obj.append(item)
			
format = []
for x in str_obj:
	x.split(',')
	str_obj.append(format)
			
print format

2017-05-04 at 2:14 AM UTC

#12

Lanny Bird of Courage

What do the contents of feed.txt look like? I'm guessing it's JSON so you don't want to try to split it by hand. I'd try:

import os
import blessings
import time
import pycurl
import StringIO
import json

b = StringIO()
c.setopt(pycurl.URL, "https://riskdiscovery.com/honeydb/api/twitter-threat-feed")
c.setopt(c.WRITEDATA, b)
feed = c.perform()
			
os.system("clear")
print "\n\n[" + t.green("+") + "]Retrieved Threat Feed, formatting..."
time.sleep(1)

response = json.dumps(b. getvalue())

print response

Note the use of StringIO in place of writing to a file. No functional difference there but it should be a bit faster and reads better. But if that doesn't work we'd need to know what the response looked like before I could say how to parse it.

2017-05-04 at 2:31 AM UTC

#13

Sophie Pedophile Tech Support

Originally posted by Lanny What do the contents of feed.txt look like? I'm guessing it's JSON so you don't want to try to split it by hand. I'd try:
import os
import blessings
import time
import pycurl
import StringIO
import json

b = StringIO()
c.setopt(pycurl.URL, "https://riskdiscovery.com/honeydb/api/twitter-threat-feed")
c.setopt(c.WRITEDATA, b)
feed = c.perform()
			
os.system("clear")
print "\n\n[" + t.green("+") + "]Retrieved Threat Feed, formatting..."
time.sleep(1)

response = json.dumps(b. getvalue())

print response
Note the use of StringIO in place of writing to a file. No functional difference there but it should be a bit faster and reads better. But if that doesn't work we'd need to know what the response looked like before I could say how to parse it.

Alright so the syntax for my version of Python is `b = StringIO.StringIO()`.

here is a response sample.


{"remote_host":"123.207.118.34","count":"1","last_seen":"2017-05-03 19:04:03"},{"remote_host":"183.6.167.106","count":"1","last_seen":"2017-05-02 12:47:31"},{"remote_host":"70.105.61.162","count":"1","last_seen":"2017-05-03 23:25:53"},{"remote_host":"115.52.136.171","count":"1","last_seen":"2017-05-02 20:20:53"},{"remote_host":"167.114.222.205","count":"1","last_seen":"2017-05-02 13:25:46"},{"remote_host":"2.178.92.129","count":"1","last_seen":"2017-05-02 11:39:24"},{"remote_host":"46.41.71.19","count":"1","last_seen":"2017-05-02 05:40:13"},{"remote_host":"123.207.125.74","count":"1","last_seen":"2017-05-03 20:31:42"},{"remote_host":"71.6.216.41","count":"1","last_seen":"2017-05-02 13:21:21"},{"remote_host":"221.15.51.46","count":"1","last_seen":"2017-05-02 17:27:07"}]

I just copypasted what you posted with the exception of the syntax for the `b` var. When i print response i get `" "` as a result. What i also don't understand is how `response = json.dumps(b. getvalue())` is supposed to know we want the value for the `feed` var. I tried assigning `feed` as an argument like so `response = json.dumps(b. getvalue(feed))` but then it just tells me that the function takes exactly one argument and that there are two given.

2017-05-04 at 3:03 AM UTC

#14

Lanny Bird of Courage

oh shoot, my bad, change "dumps" to "loads" (I'm just sure there's a joke in there somewhere).

You don't need to capture the return value of c.perform() in a variable, because you did set the WRITEDATA opt before, when you call .perform() the response will be written into `b`.

2017-05-04 at 4:04 AM UTC

#15

Sophie Pedophile Tech Support

Originally posted by Lanny oh shoot, my bad, change "dumps" to "loads" (I'm just sure there's a joke in there somewhere).

You don't need to capture the return value of c.perform() in a variable, because you did set the WRITEDATA opt before, when you call .perform() the response will be written into `b`.

Ok dank, that seems to work like it should. Printing `response` gives me the JSON now. But there's nothing written to my directory. Do i just write out `response` to a file instead?

2017-05-04 at 4:19 AM UTC

#16

Sophie Pedophile Tech Support

Oh it's dictionary comprehension time again.

2017-05-04 at 4:35 AM UTC

#17

Lanny Bird of Courage

do you want to write it to a file? If so then you can go back to your prior approach, but if you want to take some kind of programmatic action based on the response, or format it into something more manageable, then keeping it as a dictionary is the way to go.

2017-05-04 at 4:39 AM UTC

#18

Sophie Pedophile Tech Support

Originally posted by Lanny do you want to write it to a file? If so then you can go back to your prior approach, but if you want to take some kind of programmatic action based on the response, or format it into something more manageable, then keeping it as a dictionary is the way to go.

I would like to format it prettily and print out each element with their corresponding value all nicely to a textfile.

2017-05-04 at 6:09 AM UTC

#19

Sophie Pedophile Tech Support

WE'RE ALMOST THERE NIGGA


import StringIO
import json
import pycurl
from blessings import Terminal

t = Terminal()
c = pycurl.Curl()
b = StringIO.StringIO()


def hosts():
	# Options for PyCurl
	opts = ['X-HoneyDb-ApiId: ' + DB_API_ID, 'X-HoneyDb-ApiKey: ' + DB_API_KEY]
	c.setopt(pycurl.HTTPHEADER, (opts))
	c.setopt(pycurl.FOLLOWLOCATION, 1)
	
	c.setopt(pycurl.URL, "https://riskdiscovery.com/honeydb/api/bad-hosts")
	c.setopt(c.WRITEDATA, b)
	c.perform()
			
			
	os.system("clear")
	print "\n\n[" + t.green("+") + "]Retrieved Threat Feed, formatting..."
	time.sleep(1)
			
	response_h = json.loads(b. getvalue())
	pprint(response_h)
			
	format = json.dumps(response_h, indent = 2)

	with open('hosts.log', 'ab') as outfile:
		outfile.write(format)
		outfile.close()
				
	print "\n\nResults saved to 'hosts.log' in the current directory"

def feed():
	# Options for PyCurl
	opts = ['X-HoneyDb-ApiId: ' + DB_API_ID, 'X-HoneyDb-ApiKey: ' + DB_API_KEY]
	c.setopt(pycurl.HTTPHEADER, (opts))
	c.setopt(pycurl.FOLLOWLOCATION, 1)
			
	c.setopt(pycurl.URL, "https://riskdiscovery.com/honeydb/api/twitter-threat-feed")
	c.setopt(c.WRITEDATA, b)
	c.perform()
			
	os.system("clear")
	print "\n\n[" + t.green("+") + "]Retrieved Threat Feed, formatting..."
	time.sleep(1)
			
	response_f = json.loads(b. getvalue())
	pprint(response_f)
			
	format = json.dumps(response_f, indent = 2)

	with open('feed.log', 'ab') as outfile:
		outfile.write(format)
		outfile.close()
		
	
	print "\n\nResults saved to 'feed.log' in the current directory"

So the reason i have these values that could just be global instead of duplicate in teh fucntions is because once i load something into the JSON parser it wants to stay there. So i thought it was an issue of PyCurl which is why i attempt to reset all the options in their respective functions. But like i said the issue is with the JSON module.


  File "mimir.py", line 163, in <module>
    hosts()
  File "mimir.py", line 108, in hosts
    response_h = json.loads(b. getvalue())
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 369, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 718979 - line 2 column 1 (char 718978 - 1027616)

How do i reset this?

2017-05-05 at 2:09 AM UTC

#20

Lanny Bird of Courage

The issue is that `b` is reused in both feed() and hosts(), you're writing both responses to the same buffer so the contents of `b` by the time you run `b.getvalue()` in hosts() is like "{... response from first request ...}{... response from second request ...}" which is not valid JSON.

I'm not sure why you mean by "once i load something into the JSON parser it wants to stay there" but you definitely need to create a new StringIO instance before each request.

The following users say it would be alright if the author of this post didn't die in a fire!

Sophie

User Controls

Navigation

Unusual SSL Error when opening page with Mechanize.