Django Linkage

If you’re into web development and looking for a clean yet simple way to get the job done faster and be more productive, Django may be the web framework for you. Based on the Python programming language, it makes web development stupidly easy while giving access to hunderds of useful Python libraries.

I have used Django for over 6 months at my previous place of employment and it made my life so much easier!

The Django community is very active and still growing. Have a look at my Django People profile that shows some of the projects I have contributed to that use this awesome framework. Django People is an excellent place to find like-minded people and become part of the community.

BluMarble is a great blog that has plenty of tutorials and snippets of code written in Django and Python. The guy that runs the blog is a friend of mine and has mentored me while I was working alongside him.

This Week in Django and Django Dose are 2 sites that I have come across recently that seems to have some good material regarding Django. There’s also Django Sites that shows some of the sites that has been developed in Django. There’s even some open source sites in there!

Now all that’s left is for me to get back into Django and create a few awesome web apps

How To Create A Python Web Bot

About a week ago I was rather frustrated with how I had to do things and the amount of manual effort I needed to put in for a small amount of information.

So I decided to write a little web bot to do all of the work for me. Here is the juicy bits that does all the of the initial http requests and form login:

import urllib
import urllib2
import sys
import cookielib
import getpass

from datetime import date, timedelta
from ClientForm import ParseResponse

def create_proxy(user_pass = None):
  ''' Create a proxy handler since all external http requests has to go through here '''
  win_username = getpass.getuser()
  print 'Please enter your password ' + win_username

  if user_pass is not None:
    win_password = user_pass
  else:
    win_password = getpass.getpass()

  proxy_info = {
  'user' : win_username,
  'pass' : win_password,
  'host' : "proxy.domain_name.co.za",
  'port' : 8080 #proxy port
  }

  return urllib2.ProxyHandler({"http": "http://%(user)s:%(pass)s@%(host)s:%(port)d" % proxy_info})

def setup_http_handler(user_pass = None):
  ''' Create the http handler and authentication '''
  #Create a cookie (session) so that this script does not get logged off while requesting data
  cookieJar = cookielib.LWPCookieJar()

  #Build a new opener that uses a proxy requiring authorization
  proxy_support = create_proxy(user_pass)
  print 'Processing...'
  opener = urllib2.build_opener(proxy_support, urllib2.HTTPCookieProcessor(cookieJar))

#Install it
  urllib2.install_opener(opener)

#Use it
  try:
    f = urllib2.urlopen('http://www.website_to_login.co.za/admin')
  except:
    print 'Please check your password or proxy settings.'
    sys.exit(2)

  #We're using the ClientForm library to handle our form processing
  forms = ParseResponse(f, backwards_compat=False)
  form = forms[0]

  #Set the form elements values (view source of the page to find elements' names)
  form["name"] = "username"
  form["pwd"] = "password"

  #Form.click() returns a urllib2.Request object (bascially submit the form)
  urllib2.urlopen(form.click()).read()

if __name__=="__main__":
#Login and stuff
  setup_http_handler(user_pass)

Pretty cool huh?

The code is documented so I’m sure it was easy to follow along. The most amazing thing was that I put all of this together using bits and pieces of scattered info from various sources that our little friend Google brought up.

It turns out I don’t need the web bot now but it was a fun exercise! Quite interesting what one can do with a little intension…

Get Username & Prompt For Password

The below function is a cross-platform way of displaying a the logged on username and prompt for the password. Useful if you need os-level authentication!

import getpass

#Print logged on username
print getpass.getuser()

#Prompt for password
print "Please enter password: "
user_password = getpass.getpass()
print "Your password is: " + (user_password)

A Simple Function To Get Last Day Of Month

Here is a simple function to get the last day of the current month:

import calendar
from datetime import date

today = date.today()
last_day_of_month = calendar.monthrange(today.year, today.month)[1]

#Create date yyyy-mm-dd (last day of last month)
last_day = date(today.year, today.month, last_day_of_month)

print last_day

Clean A Dirty Text File Into CSV Using Python

So since I’ve been working in the BI team at MMSA in Durbs there wasn’t many requests for me to export data to csv. Now as simple as this sounds it is very difficult doing this on millions of rows. Since I’m still a nOOb at Oracle and datawarehousing, it proved to be a challenge completing this task.

One of the barriers is that there is an access limit on the production system that basically won’t let you put too much pressure on it (an I/O usage restriction). So what I have to do is dump the data at night or during certain times. Using the SQL Plus tool I spooled the data to a text file. The problem is that I didn’t set the page size and a whole host of other commands that would write the data in a much easier to work with format.

Unfortunately I was in such a rush and ended up with a very dirty text file. Since I know Python I thought maybe I can use my skills to clean the file. I spent about 4-5 hours on this little baby:

def make_csv_row(temp):
 count = 1
 the_row = ''

 for t in temp:
  t = t.replace('\t',' ')
  t = '"|"'.join(t.split())

  t = '"' + t + '"'
  if count != 4:
   t = t + "|"

  count += 1
  the_row += t
 return the_row

print 'Opening files'

OPEN_FILE = 'AUG08.csv'
OUT_FILE = 'AUG08_step_1.csv'

f = open(OPEN_FILE, 'rb') # file to read
fout = open(OUT_FILE, 'w') # file to write

print 'Process file'

counter = 0
for line in f.readlines():
 if not(line.find('---------') != -1 or line.find('FIELD1') != -1 or line.find('FIELD2') != -1 or line.find('FIELD3') != -1 or line.find('FIELD4) != -1):
  if line != ('\r\n'):
   fout.write(line)

print 'Completed'
# close all files
f.close()
fout.close()

print 'process step 2'
f = open(OUT_FILE, 'rb') # file to read
fout = open(OPEN_FILE, 'w') # file to write

count = 1
temp = []
fout.write('"FIELD1"|"FIELD2"|"FIELD3"'  + '\r\n')

for line in f.readlines():
 temp.append(line.replace('\r\n', '')) 

 if count == 4:
  t = make_csv_row(temp)
  fout.write(str(t) + '\r\n')
  count = 0
  temp = []
 count = count + 1

f.close()
fout.close()

Please don’t even ask what this does since it is too much of an effort to explain and still confuses me. Unfortunately this script almost worked but there was one little bug that prevented me from using it. So close yet so far.

There are probably a lot of refinements that could be done with this code but remember that I hadn’t touched Python in almost 6 months.