ScrollingText

That “series of tubes” known as the Internet went nuts two weeks ago because Google now allows you to use their search engine through https. I consider this a good thing; the more traffic that flows around encrypted the better. Yeah, there is some overhead to it, but everything has a price and I feel the price is well worth paying.

Of course there are a slew of blogs out there telling Firefox users how to use this feature. Saying pretty much the same thing: install a search engine plugin (of which there are 10 on Add-ons for Firefox) and you're good to go. But there is a small problem; I don’t use the search box. I type my searches in the address field and the one plugin that I did try didn’t change this - I was still searching unencrypted. So like a good geek I went searching for an answer and found it. It's a little preference called keyword.URL

All you have to do is:

• (in a new window or tab) open up about:config
• Promise to be really careful ;)
• Type “keyword” in the text field at the top and you’ll get back two entries:
  1. keyword.URL
  2. keyword.enabled
• You’ll notice the default of keyword.URL as :
• http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=

  And we’ll just make a small change of adding an “s” to the end of http:

  https://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=

• verify that “keyword.enabled” is true, otherwise this little hack won’t work.

And that is all. Now the next time you use the address bar to search, you to will be searching through https. One note though for vimperator users - this trick unfortunately doesn’t work (yeah, I’m bummed about that too). I’m not sure why and I hope to figure that out next, but for now we’ll just have to wait a little longer.

I'd just like to share with you these forty-seven lines of python code that amaze me. Not because they do anything special, but because they represent something that is pretty important to programming languages, modules, or libraries (depending on the language). Let's take a quick gander at the code:

#!/usr/bin/python
"""
#
# This is a simple script to grab the contents of a a website, 
# encoded into a MIME message, and email it
"""
import sys
import smtplib
import pycurl
from cStringIO import StringIO
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
 
data_buf = StringIO()
curl = pycurl.Curl()
 
# Setup pycurl to grab the data
curl.setopt(pycurl.URL, sys.argv[1])
curl.setopt(pycurl.WRITEFUNCTION, data_buf.write )
curl.perform()
curl.close()
 
# Begin creating email
#create html & text parts of the email
part1 = MIMEText(data_buf.getvalue(), 'html')
part2 = MIMEText(data_buf.getvalue(), 'text')
 
# next 5 lines put it all together
msg = MIMEMultipart('alternative')
msg['Subject'] = 'Website by Email'
 
msg.attach(part1)
msg.attach(part2)
 
# email away
# this code was copied from:
# <a href="http://www.mkyong.com/python/how-do-send-email-in-python-via-smtplib/<br />
#" title="http://www.mkyong.com/python/how-do-send-email-in-python-via-smtplib/<br />
#">http://www.mkyong.com/python/how-do-send-email-in-python-via-smtplib/<br />
#</a> Geshi keeps adding extra lines here, I'm not sure why
to = 'add_your_own'
gmail_user = 'add_your_own'
gmail_pwd = 'add_your_own'
smtpserver = smtplib.SMTP("smtp.gmail.com", 587)
smtpserver.ehlo()
smtpserver.starttls()
smtpserver.login(gmail_user, gmail_pwd)
smtpserver.sendmail(gmail_user, to, msg.as_string())
print 'done!'
smtpserver.quit()

This little script started out as a work assignment (which I have modified to make it more general). The task was to grab the contents of a particular website and email them. Not terribly complicated, but also not one I was thinking would be accomplished with twenty-eight lines of actual code.

Let's take a second to review which modules are being used above:

The code used by PycURL to grab the website, and deposit the data into the string.

StringIO module, to hold the contents of the website. (I wasn't able to get a regular string to work here. If you know how, please tell me.)

The MIME module for reformatting the website contents.

The SMTPlib module, for having all the code in it to properly communicate with a SMTP server, including TLS for encryption, to send said email through the web.

It wasn't until doing this project that I fully realized the importance of modules. It was these four modules that saved me uncountable hours of coding, testing, and debugging. Even if I had written out the functionality I needed from scratch, what I would have written would not have anywhere near the functionality that these other modules provided. In correlation to this, I wonder if there is some kind of connection between how popular a lanugage is and how easily extendable it is. I have no way to prove this of course, but both Perl & Python could be good examples (and also happen to be the languages I'm most familiar with). Both languages are popular, as shown by the normalized graph by langpop.com, and both are also extremely easy to add extra functionality to. The PycURL, CLyther, and PyCUDA modules are great cases in point. PycURL allows Python to tap into the CURL library. CLyther allows Python to use OpenCL, and PyCUDA allows Python to access the CUDA libraries. It makes my head hurt just to think about the amount of code that it would take to perform these same functions, if written from scratch.

After having this moment of realization, I find I am extremely grateful to all the programmers out there who put in their time to help create modules like these and to help them perform as well as they do. I tip my hat at you all.

Even though Poisonbit's solution to the original problem is the fastest (Thanks again PoisenBit!) I decided to use the original code as an excuse to learn multi-threading programming in PERL and see if it might improve the performance of the original code. One of those "for shits and giggles" moments.

First off, here is the new and improved code:

#!/usr/bin/perl
#######################################################################
#	Created By: Bryce Verdier
#	on 4/14/10
# 
#	Function: grab all installed packages, using threads
#           find their exact versions, and display them
#	NOTE: FOR USE ON DEBIAN BASED MACHINES
#######################################################################
 
use threads;
 
my $temp_pack;
my $temp_ver;
my $returned_version;
my %pack_hash :shared;
my $thread_count = 0;
my $pack_count;
my @return = `dpkg --get-selections`;
 
sub get_package_ver
{
  my %args = @_;
  my $temp_ver;
  my $returned_ver = `dpkg -s $args{package}`;
 
  $returned_ver =~ m/^Version: (.+)$/m;
 
  $temp_ver = $1;
 
  lock($args{hash});
  $args{hash}{$args{package}} = $temp_ver;
}
 
foreach (@return)
{
  $_ =~ m/^(\S+)[ \t].*/;
 
  $_ = $1;
}
 
 
$pack_count = @return;
while ( $pack_count - $thread_count >= 1)
{
    my $th1 = threads->create(\&get_package_ver, hash => \%pack_hash, 
                              package => $return[$thread_count]);
    my $th2 = threads->create(\&get_package_ver, hash => \%pack_hash, 
                              package => $return[$thread_count+1]);
 
    $th1->join();
    $th2->join();
 
    $thread_count = $thread_count + 2;
}
 
# Get the odd package, if there is one
if ($pack_count - $thread_count == 1)
{
    my $th1 = threads->create(\&get_package_ver, hash => \%pack_hash, 
                              package => $return[$thread_count]);
 
    $th1->join();
 
    $thread_count++;
}
 
while((my $key, my $value) = each(%pack_hash))
{
  print "$key : $value\n";
}

For all the number crunchers out there, putting things into two threads reduced the program execution time by more than 1 minute. For a program that took two and a half seconds to complete, a reduction to one and a half seconds is pretty significant.Well, in my book anyway.

After first following the suggestions of Sam for the regexes, (Thanks again Sam!) I pulled out the version checking code into its own function so that each thread would have a very specific thing to do. After that I realized I needed to clean up the package names that were being sent to the threads, creating the foreach loop on line 35. Within the foreach loop I tried something that I didn't expect to work - the line:

$_ = $1;

There isn't a reason for the line above to not work, but in the PERL code I've seen "$_" is not being used as a pointer to write data to, only to retrieve data from. And I guess that is why I did not expect it to work. But then, that's why I'm doing this - to learn things. :-D

After these changes and adding the threads code, I was almost done. For some reason the data from each thread wasn't getting stored into the hash. It wasn't until I looked a little deeper at the examples on perldoc that I saw what I needed to do. In the section "Shared And Unshared Data" I noticed I needed to mark %pack_hash as shared, so all the threads could access it. Which I did like so:

my %pack_hash :shared;

All in all, my first multi-threading coding expirence in perl wasn't bad. Granted the program isn't complicated, but this was truely new territory for me. I haven't tried doing any kind of multi-process/multi-threading programming since my operating systems class almost 3 years ago, so there were some battles to fight in my head on how to modify things to work with multiple threads. But again, it was a good experience. And I'm going to reiterate this so everyone remembers: don't use this code in production. Poisonbit's solution is MUCH faster than mine. Like me, use this code for learning.

AAAHHH work! Everyone has those horror stories where your boss, or the client, comes and asks for some horrible feature that will require an entire rewrite of the program. Fortunately, that hasn't happened to me (yet) and that is not what this blog entry is about. It's about the even more (what I believe to be) unlikely scenario when someone wants a feature that they think will be difficult to create and after some research you implement that feature in a small amount of time. It's a rare event and great confidence booster when it does happen though.

A co-worker wanted a feature added to a project. He thought that it might take a while to complete so he talked to me first about it to “plant the seed” and get my brain started on figuring out a solution, not expecting a quick turnaround. Of course, as the title hints at, the problem was to find out all the packages installed on our Debian boxes as well as their version numbers. So after a little bit of googleing and man page reading I had a basic algorithm to build on. Twenty minutes of coding and testing later, I had this script:

#!/usr/bin/perl
#######################################################################
#	Created By: Bryce Verdier
#	on 4/14/10
# 
#	Function: grab all installed packages, find their exact
#           versions, and display them
#	NOTE: FOR USE ON DEBIAN BASED MACHINES
#######################################################################
 
my $temp_pack;
my $temp_ver;
my $returned_version;
my %pack_hash;
my @return = `dpkg --get-selections`;
 
foreach (@return)
{
  $_ =~ m/(\S*)[ \t].*/i;
 
  $temp_pack = $1;
 
  $returned_version = `dpkg -s $temp_pack`;
 
  $returned_version =~ m/Version: (.*)/i;
 
  $temp_ver = $1;
 
  $pack_hash{$temp_pack} = $temp_ver;
 
}
 
while((my $key, my $value) = each(%pack_hash))
{
  print "$key : $value\n";
}

I will admit that is a little slow (on my desktop it takes around two and a half minutes to complete) and could probably benefit from some parallelization. However, that might be over-engineering for such a simple task. I'll code that feature up next, grab a stopwatch, and test it just to find out. Anybody gonna place bets one way or another? In the meantime, I'm proud of my use of regex's in this script, the "\S" removes the trailing whitespace from the package names, instead of just using ".*", which should speed things up a bit because I don't have to call chomp on each package name. Also using a hash for storage simplifies the data management and should allow for an easier time porting the code into a larger script later.