Python

Hiding Keyboard Input From the Screen

Submitted by Bryce on Tue, 03/02/2010 - 00:21

For my current job I'm in charge of writing a script that requires a person to input their password and me being the semi-security conscious individual that I am knew that regular input for a script like this would be bad, as that would allow for over the shoulder (literally) view of someone's password. This is just unacceptable.

To solve this problem I hit the internets for a way to hide user input in Perl script. And in my travels I came across this old PERL FAQ site. I decided to try out the code and update it, throw it into a little dummy script. This is what I got (minus the dummy script):

sub get_password
{
  print "enter your password: ";
  system("stty -echo");
  chop(my $password=<>);
  print "\n";
  system("stty echo");
 
  return $password;
}

Saved it, ran it, and Holy Missing Characters Batman, it worked. So just as a mental exercise, I wanted to see if I could get it to emulate the password request when you run sudo, and I wrote this up in Python (because I need the practice):

#!/usr/bin/python
 
import os
import getpass
 
def get_password():
    #print "Please enter your password:"
    os.system("stty -echo")
    password = raw_input( getpass.getuser() + "'s password")
    os.system("stty echo")
    print "\n"
 
    return password
 
def main():
    local_password = get_password()
    print local_password
 
main()

And again... it worked. Of course I finally realized that the reason why it worked is because of the line "stty -echo". Taking that I typed it into a standard terminal and after hitting enter the results were the same. My terminal functioned normally, but without showing me the characters I was typing. After returning my terminal back to normal I decided to read a little more about stty. After digging through the man page for a little while I came across this argument which is an alias for echo:
[-]crterase echo erase characters as backspace-space-backspace

Thus the reason why I couldn't see anything change is because my terminal is treating all characters I type on the screen as a three part sequence of non alphanumeric characters. Although I find this a little odd; I have to just go with the words of Bugs Bunny on this one, “I don't ask questions. I just have fun.”

2 comments

Tweet Dump part 3

Submitted by Bryce on Mon, 12/21/2009 - 16:06

Welcome back to another, and probably the last, instalment of the tweet dump project.

The old tweet dump code was done locally. I wanted to see how things would be affected if I ran the tweet dump code to a remote server. Just to get an idea on how travel would affect latency.

So I changed things up a bit, I created a sql database through my hosting provider and imported the schema into the remote database. At which point I also modified up the tweet dump code to use this new remote database. And after running the python script 5 times, here are my numbers:

Run 1	43.998
Run 2	46.352
Run 3	45.029
Run 4	55.024
Run 5	49.174
Average	47.92

In my tweetdump code I have a lot of going back and forth between the server and client;

def getId(in_id):
	sql =  "select id from ids where twit_id = '" + in_id + "'"
	return runQuery(sql)
 
def addId(in_id):
	sql= "insert into ids values (null, '" + in_id + "')"
	runQuery(sql)
 
def addTweet(in_id, in_tweet):
	sql = "insert into tweets values ('" + in_id + "',\"" + in_tweet + "\")"
	runQuery(sql)
...
site_id = getId(ids[j])
if( not site_id):
    addId(ids[j])
    site_id = getId(ids[j])

When you see the three functions making up the code block near the bottom, you might realize that there are three seperate sql calls to the server. And while this might be fine for a local database, this is not good for a remote one. So I decided to try my hand at creating a sql function to reduce some of the back and forth between the two entities.

delimiter //
 
drop function if exists `get_set_ids`//
create function get_set_ids( in_tweet_id BIGINT )
RETURNS int(10) unsigned
BEGIN
DECLARE new_id int(10) unsigned;
select id into new_id from ids where twit_id = in_tweet_id;
IF ISNULL(new_id) THEN
     insert into ids values (null, in_tweet_id);
     select id into new_id from ids where twit_id = in_tweet_id;
END IF;
return new_id;
END //
 
delimiter ;

The function above takes the guess work away from the client, and keeps it within the server. By doing this we avoid and entire round of communication between the client & server. So in theory we should see at least some kind of speed-up.

With the function above created I then went and modified the script to take advantage of the function:

def get_set_id(in_id):
	sql = "select get_set_ids(" + in_id + ")"
	return runQuery(sql)
...
site_id = get_set_id(ids[j])
addTweet(str(site_id[0][0]), final_texts[j])

After a bit of testing to make sure things worked, I ran five timed tests (on the same hardware and from the same location, to try and reduce any variables that might crop up).

Run 1	25.948
Run 2	24.35
Run 3	26.181
Run 4	24.667
Run 5	25.352
Average	25.3

The difference between the two averages is about 22 seconds. And to be honest, I did not expect these changes to cut down my times by about half. This was a little bit of a shock to me. I guess in this end this is an example of how design can really matter.

Add new comment

Tweet Dump

Submitted by Bryce on Mon, 10/19/2009 - 21:56

So one day I was browsing the API of Twitter. And on this particular page on Twitter's website, they talked using a command line utility to interact with Twitter. I was intrigued by this, so I decided to write up a simple script to deposit the tweets and twitter ID's into a database. Why? Because I could. ;)

So if for whatever reason you have to recreate my little experiment, you'll need a couple of things:

1.1) grep

2.1) the mysqldb python module installed

So here is my MySQL schema:

-- MySQL dump 10.11
--
-- Host: localhost    Database: twit_bank
-- ------------------------------------------------------
-- Server version	5.0.84-log
 
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
 
--
-- Table structure for table `ids`
--
 
DROP TABLE IF EXISTS `ids`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `ids` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `twit_id` bigint(20) default NULL,
  UNIQUE KEY `id` (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=50250 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
 
--
-- Table structure for table `tweets`
--
 
DROP TABLE IF EXISTS `tweets`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `tweets` (
  `ids` int(10) unsigned default NULL,
  `tweet` char(150) default NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
 
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
 
-- Dump completed on 2009-10-19  0:22:26

And once you have that schema install, you can use the script below (Obviously you'll need to change the username and password to your particular database, but you don't need me to tell you that. My readers are S-M-A-R-T!)

#!/usr/bin/python
 
import os
import subprocess
import MySQLdb
import re
import sys
 
#variables
databaseUser = "*****"
databasePassword = "*****"
databaseName = "twit_bank"
 
 
def runQuery(query):
 
	db = MySQLdb.connect(user=databaseUser, passwd=databasePassword, db=databaseName)
	c = db.cursor()
	c.execute(query)
	db.commit()
 
	return c.fetchall()
 
def getId(in_id):
	sql =  "select id from ids where twit_id = '" + in_id + "'"
	return runQuery(sql)
 
def addId(in_id):
	sql= "insert into ids values (null, '" + in_id + "')"
	runQuery(sql)
 
def addTweet(in_id, in_tweet):
	sql = "insert into tweets values ('" + in_id + "',\"" + in_tweet + "\")"
	runQuery(sql)
 
def main():
 
	final_texts = list()
	j = 0
	site_id = list()
 
#code structure grabbed from: 
#http://buffis.com/2007/01/09/getting-output-of-a-system-call-in-python/comment-page-1/
 
        command = "curl <a href="http://twitter.com/statuses/public_timeline.xml" title="http://twitter.com/statuses/public_timeline.xml">http://twitter.com/statuses/public_timeline.xml</a> | grep -A 3 '<status>' "
	process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
	os.waitpid(process.pid, 0)
	curl = process.stdout.read().strip()
	sys.stdout.flush()
 
	ids = re.findall("[0-9]{10,13}", curl)
	texts = re.findall("<text>[\x20-\x7E]{1,150}", curl)
 
	if ( len(ids) == len(texts)):
		for i in texts:
			temp = i.lstrip('<text>')
			final_texts.append(temp.rstrip('</text>'))
 
		while( j < 19):
			site_id = getId(ids[j])
			if( not site_id):
				addId(ids[j])
				site_id = getId(ids[j])
 
			addTweet(str(site_id[0][0]), final_texts[j])
			j = j + 1
 
main()

And this final last piece to put it all together, so you don't have to do all the grabbing manually.(Don't forget to change the location of the script.)

#!/bin/bash
 
while [ 1 ]
do
/home/bryce/programming/personal/scripts/python/twit2mysql.py
sleep 30
done

Now we get to the fun part. After about 24 hours of grabbing the public timeline at 40 tweets a minute I got 50249 ID's and 50234 tweets on record. Now while watching the numbers grow, there was a weird issue. Most of the time there were more ID's than tweets. I'm not sure why this is at the moment, but I'm curious enough to try and solve this bug.

Oh, and something that people should probably know. The reason why I'm only grabbing 40 tweets a minute is because the first time I tried to run this, I was grabbing more like 120 tweets a minute. And I was blocked by Twitter for viewing the public timeline more than 150 times in one hour. And I was blocked for the rest of the hour. So consider that your warning.

Add new comment

My First Python Script

Submitted by Bryce on Mon, 10/12/2009 - 16:19

Python

I would like to introduce everyone to my first Python script. Just for the sake of full disclosure it was originally put together by a coworker and then passed off to me for completion. I haven't had much interest until recently to even touch Python. I guess all of us should be so lucky to have a job that forces us to grow in unexpected directions every once and a while.

The script itself is nothing overly complicated, but I'm still proud of it. The reason for that is because I had to dissect the language and semantics to figure out what functionality was still needed to be added to the script. As well as learn to read the language in order to work with the parts that were existing before I touched it.

Just a quick overview of what it actually does ( or is at least suppose to do ;) ). In a MySQL database somewhere in the world, there is a database that holds a schema that was created to try and allow for dynamic recreation of Apache virtualhosts config files at the push of a button. (I'm being a little terse regarding things because I really enjoy the project and have an upcoming blog post with more details. So I don't want to spoil all the fun now. :) )

# -*- coding: utf-8 -*-
import commands
import datetime
import unittest
import MySQLdb
import sys
 
###################### Constants #############################
 
# What user should we connect to the database as, for updating tables and such?
databaseUser = "******"
 
# What is the password to the database for the given user?
databasePassword = "******"
 
# What is the name of the database that we are connected to?
databaseName = "apache_config"
 
##################### Begin Code Implementation ###################
 
# Returns a set containing all tuples satisfying the given query.
def runQuery(query):
  db = MySQLdb.connect(user=databaseUser, passwd=databasePassword, db=databaseName)
  #db = MySQLdb.connect(host=host,user=databaseUser, passwd=databasePassword, db=databaseName)
 
  # Execute the query and return the data.
  c = db.cursor()
  c.execute(query)
  return c.fetchall()
 
# Retrieves the master table that holds information on each of the virtualhosts currently being stored.
def retrieveMasterVirtualhostTable(in_virtualhost):
  return runQuery("select name, ipaddress from virtualhosts WHERE name = \"" + in_virtualhost + "\"" )
 
#get all information for each virtualhosts based on virtualhost name
def retrieveSitesbyVirtualhosts(in_virtualhost):
  return runQuery("SELECT virtualhosts.name, virtualhosts.ipaddress, site.site_name, site.docroot, site.admin_email, site.redirects, site.misc, site.site_id, site.misc FROM site,virtualhosts WHERE site.vhost_id = virtualhosts.id AND virtualhosts.name= \"" + in_virtualhost + "\"")
 
# retrieve all aliaeses for a particular site
def retrieveAliasesbysite_id(in_aliases):
  return runQuery("SELECT alias_name FROM site_aliases WHERE site_id= " + in_aliases )
 
# Given a virtualhost entry, returns the header for the associated virtualhost.
def formatVirtualhostHeader(virtualhost):
  return 'NameVirtualHost ' + virtualhost[1] + '\nLogFormat "%v %h %l %u %t \\\"%r\\\" %>s %b" vhost\n'
 
def convertWebsiteToVirtualhostEntry(site):
  if( not site[5] ):
      if ( site[4]): 
          ret= "\n<VirtualHost " + site[1] + ">\n" + \
                "	ServerAdmin " + "webmaster@" + site[2] + "\n" + \
                "	DocumentRoot " + site[3] + "\n" + \
                "	ServerName " + site[2] + "\n"
      else:
          ret= "\n<VirtualHost " + site[1] + ">\n" + \
                "	DocumentRoot " + site[3] + "\n" + \
                "	ServerName " + site[2] + "\n"
 
      # Add in site aliases.
      for alias in retrieveAliasesbysite_id(str(site[7])):
            ret += "	ServerAlias " + alias[0] + "\n"
 
      ret+=\
         "	ErrorLog /usr/local/var/apache/log/" + site[0] + ".error_log\n" + \
         "	CustomLog /usr/local/var/apache/log/" + site[0] + ".access_log vhost\n"
 
      if site[8]:
         ret += site[8] + "\n"
 
  else:
      ret= "\n<VirtualHost " + site[1] + ">\n" + \
            "	ServerName " + site[2] + "\n" 
 
      for alias in retrieveAliasesbysite_id(str(site[7])):
            ret += "	ServerAlias " + alias[0] + "\n"
 
      ret += "        Redirect " + site[5] + "\n"
 
  ret += "</Virtualhost>"      
  return ret
 
# Execution begins here.
def main():
 
  # grab information pertaining 
  virtualhost = retrieveMasterVirtualhostTable(sys.argv[1])
 
  # print the format stuff out at the top of the file
  print formatVirtualhostHeader(virtualhost[0])
 
  #now grab sites based on the virtualhost called
  virtualhostMaster = retrieveSitesbyVirtualhosts(virtualhost[0][0])
 
  # For each site in the virtualhostMaster "list"..
  for v in virtualhostMaster:
 
  ### Print entire output to the screen
    print convertWebsiteToVirtualhostEntry(v)
 
main()

I would just like to point out that although I'm proud of my work. I'm not sharing this to try and receive praise from the internet. I'd probably have better luck getting a STD from the internet than praise for this little script. On the contrary, I'm posting this because I'm new to Python, and I would like someone who has a little more experience to guide me in the places that I might not have used the best python methods to get things accomplished. For example, the line:

virtualhostMaster = retrieveSitesbyVirtualhosts(virtualhost[0][0])

Although this works, my intuition tells me that this isn't a very python way of getting that data, using a double index like that. However I just haven't been able to find the correct way to do this. So if someone out there knows of a better way, I'm all eyes!

Add new comment

Recent Thoughts (a.k.a Tweets)


	@TheColonial Hell, I'm using #git-svn under the radar. Go rebellious streak!!! And it makes a better svn client than svn does. — 4 hours 49 min ago

@TheColonial That's be good when I start using hg again. Right now I'm forced to use git because hg svn is worthless. And work is svn :( — 4 hours 51 min ago
#git aware PS1: PS1='\h:\W$(__git_ps1 "(%s)") \u\$ ' Freaking awesome!!! — 6 hours 16 min ago
@adymitruk That is awesome!! Thanks for sharing! — 5 days 6 hours ago
@TheColonial I might wanna borrow the Pro #Git & #Mercurial books once your done with them. ;) — 5 days 7 hours ago
"#Hella" could become a scientific term. That is freaking awesome! http://bit.ly/bnB0XV — 5 days 8 hours ago

Tags in Catagories

HowTo Linux Mysql Pop Quiz Python games Personal Twitter Flash Bitbucket Humor Gentoo PERL Tutorial

more tags

ScrollingText

Python

Hiding Keyboard Input From the Screen

Tweet Dump part 3

Tweet Dump

My First Python Script

Top 5 most viewed posts

All time:

Last viewed:

Recent Thoughts (a.k.a Tweets)

Google Friend

Tags in Catagories