Monday, December 16, 2013

Raspberry Pi, CherryPy and asynchronous communication

I have a project I am working on where graphical output is required to be sent to the screen. Using Python, where by default the output only goes to the console, one then immediately thinks of Tkinter, which is a Python module that provides a graphical interface to Python.

However, that would mean learning yet another ‘language’ as such. So I decided to pursue a different avenue. I am fairly familiar with HTML and JavaScript, even though the latter is extremely quirky and non-intuitive. Therefore, I let my gaze wander to a browser based solution.

Now what I am looking for goes beyond the request-response model usually found in browser based interfaces: the user clicks on a button (or link), the browser sends the information to the server, the server responds by sending the requested information and the connection closes. A classic case of synchronous communication.

What I am looking for is a scenario where the browser’s window is automatically updated with new information as it becomes available and no if any user action is required. Asynchronous communication.

So I started researching. What I really wanted was a short, sweet, concise code example of how to do asynchronous communication with Python. There were lots of examples, code pieces, snippets for Bottle, Flask, Tornado (all Python based webservers), but nothing that was simple enough to just run.

I even spent some time installing node.js, which is supposed to be the next best thing since HTML. Once installed though, I couldn’t find any examples that were straight forward enough or worked ‘out-of-the-box’.

Finally, I hit on CherryPy, yet another Python based web server module, specifically this page in their wiki:http://tools.cherrypy.org/wiki/Comet. On this page, author Dan McDougall discusses CherryPy approach to ‘Comet’. Comet (a cleanser). Ajax (a cleanser), however in IT lingo, Ajax can provide feedback to a browser without the entire browser window having to be refreshed. I decided to download CherryPy, run the application in Python, open a browser and voila: it worked. Success, end of a long searching marathon.

CherryPy does multithreading on its own. At its peak I had three web browsers open on various machines and the CPU Usage meter barely moved. Another neat feature that I discovered is that when I saved a new version of the application in Geany, with the old application still running, it will shut the old application down and start the new one all on its own.

The output of the application I downloaded spits out the results of pings to an ip address or web site you provide. The results just keep on streaming by on the screen. Great example, but not what I needed: what I wanted was a number of string that simply replaced the previous string, i.e. automatic updating of a value. More searching. This time I stumbled on a neat solution provided by Encosia (Dave Ward), which I adapted to work with my application.

The original CherryPy application created an iframe at the bottom of the HTML (just before the </body> tag). The asynchronous info being sent from the server gets displayed there. An iframe is really a webpage within your web page, usually used for ads.

Dave Ward’s solution is to hide the iframe (style=”display:none”), then have the web server send a function call embedded in the asynchronous communication along with the latest value of whatever we are trying to display. The function called is on our main web page (‘the parent’) and so can update anything on that page. The iframe is still there and working, but invisible to us.

In my case, I have a table and the content of the function simply reads:

document.getElementById(‘11’).innerHTML = cTimeReceived

where ‘11’ is the id of the first table data cell and cTimeReceived the new value received from the server.

I tested this on the Raspberry Pi’s various assortment of web browsers. It worked properly on Midori, Chromium, Luakit, Iceweasel but not on Dillo or Netsurfer. I let it run for hours without a problem. I ran it from a laptop elsewhere and  intentionally interrupted the network connection, and, once the connection was re-established, the page kept right on going.

On the Pi, Luakit by far required the least amount of resources (around 30% in CPU Usage Monitor), while Chromium maxed out at 100%, the others somewhere in between.

Obligatory screen shot:

cherrypyscreenshot

Below the application. I ran this on the Raspberry Pi Model B, 512 Mb, with Raspbian updated to 2013-12-15. For it to run, you need to have Python (my version 2.7). To install CherryPy:

pip install cherrypy

Then copy the application below, change my IP address (towards the bottom) to your own, change my ‘pythonprogs’ directory name to you directory name and run. Open up a browser, point to your IP address, followed by :8080 and press enter. Once the page appears click’ Start’. That’s it!

Note that the HTML for the starting page is embedded in the application, hence 1 file is all you need to run this example.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# string.Template requires Python 2.4+
from string import Template
import cherrypy
import time,datetime

__author__ = 'Dan McDougall <YouKnowWho@YouKnowWhat.com>'

# Trying to cut down on long lines...
jquery_url = 'http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js'
jquery_ui_url = 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js'
jquery_ui_css_url = \
'http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/themes/black-tie/jquery-ui.css'

class Comet(object):
"""An example of using CherryPy for Comet-style asynchronous communication"""
@cherrypy.expose
def index(self):
"""Return a basic HTML page with a ping form, a kill form, and an iframe"""
# Note: Dollar signs in string.Template are escaped by using two ($$)
html = """\
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<link rel="stylesheet" type="text/css" href="${jquery_ui_css_url}" media="screen" />
<script type="text/javascript" src="${jquery_url}"></script>
<script type="text/javascript" src="${jquery_ui_url}"></script>

<style>
input[type=button], input[type=submit], input[type=reset] {
background: #eee;
color: #222;
border: 1px outset #ccc;
padding: .1em .5em;
}
</style>
<script type="text/javascript">
function UpdateProgress(cTimeReceived)
{
document.getElementById('score').innerHTML = cTimeReceived
}
</script>

</head>
<body>
<script type="text/javascript">
$$(function(){
$$('#result').hide();
$$('#kill_ping').click(function() {
$$.ajax({
url: "/kill_proc",
cache: false,
success: function(html){
window.frames[0].stop();
$$("#result").html(html);
$$("#result").show('slow');
}
});
return false;
});
});
</script>
<script type="text/javascript">
$$(function(){
$$('#ping').click(function() {
$$('#result').hide();
});
});
</script>
<h3>CherryPy Comet Example</h3>
<form id="ping_form" target="console_iframe" method="post" action="/ping">
<input id="ping" type="submit" value="Start"></input>
</form>
<BR>
<form id="kill_form" method="post" action="/kill_proc">
<input id="kill_ping" type="submit" value="Stop"></input>
</form>
<div id="result" class="ui-state-highlight">
<span class="ui-icon ui-icon-check ui-icon-left" style="margin-right: .3em;">
</span>
</div>
<div><table><tr><td id="score" style="font-size:96px"></td><td></td></tr></table></div>
<iframe name="console_iframe" style="display:none"/>
</body>
</html>
"""
t = Template(html)
page = t.substitute(
jquery_ui_css_url=jquery_ui_css_url,
jquery_url=jquery_url,
jquery_ui_url=jquery_ui_url)
return page

@cherrypy.expose
def ping(self, **kw):
"
""Start a time loop (reporting time and stream the output"""
def run_command():
while True:
now = datetime.datetime.now()
cTime = str(now)[:22]
yield '<script>parent.UpdateProgress("
'+ cTime + '")</script>'
time.sleep(2)

return run_command()

# Enable streaming for the ping method. Without this it won't work.
ping._cp_config = {'response.stream': True}

@cherrypy.expose
def kill_proc(self, **kw):
"""Kill the process """
return "<strong>Success:</strong> The process was stopped successfully."

cherrypy.config.update({
'log.screen':True,
'tools.sessions.on': True,
'checker.on':False,
'server.socket_host':'192.168.0.135',
'tools.staticdir.root': '/home/pi/pythonprogs',
'tools.staticdir.on': True,
'tools.staticdir.dir':'static'})
cherrypy.tree.mount(Comet(), config=None)
cherrypy.engine.start()
cherrypy.engine.block()

Raspberry Pi: 1.6 million records in SQLite database.

Yesterday was a bit of a milestone for me in that my generator monitoring application had been in production for one year exactly.

During this time, the SQLite database logged 1,630,000+ temperature records, (3 records every minute). There have been a few minor hiccups, programming bugs, which I corrected, but other than that, the Raspberry Pi and Python have acted flawlessly. No mysterious hang ups, no unresponsiveness, it just plain old works.

It faithfully still sends out SMS messages and emails when needed. The webserver also works flawlessly all of the time.

I’m especially impressed with the gnuplot utility. Quirky to set up, yes, but very solid and quick to generate complex graphs.

The graph below (from today), as an example, shows high and low temperatures, by the hour, over the last year at our house. The SQLite database selected 8,750 records out of the 1.6 million, handing them over to Gnuplot which plotted this dataset, in about 3 seconds. Not bad at all.

Interesting things about the graph:

- spread between daytime highs and nighttime lows in a lot larger in summer than in winter.

- frost free period was from end of April till end of October, not bad considering the cold spring we had here in southern Ontario.

- so far December this year is shaping up to be a heck of a lot colder than last year.

tempsextended

Monday, September 30, 2013

Raspberry Pi and a solution waiting for a problem…

Well, not quite, maybe. Let me explain. In previous posts, I played around getting the Nokia 5110 LCD to properly work with the Raspberry Pi. After the usual set of problems, it fully cooperated and I incorporated it as my display in my carputer.

Now, I would like to mount the display on the dash of my car somewhere, but, although the Pi is small, it is still too big to fit in properly.

So I got to thinking, what if I place the display at some distance from the Pi? But how? I decided to go out on a limb and use RJ45 connectors and jacks.

I bought some RJ45 jacks via Ebay and some RJ45 breakout boards from Cooking Hacks. These breakout boards, at a whopping $0.50 a board, allow for easy hookups of the RJ 45 jack to the wires that lead either from the Pi’s GPIO pins or go to the input pins on the Nokia LCD.

After soldering the jacks and the wires to the breakout boards, it was time to grab an Ethernet cable (6 feet in this case), slip the ends into the jacks and turn on the Pi. At first, the results were ambiguous. Sometimes, the display showed correct results, sometimes there were errors in the text, sometimes nothing. So I decided to do some experimenting.

I noticed that the speed at which the Nokia LCD was processing data was set at 5Mhz. I reduced this to .5 Mhz. Voila, it worked.

Next, I tried a longer cable, 24 feet. No problem! I don’t have a longer cable, it would be interesting to see what would happen.

Here’s the 24 foot cable in action (the error message on the display has nothing to do with this project).

 IMG_9673

So, if you need a display a long way away from the Pi itself, may be this is the way to go…

Tuesday, September 03, 2013

Raspberry Pi and “sudo poweroff”

A few weeks ago I ordered a power switch for the Raspberry Pi from www.mausberrycircuits.com with the aim of using this on my Raspi Fuel Miser setup. (see previous post).

Illuminated LED shutdown switch

When this switch is situated between your Raspberry Pi’s power jack and your phone charger and you plug in the phone charger, nothing happens. You first need to press the button before the Pi lights up. Then, once you are through using it, you keep the button pressed for 2 seconds and the Pi powers off.

You need to hook up 2 wires from the switch to the GPIO pins, suggested are 23 and 24. If you do use other ones, then you need to modify /etc/switch.sh after you run setup .sh.

I first ran it using the switch.sh configuration, but then I got to thinking, why run it separately when it can be included in my Python script, which is running continuously anyway, dutifully computing gas mileage.

So I amended my script, created a new function has_ShutdownButtonBeenPressed and inserted a call to it at the start of the main loop in the obd_capture.py program. This way, it will check for a pressed key once every 3/4 second or so, the time it takes to complete a cycle of OBDII readings. I disabled the switch.sh and remove the entry in /etc/rc.local which started up switch.sh upon start up.

The ‘sudo poweroff’ command, by the way, cuts all power to the board, so it is completely disabled, unlike the ‘shutdown’ command, which still leaves the main red LED lit.

Now, at the end of my trip, if I have used the fuel miser, all I need to do is press the power button for 2 seconds, and the Python script will close the SQlite database gracefully before issuing the os.system(“sudo poweroff”) command.

Tuesday, August 27, 2013

Raspberry Pi MPG Fuel Economy Carputer

I’ve always had an interest in getting the best fuel economy out of cars that I drove. On June 14, 2006, I blogged about my Toyota Echo and its gas mileage. I’ve often wanted to buy a device like ScanGauge so I could see the instantaneous fuel economy I was getting.

A few weeks ago, the Raspberry.org site’s main item had to do with carputers. That, along with an article about the Quite Rubbish clock got me to thinking, how about a Raspberry Pi MPG reader?

Here’s the final result, well final, there is no case and I’m also thinking of adding various buttons to vary the output of the display. But for the moment…

The MPG computer in action

The top line lists the average of the last 3 seconds. The bottom line the average for the entire trip, followed by the duration of the trip in minutes. Since I’m in Canada, where fuel economy is expressed in litres consumed per 100 km driven, that’s what it is set up for. However, it would be fairly trivial to change the formula and output based on the user’s preference.

The code is based on Martin O’Hanlon’s obd_capture.py. However, I made significant changes in that the data is being captured into a SQLite3 database (which is actually used to compute the 3 second average). Furthermore, I included the output to the LCD in this class as well. (Yeah, I know, you shouldn’t be mixing data with user interface, but hey, I’m not getting paid for this, so I can do whatever I want.)

Every time the car starts, a new database with a unique name is created. It takes about 15 seconds for the Pi to boot and another 5 seconds for the Python script to start running. This database can be retrieved later for further analysis of the car’s performance.

The fuel economy is derived from the values given by the ELM327 (about $12 from China) OBDII reader for MAF (Mass Airflow meter) and VSS (vehicle speed). The formula for litres per 100 km is:

(3600 * MAF)/(9069.90 * VSS)

Credit for this formula should go to Bruce Lightner, who is probably one of the most knowledgeable people on this planet when it comes to OBDII (and moon rocks and more stuff)

The output goes to a Nokia 5110 LCD (Deal Extreme $5.60). In order to get a fairly large readout for driver visibility, I actually created an image in Python and then place that image on the LCD as a whole.

The OBDII connector in my car points downwards, and that might interfere with the driver feet, so I’ve ordered an extension cable from China for a few dollars which will alleviate this problem. This cable exits at right angles from the connector.

If you intend to build this device using the Nokia 5110 LCD, you’ll need to read my previous post regarding this device.

For Martin O’Hanlon excellent code, and related libraries type this at the command prompt:

sudo apt-get install python-serial
sudo apt-get install git-core
cd ~
git clone https://github.com/martinohanlon/pyobd
cd pyobd
python obd_capture.py

Here is my version of obd_capture.py:

   1: #!/usr/bin/env python
   2:  
   3: import obd_io
   4: import serial
   5: import platform
   6: import obd_sensors
   7: from datetime import datetime
   8: from PIL import Image,ImageDraw,ImageFont
   9: import ImageOps
  10: import nokiaSPI
  11: import time
  12: import os
  13: import sqlite3
  14: #from datetime import timedelta
  15:  
  16: from obd_utils import scanSerial
  17:  
  18: class OBD_Capture():
  19:     def __init__(self):
  20:         self.port = None
  21:         localtime = time.localtime(time.time())
  22:  
  23:     def connect(self):
  24:         portnames = scanSerial()
  25:         print portnames
  26:         for port in portnames:
  27:             self.port = obd_io.OBDPort(port, None, 2, 2)
  28:             if(self.port.State == 0):
  29:                 self.port.close()
  30:                 self.port = None
  31:             else:
  32:                 break
  33:  
  34:         if(self.port):
  35:             print "Connected to "+self.port.port.name
  36:  
  37:     def is_connected(self):
  38:         return self.port
  39:  
  40:     def nokiprint(self,cShortTerm,cLongTerm,cLongTermMinutes):
  41:         cShortTerm = cShortTerm[:4]
  42:         cLongTerm = cLongTerm[:4]
  43:         noki.cls()
  44:         im = Image.new('1', (84,48))
  45:         draw = ImageDraw.Draw(im)
  46:         print cShortTerm
  47:         draw.text((0,0),cShortTerm, font=font, fill=1)
  48:         draw.text((0,24),cLongTerm, font=font, fill=1)
  49:         draw.text((54,7),"3 s", font=fontsmall, fill=1)
  50:         draw.text((54,31),cLongTermMinutes + " m", font=fontsmall, fill=1)
  51:         # Copy it to the display
  52:         noki.show_image(im)
  53:         #noki.next_row()
  54:         
  55:     def ComputeFuelConsumption(self):
  56:         nCurrentTime = time.time()
  57:         
  58:         try:
  59:             nStart = nCurrentTime - 20
  60:             cLimit = " and time_read > " + str(nStart) + " order by time_read desc limit 6"
  61:             cursor.execute('''SELECT maf,speed from SensorReadings where speed > "0" and maf > "0" and rpm != "NODATA" ''' + cLimit)
  62:             data = cursor.fetchall()
  63:         except sqlite3.OperationalError,msg:
  64:             return msg
  65:         #print len(data)
  66:         if (len(data) > 0):
  67:             nFuelConsumption = 0
  68:             for x in range(0,len(data)):
  69:                 nFuelConsumption += (3600 * float(data[x][0]))/(9069.90 * float(data[x][1]))
  70:                 
  71:             nAvgFuelConsumption = nFuelConsumption/len(data)
  72:             print nAvgFuelConsumption
  73:             print type(nAvgFuelConsumption)
  74:             #print data[x][0],data[x][1],data[x][2]
  75:             return "{:5.2f}".format(nAvgFuelConsumption).lstrip()
  76:         else:
  77:             return "No data"
  78:  
  79:         #print nAvgFuelConsum
  80:         
  81:     def is_number(self,DataToTest):
  82:         try:
  83:             float(DataToTest)
  84:             return True
  85:         except ValueError:
  86:             return False    
  87:                 
  88:                 
  89:  
  90:                 
  91:     def capture_data(self):
  92:         #Creating new database
  93:         for kounter in range(10000):
  94:             cKounter = "{0:05d}".format(kounter)
  95:             cNewDatabase = "obdii" + cKounter + ".db"
  96:             #print cNewDatabase
  97:             if not (os.path.exists(cNewDatabase)):
  98:                 #print "New database name: " + cNewDatabase
  99:                 break
 100:  
 101:         global conn
 102:         global cursor
 103:         conn = sqlite3.connect(cNewDatabase)
 104:         cursor = conn.cursor()
 105:             
 106:         #Find supported sensors - by getting PIDs from OBD
 107:         # its a string of binary 01010101010101 
 108:         # 1 means the sensor is supported
 109:         self.supp = self.port.sensor(0)[1]
 110:         self.supportedSensorList = []
 111:         self.unsupportedSensorList = []
 112:         
 113:         
 114:         # loop through PIDs binary
 115:         for i in range(0, len(self.supp)):
 116:             if self.supp[i] == "1":
 117:                 # store index of sensor and sensor object
 118:                 self.supportedSensorList.append([i+1, obd_sensors.SENSORS[i+1]])
 119:             else:
 120:                 self.unsupportedSensorList.append([i+1, obd_sensors.SENSORS[i+1]])
 121:         
 122:         sqlCreateTable = "CREATE TABLE SensorReadings (time_read real, "
 123:         sqlInsertTemplate = "INSERT INTO SensorReadings(time_read, "
 124:         
 125:         for supportedSensor in self.supportedSensorList:
 126:             #print "supported sensor index = " + str(supportedSensor[0]) + " " + str(supportedSensor[1].shortname)
 127:             sqlCreateTable += str(supportedSensor[1].shortname)  + " text,"    
 128:             sqlInsertTemplate += str(supportedSensor[1].shortname)  + ","
 129:             
 130:         sqlCreateTable = sqlCreateTable[:sqlCreateTable.rfind(",")] + ")"
 131:         #print sqlCreateTable
 132:         try:
 133:             cursor.execute(sqlCreateTable)
 134:             conn.commit()
 135:             cursor.execute('''CREATE INDEX time_read_index on SensorReadings(time_read)''')
 136:             cMessage = "Database " + cNewDatabase + " created..."
 137:         except sqlite3.OperationalError,msg:
 138:             cMessage = msg
 139:         noki.cls()
 140:         noki.text(cMessage,wrap=True)
 141:         
 142:         sqlInsertTemplate = sqlInsertTemplate[:sqlInsertTemplate.rfind(",")] + ") VALUES ("
 143:         #print sqlInsertTemplate
 144:         
 145:         time.sleep(3)
 146:  
 147:         if(self.port is None):
 148:             return None
 149:         
 150:         #Loop until Ctrl C is pressed        
 151:         try:
 152:             nRunningTotalFuelConsumption = 0
 153:             nStartTime = time.time()
 154:             x = 0
 155:             while True:
 156:                 current_time = time.time()
 157:                 #current_time = str(localtime.hour)+":"+str(localtime.minute)+":"+str(localtime.second)+"."+str(localtime.microsecond)
 158:                 #log_string = current_time + "\n"
 159:                 sqlInsert = sqlInsertTemplate + '"' + str(current_time) + '",'
 160:                 results = {}
 161:                 for supportedSensor in self.supportedSensorList:
 162:                     sensorIndex = supportedSensor[0]
 163:                     #print sensorIndex
 164:                     (name, value, unit) = self.port.sensor(sensorIndex)
 165:                     #log_string += name + " = " + str(value) + " " + str(unit) + "\n"
 166:                     #print value,type(value)
 167:                     sqlInsert += '"' + str(value) + '",'    
 168:                 
 169:                 
 170:                 sqlInsert = sqlInsert[:sqlInsert.rfind(",")] + ")"
 171:                 #print sqlInsert
 172:  
 173:                 try:
 174:                     cursor.execute(sqlInsert)
 175:                     conn.commit()
 176:                 except sqlite3.OperationalError,msg:
 177:                     noki.cls()
 178:                     noki.text(msg,wrap=True)    
 179:                     continue
 180:                     
 181:                 cFuelConsumption = self.ComputeFuelConsumption()
 182:                 if (cFuelConsumption != "No data"):
 183:                     x += 1
 184:                 if (x > 0):
 185:                     if (self.is_number):
 186:                         nRunningTotalFuelConsumption += float(cFuelConsumption)
 187:                     nTripAverage = nRunningTotalFuelConsumption/x
 188:                     cTripAverage = "{:5.2f}".format(nTripAverage).lstrip()
 189:                 else:
 190:                     cTripAverage = "Nodata"
 191:                 
 192:                 cDurationInMinutes = "{:3.0f}".format((current_time - nStartTime)/60).lstrip()
 193:                 self.nokiprint(cFuelConsumption,cTripAverage,cDurationInMinutes)
 194:                     
 195:                 #print log_string,
 196:                 #time.sleep(0.5)
 197:                 
 198:  
 199:         except KeyboardInterrupt:
 200:             self.port.close()
 201:             print("stopped")
 202:             
 203: if __name__ == "__main__":
 204:     font = ImageFont.truetype("/usr/share/fonts/truetype/freefont/FreeSansBold.ttf",26)
 205:     fontsmall = ImageFont.truetype("/usr/share/fonts/truetype/freefont/FreeSansBold.ttf", 16)
 206:     # New b-w image
 207:     im = Image.new('1', (84,48))
 208:     noki = nokiaSPI.NokiaSPI(brightness=268)              # create display device
 209:     noki.cls()
 210:     noki.text("Initializing..",wrap=True)
 211:  
 212:  
 213:     o = OBD_Capture()
 214:     o.connect()
 215:     time.sleep(3)
 216:     if not o.is_connected():
 217:         print "Not connected"
 218:         noki.cls()
 219:         noki.text("Error: Not connected to OBDII...",wrap=True)
 220:         time.sleep(10)
 221:         noki.set_brightness(0)
 222:         noki.cls()
 223:         exit()
 224:     else:
 225:         o.capture_data()

In order for the Python script to automatically start when the Pi boots up, I added this line to /etc/rc.local :


(cd /home/pi/pythonprogs;python obd_capture.py)&


That line should only be added when you are relatively confident it will all work. First, for testing, you will need to run a network cable to your car (or tap in wirelessly, if you are within range). This is what my setup looked like, during early testing:


IMG_1136


The blue cable was networking, the yellow powered the Pi. Later on, I removed the power cable and ran off the car’s battery. It is quite tricky to have to program for something that has to run headless and detached from everything. So when making changes to your code, check, check and double check again.


I’ve also ordered from MausBerryCircuits.com an illuminated LED shutdown switch, which will gracefully shut down the Pi before power is removed, reducing the chance of corrupting the SD card.


That’s it!


Update 2013-08-31:


I’ve optimized the fuel consumption function and made SQLite do the average calculation, alleviating the need do to the calculation in a Python ‘for’ loop. Also, made the SQLite connection and cursor class attributes rather than global variables, which are generally considered undesirable.