Monday, December 16, 2013

Raspberry Pi, CherryPy and asynchronous communication

I have a project I am working on where graphical output is required to be sent to the screen. Using Python, where by default the output only goes to the console, one then immediately thinks of Tkinter, which is a Python module that provides a graphical interface to Python.

However, that would mean learning yet another ‘language’ as such. So I decided to pursue a different avenue. I am fairly familiar with HTML and JavaScript, even though the latter is extremely quirky and non-intuitive. Therefore, I let my gaze wander to a browser based solution.

Now what I am looking for goes beyond the request-response model usually found in browser based interfaces: the user clicks on a button (or link), the browser sends the information to the server, the server responds by sending the requested information and the connection closes. A classic case of synchronous communication.

What I am looking for is a scenario where the browser’s window is automatically updated with new information as it becomes available and no if any user action is required. Asynchronous communication.

So I started researching. What I really wanted was a short, sweet, concise code example of how to do asynchronous communication with Python. There were lots of examples, code pieces, snippets for Bottle, Flask, Tornado (all Python based webservers), but nothing that was simple enough to just run.

I even spent some time installing node.js, which is supposed to be the next best thing since HTML. Once installed though, I couldn’t find any examples that were straight forward enough or worked ‘out-of-the-box’.

Finally, I hit on CherryPy, yet another Python based web server module, specifically this page in their wiki:http://tools.cherrypy.org/wiki/Comet. On this page, author Dan McDougall discusses CherryPy approach to ‘Comet’. Comet (a cleanser). Ajax (a cleanser), however in IT lingo, Ajax can provide feedback to a browser without the entire browser window having to be refreshed. I decided to download CherryPy, run the application in Python, open a browser and voila: it worked. Success, end of a long searching marathon.

CherryPy does multithreading on its own. At its peak I had three web browsers open on various machines and the CPU Usage meter barely moved. Another neat feature that I discovered is that when I saved a new version of the application in Geany, with the old application still running, it will shut the old application down and start the new one all on its own.

The output of the application I downloaded spits out the results of pings to an ip address or web site you provide. The results just keep on streaming by on the screen. Great example, but not what I needed: what I wanted was a number of string that simply replaced the previous string, i.e. automatic updating of a value. More searching. This time I stumbled on a neat solution provided by Encosia (Dave Ward), which I adapted to work with my application.

The original CherryPy application created an iframe at the bottom of the HTML (just before the </body> tag). The asynchronous info being sent from the server gets displayed there. An iframe is really a webpage within your web page, usually used for ads.

Dave Ward’s solution is to hide the iframe (style=”display:none”), then have the web server send a function call embedded in the asynchronous communication along with the latest value of whatever we are trying to display. The function called is on our main web page (‘the parent’) and so can update anything on that page. The iframe is still there and working, but invisible to us.

In my case, I have a table and the content of the function simply reads:

document.getElementById(‘11’).innerHTML = cTimeReceived

where ‘11’ is the id of the first table data cell and cTimeReceived the new value received from the server.

I tested this on the Raspberry Pi’s various assortment of web browsers. It worked properly on Midori, Chromium, Luakit, Iceweasel but not on Dillo or Netsurfer. I let it run for hours without a problem. I ran it from a laptop elsewhere and  intentionally interrupted the network connection, and, once the connection was re-established, the page kept right on going.

On the Pi, Luakit by far required the least amount of resources (around 30% in CPU Usage Monitor), while Chromium maxed out at 100%, the others somewhere in between.

Obligatory screen shot:

cherrypyscreenshot

Below the application. I ran this on the Raspberry Pi Model B, 512 Mb, with Raspbian updated to 2013-12-15. For it to run, you need to have Python (my version 2.7). To install CherryPy:

pip install cherrypy

Then copy the application below, change my IP address (towards the bottom) to your own, change my ‘pythonprogs’ directory name to you directory name and run. Open up a browser, point to your IP address, followed by :8080 and press enter. Once the page appears click’ Start’. That’s it!

Note that the HTML for the starting page is embedded in the application, hence 1 file is all you need to run this example.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# string.Template requires Python 2.4+
from string import Template
import cherrypy
import time,datetime

__author__ = 'Dan McDougall <YouKnowWho@YouKnowWhat.com>'

# Trying to cut down on long lines...
jquery_url = 'http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js'
jquery_ui_url = 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/jquery-ui.min.js'
jquery_ui_css_url = \
'http://ajax.googleapis.com/ajax/libs/jqueryui/1.7.2/themes/black-tie/jquery-ui.css'

class Comet(object):
"""An example of using CherryPy for Comet-style asynchronous communication"""
@cherrypy.expose
def index(self):
"""Return a basic HTML page with a ping form, a kill form, and an iframe"""
# Note: Dollar signs in string.Template are escaped by using two ($$)
html = """\
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<link rel="stylesheet" type="text/css" href="${jquery_ui_css_url}" media="screen" />
<script type="text/javascript" src="${jquery_url}"></script>
<script type="text/javascript" src="${jquery_ui_url}"></script>

<style>
input[type=button], input[type=submit], input[type=reset] {
background: #eee;
color: #222;
border: 1px outset #ccc;
padding: .1em .5em;
}
</style>
<script type="text/javascript">
function UpdateProgress(cTimeReceived)
{
document.getElementById('score').innerHTML = cTimeReceived
}
</script>

</head>
<body>
<script type="text/javascript">
$$(function(){
$$('#result').hide();
$$('#kill_ping').click(function() {
$$.ajax({
url: "/kill_proc",
cache: false,
success: function(html){
window.frames[0].stop();
$$("#result").html(html);
$$("#result").show('slow');
}
});
return false;
});
});
</script>
<script type="text/javascript">
$$(function(){
$$('#ping').click(function() {
$$('#result').hide();
});
});
</script>
<h3>CherryPy Comet Example</h3>
<form id="ping_form" target="console_iframe" method="post" action="/ping">
<input id="ping" type="submit" value="Start"></input>
</form>
<BR>
<form id="kill_form" method="post" action="/kill_proc">
<input id="kill_ping" type="submit" value="Stop"></input>
</form>
<div id="result" class="ui-state-highlight">
<span class="ui-icon ui-icon-check ui-icon-left" style="margin-right: .3em;">
</span>
</div>
<div><table><tr><td id="score" style="font-size:96px"></td><td></td></tr></table></div>
<iframe name="console_iframe" style="display:none"/>
</body>
</html>
"""
t = Template(html)
page = t.substitute(
jquery_ui_css_url=jquery_ui_css_url,
jquery_url=jquery_url,
jquery_ui_url=jquery_ui_url)
return page

@cherrypy.expose
def ping(self, **kw):
"
""Start a time loop (reporting time and stream the output"""
def run_command():
while True:
now = datetime.datetime.now()
cTime = str(now)[:22]
yield '<script>parent.UpdateProgress("
'+ cTime + '")</script>'
time.sleep(2)

return run_command()

# Enable streaming for the ping method. Without this it won't work.
ping._cp_config = {'response.stream': True}

@cherrypy.expose
def kill_proc(self, **kw):
"""Kill the process """
return "<strong>Success:</strong> The process was stopped successfully."

cherrypy.config.update({
'log.screen':True,
'tools.sessions.on': True,
'checker.on':False,
'server.socket_host':'192.168.0.135',
'tools.staticdir.root': '/home/pi/pythonprogs',
'tools.staticdir.on': True,
'tools.staticdir.dir':'static'})
cherrypy.tree.mount(Comet(), config=None)
cherrypy.engine.start()
cherrypy.engine.block()

Raspberry Pi: 1.6 million records in SQLite database.

Yesterday was a bit of a milestone for me in that my generator monitoring application had been in production for one year exactly.

During this time, the SQLite database logged 1,630,000+ temperature records, (3 records every minute). There have been a few minor hiccups, programming bugs, which I corrected, but other than that, the Raspberry Pi and Python have acted flawlessly. No mysterious hang ups, no unresponsiveness, it just plain old works.

It faithfully still sends out SMS messages and emails when needed. The webserver also works flawlessly all of the time.

I’m especially impressed with the gnuplot utility. Quirky to set up, yes, but very solid and quick to generate complex graphs.

The graph below (from today), as an example, shows high and low temperatures, by the hour, over the last year at our house. The SQLite database selected 8,750 records out of the 1.6 million, handing them over to Gnuplot which plotted this dataset, in about 3 seconds. Not bad at all.

Interesting things about the graph:

- spread between daytime highs and nighttime lows in a lot larger in summer than in winter.

- frost free period was from end of April till end of October, not bad considering the cold spring we had here in southern Ontario.

- so far December this year is shaping up to be a heck of a lot colder than last year.

tempsextended