Tuesday, March 12, 2013

Raspberry Pi and monitoring sump pump water level

In previous posts, I have detailed how I keep track of the status of my stand by generator. One of the main tasks of the stand by generator is to ensure that there is a steady supply of power for the sump pump. So, wouldn’t it be nice if we could monitor the level of the sump, preferably remotely, so we can take action if it is too high?

Raspberry Pi to the rescue. In the course of investigating distance sensors, I came across http://www.raspberrypi-spy.co.uk/2012/12/ultrasonic-distance-measurement-using-python-part-1/. It explains the use of the ultrasonic distance sensor HY-SRF05, which can read distances between 5cm and 2m. Perfect for my purpose. Besides, they only cost around $3.00 each!

I took the very simple voltage divider circuit mentioned on that webpage and created a little circuit board for it. Here’s it is, fresh out of the etcher, with the ink of hand drawn traces still covering the only copper left on the board. To the top you can see the 4 points of the board to which the HY-SRF05 will be attached by solder points.

circuitboardoutofbath

Once I sanded the ink off, drilled the holes and mounted the resistors, connecting wires and sensor, I did a test. It worked! Woohoo! Next, accuracy testing was conducted, proving that most of the time the reading is within 5%. Good enough for my purpose.

To attach the sensor in a convenient spot, I used a hose clamp and short piece of wire strapping. Through experimenting I found out that in order to get a fairly accurate reading, you need to get as much clearance on the sides of the sensor as possible. The wire strapping allows for this, as it is bendable and strong enough to support the lightweight sensor. The sensor and circuit board are kept in place with a colourful binder clip, for easy removal in case the pump needs to be serviced.

sensoronpump

The Raspberry Pi itself is mounted some distance away (+/- 50 cm) on the sump pump closet wall, using a case I obtained from Allied Electronics. A couple of screws inserted in the drywall keep the Pi high and dry.

 

pionwall

So much for the hardware. Now, I should point out that this Pi is the second one I own, the first one being used to monitor generator temperature. The Pi uses a concept called ‘bit-banging’ to get the readings from the sensor. If the CPU is busy servicing a lot of processes, that could read to erroneous readings, since the CPU ‘time slices’ and is too busy to pay attention to our sensor when needed. So, for now, this is all the Pi does. I adapted the Python program from Raspberry Pi Spy to every five minutes take 10 readings and store these, along with the current time) in a simple text file. I also installed an FTP service on this Pi (‘vsfptd’) which will be used by my primary Pi to obtain the data as needed.

Here’s the amended program:

 

   1: #!/usr/bin/python
   2: #+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   3: #|R|a|s|p|b|e|r|r|y|P|i|-|S|p|y|.|c|o|.|u|k|
   4: #+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   5: #
   6: # ultrasonic_1.py
   7: # Measure distance using an ultrasonic module
   8: #
   9: # Author : Matt Hawkins
  10: # Date   : 09/01/2013
  11: # Modified: Keith Hekker
  12: # Date    : 2013/03/11
  13: # Import required Python libraries
  14: import time
  15: import RPi.GPIO as GPIO
  16: import os
  17: import datetime
  18:  
  19:  
  20: # Use BCM GPIO references
  21: # instead of physical pin numbers
  22: GPIO.setmode(GPIO.BCM)
  23:  
  24: # Define GPIO to use on Pi
  25: GPIO_TRIGGER = 23
  26: GPIO_ECHO    = 24
  27:  
  28: print "Ultrasonic Measurement"
  29:  
  30: # Set pins as output and input
  31: GPIO.setup(GPIO_TRIGGER,GPIO.OUT)  # Trigger
  32: GPIO.setup(GPIO_ECHO,GPIO.IN)      # Echo
  33:  
  34: # Set trigger to False (Low)
  35: GPIO.output(GPIO_TRIGGER, False)
  36:  
  37: while 1:
  38:     # Allow module to settle
  39:     dTimeRecorded = datetime.datetime.now()
  40:     print dTimeRecorded
  41:  
  42:     tdistance = 0
  43:     #Create a list to store distances
  44:     lDistances = []
  45:     for x in range(0,10):
  46:         time.sleep(1.0)
  47:         # Send 10us pulse to trigger
  48:         GPIO.output(GPIO_TRIGGER, True)
  49:         time.sleep(0.00001)
  50:         GPIO.output(GPIO_TRIGGER, False)
  51:         start = time.time()
  52:  
  53:         while GPIO.input(GPIO_ECHO)==0:
  54:             start = time.time()
  55:  
  56:         while GPIO.input(GPIO_ECHO)==1:
  57:             stop = time.time()
  58:  
  59:         # Calculate pulse length
  60:         elapsed = stop-start
  61:  
  62:         # Distance pulse travelled in that time is time
  63:         # multiplied by the speed of sound (cm/s)
  64:         distance = elapsed * 34300
  65:  
  66:         # That was the distance there and back so halve the value
  67:         distance = distance / 2
  68:         print "Distance : %.1f" % distance
  69:         if x > 0 and distance > 0:
  70:             tdistance = tdistance + distance
  71:             lDistances.append(distance)
  72:     
  73:     fdistance = tdistance/9
  74:     print "Final Distance : %.1f" % fdistance
  75:  
  76:     lDistances.append(dTimeRecorded)
  77:     with open('distances','w') as file:
  78:         for item in lDistances:
  79:             file.write("{}\n".format(item))
  80:  
  81:     time.sleep(300)
  82:  
  83: # Reset GPIO settings
  84: GPIO.cleanup()



It does need to use supervisory permission to run, so once you switched to the directory where your Pi program is stored, you need to type in on the command line sudo python ultrasonic_1.py.


So much for the data generating Pi. Now we switch our attention to the data consuming Pi. It already runs as a webserver using bottle.py straight from within Python. All I did was add a function that generates an FTP call(using the Python library FTPlib), opens the file downloaded, reads the 9 values, adds them up and divides the total by nine to get the average value. It also retrieves the time when these values were recorded. Lastly, it generates a line of text along the lines of ‘Sump pump water level: 8.08 cm. Recorded at: 2013-03-12 19:18:47’, which can be plugged into the HTML template being used. Speaking of this template, in the appropriate place on the front page, I added a variable that bottle.py would populate at run time. That’s it!


Here’s the function I added into my bottle web server program



   1: def getSumpPumpDepth():
   2:     sftp = ftplib.FTP('192.168.0.151','fred','fredspassword')
   3:     sftp.cwd("pythonprogs")
   4:     gFile = open("distancemeasurement","wb")
   5:     sftp.retrbinary("RETR distances",gFile.write)
   6:     sftp.quit()
   7:     gFile.close()
   8:     listdata = [line.strip() for line in open("distancemeasurement","r")]
   9:     nTotalDistance = 0
  10:     nListLength = len(listdata)
  11:     for x in range(0,nListLength):
  12:         if x < nListLength -1:
  13:             nTotalDistance = nTotalDistance + float(listdata[x])
  14:         else:
  15:             cTimeRecorded = listdata[x]
  16:             
  17:     #print nTotalDistance
  18:     nAvgDistance = nTotalDistance/(nListLength -1)
  19:     #print nAvgDistance
  20:     #print cTimeRecorded
  21:     return "Sump pump water level: " + str(nAvgDistance)[:4] + " cm. Recorded at: " + cTimeRecorded[:19]

And here is the line it produces on the web page:


sumppumpscreenprint

Monday, March 04, 2013

Raspberry Pi–Using FTP between 2 Pi’s programmatically

I have a need to transfer sensor data periodically from on Pi to another. For no good reason, I decided to use FTP for this. Actually, as it turned out, it was a good decision, since the process works well and the overhead involved, i.e. installation and programming as well as CPU time, is minimal.

Both Pi’s are running Raspbian Wheezy, latest version.

This is what I did:

On the Pi that has the data that I want, I installed vsftp thusly:

sudo apt-get install vsftpd

Once the installation finished I adjusted the configuring parameters:

sudo nano /etc/vsftpd.conf

Once this file opened up, I removed the # tags in front of local_enable=YES and write_enable=YES, changed anymous_enable to NO, then closed and saved the file when I was prompted.

Next, I restarted the service like this:

sudo service vsftpd restart

On the Pi that needs the data I do the retrieving via FTP in a python program using a standard Python library, surprising called ftplib.

Here’s the program:

   1: import ftplib
   2: sftp = ftplib.FTP('192.168.0.151','pi','mypassword')
   3: sftp.cwd("pythonprogs")
   4: gFile = open("distancemeasurement","wb")
   5: sftp.retrbinary("RETR distancemeasurement",gFile.write)
   6: sftp.quit()
Line 3 (cwd) is Change Working Directory)

That’s it! Pretty straightforward, I think.

Sunday, February 10, 2013

Raspberry Pi, gnuplot and jquery

In my last post, I wrapped up the standby generator project. However, I felt that logging temperatures to a database and not really doing much with them afterwards was not really logical. Either you don’t use them and clean up the database, or, you do something funky with them. In my case, I just cannot bear to get rid of organized, well defined data, so I decided to employ the Pi to display the data in graphs.

This is the end result:

tempgraphweek

Click on the graph to see it in full detail and observer some of the wild temperature swings we see around here.

To create the graphs, I use gnuplot. This cross platform graphing application has been around for many years. In order to produce graphs, you need to, as a minimum, 2 have created 2 files: a config file and a data file.

To install gnuplot on a Raspberry Pi:

sudo apt-get install gnuplot-x11

The config file (gnuplotconfigweek.txt):

   1: set terminal gif small size 600, 500 transparent
   2: set output "images/tempsweek.gif"
   3: set time
   4:  
   5: set xtics rotate
   6: set xdata time
   7: set timefmt "%Y-%m-%d %H:%M:%S"
   8: set format x "%Y-%m-%d"
   9: set y2tics
  10: set grid
  11: set title "Ambient Temperature Trend Per Week"
  12: set ylabel "Degrees Celsius"
  13: set xlabel "\n1 hour interval"
  14: plot 'graphdataweek.txt' using 1:3 title "Ambient" with lines
  15: set key below
  16: set output

The data file (graphdataweek.txt):



   1: 2013-02-03 01:00:00 -15.437
   2: 2013-02-03 02:00:00 -16.25
   3: 2013-02-03 03:00:00 -15.875
   4: 2013-02-03 04:00:00 -15.625
   5: 2013-02-03 05:00:00 -15.25
   6: 2013-02-03 06:00:00 -14.312
   7: 2013-02-03 07:00:00 -13.875
   8: 2013-02-03 08:00:00 -12.812
   9: 2013-02-03 09:00:00 -10.875
  10: 2013-02-03 10:00:00 -7.375
  11: 2013-02-03 11:00:00 -5.562
  12: 2013-02-03 12:00:00 -4.875
  13: 2013-02-03 13:00:00 -2.562
  14: 2013-02-03 14:00:00 -2.312
  15: 2013-02-03 15:00:00 -3.562
  16: 2013-02-03 16:00:00 -5.187
  17: 2013-02-03 17:00:00 -6.75
  18: 2013-02-03 18:00:00 -8.0
  19: 2013-02-03 19:00:00 -8.5
  20: 2013-02-03 20:00:00 -8.937
  21: 2013-02-03 21:00:00 -9.25
  22: 2013-02-03 22:00:00 -8.5
  23: 2013-02-03 23:00:00 -9.0
  24: 2013-02-04 00:00:00 -8.625
  25: 2013-02-04 01:00:00 -8.687
  26: 2013-02-04 02:00:00 -9.5
  27: 2013-02-04 03:00:00 -10.562
  28: 2013-02-04 04:00:00 -11.187
  29: 2013-02-04 05:00:00 -11.5
  30: 2013-02-04 06:00:00 -11.562
  31: 2013-02-04 07:00:00 -11.687
  32: 2013-02-04 08:00:00 -11.25
  33: 2013-02-04 09:00:00 -9.437
  34: 2013-02-04 10:00:00 -3.125
  35: 2013-02-04 11:00:00 -5.375
  36: 2013-02-04 12:00:00 -3.0
  37: 2013-02-04 13:00:00 -2.625
  38: 2013-02-04 14:00:00 -2.937
  39: 2013-02-04 15:00:00 -3.062
  40: 2013-02-04 16:00:00 -4.062
  41: 2013-02-04 17:00:00 -5.062
  42: 2013-02-04 18:00:00 -5.937
  43: 2013-02-04 19:00:00 -6.25
  44: 2013-02-04 20:00:00 -6.562
  45: 2013-02-04 21:00:00 -6.937
  46: 2013-02-04 22:00:00 -7.25
  47: 2013-02-04 23:00:00 -8.125
  48: 2013-02-05 00:00:00 -10.0
  49: 2013-02-05 01:00:00 -11.812
  50: 2013-02-05 02:00:00 -12.562
  51: 2013-02-05 03:00:00 -12.875
  52: 2013-02-05 04:00:00 -13.375
  53: 2013-02-05 05:00:00 -13.5
  54: 2013-02-05 06:00:00 -14.25
  55: 2013-02-05 07:00:00 -15.062
  56: 2013-02-05 08:00:00 -15.312
  57: 2013-02-05 09:00:00 -14.125
  58: 2013-02-05 10:00:00 -9.062
  59: 2013-02-05 11:00:00 -4.437
  60: 2013-02-05 12:00:00 -2.062
  61: 2013-02-05 13:00:00 -1.25
  62: 2013-02-05 14:00:00 -0.812
  63: 2013-02-05 15:00:00 -1.875
  64: 2013-02-05 16:00:00 -3.312
  65: 2013-02-05 17:00:00 -4.562
  66: 2013-02-05 18:00:00 -5.062
  67: 2013-02-05 19:00:00 -5.187
  68: 2013-02-05 20:00:00 -5.125
  69: 2013-02-05 21:00:00 -5.25
  70: 2013-02-05 22:00:00 -4.75
  71: 2013-02-05 23:00:00 -4.25
  72: 2013-02-06 00:00:00 -3.875
  73: 2013-02-06 01:00:00 -3.5
  74: 2013-02-06 02:00:00 -3.562
  75: 2013-02-06 03:00:00 -3.5
  76: 2013-02-06 04:00:00 -4.25
  77: 2013-02-06 05:00:00 -4.625
  78: 2013-02-06 06:00:00 -5.5
  79: 2013-02-06 07:00:00 -6.562
  80: 2013-02-06 08:00:00 -7.437
  81: 2013-02-06 09:00:00 -6.812
  82: 2013-02-06 10:00:00 -5.125
  83: 2013-02-06 11:00:00 -2.062
  84: 2013-02-06 12:00:00 0.187
  85: 2013-02-06 13:00:00 -0.25
  86: 2013-02-06 14:00:00 3.187
  87: 2013-02-06 15:00:00 0.812
  88: 2013-02-06 16:00:00 -0.687
  89: 2013-02-06 17:00:00 -3.437
  90: 2013-02-06 18:00:00 -4.687
  91: 2013-02-06 19:00:00 -5.812
  92: 2013-02-06 20:00:00 -6.75
  93: 2013-02-06 21:00:00 -7.562
  94: 2013-02-06 22:00:00 -8.625
  95: 2013-02-06 23:00:00 -9.625
  96: 2013-02-07 00:00:00 -10.187
  97: 2013-02-07 01:00:00 -10.687
  98: 2013-02-07 02:00:00 -10.0
  99: 2013-02-07 03:00:00 -8.375
 100: 2013-02-07 04:00:00 -7.625
 101: 2013-02-07 05:00:00 -7.062
 102: 2013-02-07 06:00:00 -6.75
 103: 2013-02-07 07:00:00 -6.187
 104: 2013-02-07 08:00:00 -5.5
 105: 2013-02-07 09:00:00 -3.75
 106: 2013-02-07 10:00:00 -1.875
 107: 2013-02-07 11:00:00 0.562
 108: 2013-02-07 12:00:00 0.937
 109: 2013-02-07 13:00:00 1.437
 110: 2013-02-07 14:00:00 0.937
 111: 2013-02-07 15:00:00 0.625
 112: 2013-02-07 16:00:00 -0.25
 113: 2013-02-07 17:00:00 -1.75
 114: 2013-02-07 18:00:00 -2.562
 115: 2013-02-07 19:00:00 -2.812
 116: 2013-02-07 20:00:00 -3.0
 117: 2013-02-07 21:00:00 -3.187
 118: 2013-02-07 22:00:00 -3.625
 119: 2013-02-07 23:00:00 -3.5
 120: 2013-02-08 00:00:00 -3.25
 121: 2013-02-08 01:00:00 -3.062
 122: 2013-02-08 02:00:00 -2.75
 123: 2013-02-08 03:00:00 -2.5
 124: 2013-02-08 04:00:00 -2.437
 125: 2013-02-08 05:00:00 -2.625
 126: 2013-02-08 06:00:00 -2.437
 127: 2013-02-08 07:00:00 -2.187
 128: 2013-02-08 08:00:00 -2.125
 129: 2013-02-08 09:00:00 -1.625
 130: 2013-02-08 10:00:00 -1.062
 131: 2013-02-08 11:00:00 -3.25
 132: 2013-02-08 12:00:00 -4.562
 133: 2013-02-08 13:00:00 -3.687
 134: 2013-02-08 14:00:00 -4.187
 135: 2013-02-08 15:00:00 -4.75
 136: 2013-02-08 16:00:00 -4.187
 137: 2013-02-08 17:00:00 -6.062
 138: 2013-02-08 18:00:00 -6.75
 139: 2013-02-08 19:00:00 -6.937
 140: 2013-02-08 20:00:00 -6.937
 141: 2013-02-08 21:00:00 -7.437
 142: 2013-02-08 22:00:00 -8.687
 143: 2013-02-08 23:00:00 -8.5
 144: 2013-02-09 00:00:00 -8.625
 145: 2013-02-09 01:00:00 -9.937
 146: 2013-02-09 02:00:00 -11.0
 147: 2013-02-09 03:00:00 -10.875
 148: 2013-02-09 04:00:00 -10.937
 149: 2013-02-09 05:00:00 -11.875
 150: 2013-02-09 06:00:00 -13.875
 151: 2013-02-09 07:00:00 -13.75
 152: 2013-02-09 08:00:00 -15.0
 153: 2013-02-09 09:00:00 -13.625
 154: 2013-02-09 10:00:00 -9.5
 155: 2013-02-09 11:00:00 -2.75
 156: 2013-02-09 12:00:00 -2.937
 157: 2013-02-09 13:00:00 -0.375
 158: 2013-02-09 14:00:00 2.312
 159: 2013-02-09 15:00:00 -1.5
 160: 2013-02-09 16:00:00 -3.0
 161: 2013-02-09 17:00:00 -4.187
 162: 2013-02-09 18:00:00 -5.625
 163: 2013-02-09 19:00:00 -7.625
 164: 2013-02-09 20:00:00 -9.25
 165: 2013-02-09 21:00:00 -10.687
 166: 2013-02-09 22:00:00 -11.812
 167: 2013-02-09 23:00:00 -13.125
 168: 2013-02-10 00:00:00 -14.187

This is all you need to produce good looking graphs. However, having a graph available and stored on your machine is one thing. Now it needs to be available on the web.


To do that I decided to add some date input fields on the ‘Manage’ tab of the monitor’s web page. Three fields and 2 buttons to be exact. The first date field allows you to pick a date and generate a week of daily temperatures starting at that date. The second set of 2 allows for a starting and ending range of dates and will generate a graph of high and low temperatures for all dates in between.


This is where jQuery comes in handy. According to Wikipedia, jQuery is a multi-browser JavaScript library designed to simplify the client-side scripting of HTML. In other words, it allows a web site developer to do fairly fancy operations fairly quickly by using a library of functions written in JavaScript that have been previously debugged. If you were foolish enough, you could write your own JavaScript to do all this, but it would be a long and tedious process. And, just when you think it all works properly, it all blows up in your face. Been there, done that.


In order to get properly formatted dates each and every time, I used the datepicker from Keith Wood, since it seemed to be sufficiently small, with lots of options for limiting input ranges, starting dates etc. To use, simply add these 2s line in the <HEAD> section of your HTML:


<link type="text/css" href="jquery.datepick.css" rel="stylesheet" />


<script type="text/javascript" src="jquery.datepick.js"></script>


Of course, you should have jquery itself also installed on this page, so the whole thing should look like this:



   1: <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.6.4/jquery.min.js"></script>
   2:     <script type="text/javascript" src="jquery.address-1.4.min.js"></script>
   3:     <script type="text/javascript" src="jquery.datepick.js"></script>
   4:     <script type="text/javascript" src="zebra_dialog.js"></script>


In my case, the jquery.address line is for the tabbed page, and the zebra_dialog is for displaying the graph. More about that later.


So, once you have the reference for the datepicker included in the HTML HEAD section, you can include the code to allow the date popups to be generated anytime a user clicks on a date field.


Still within the HEAD section, I included these lines


   1: <script type="text/javascript">
   2: $(function () {
   3:         $('#popupDatepicker').datepick({ dateFormat: $.datepick.ATOM, minDate: '2012-12-15', maxDate: -7 });
   4:         $('#1popupDatepicker').datepick({ dateFormat: $.datepick.ATOM, minDate: '2012-12-15', maxDate: -7 });
   5:         $('#2popupDatepicker').datepick({ dateFormat: $.datepick.ATOM, maxDate: 0 });
   6:     });
   7: </script>


As you can see, I have references to 3 date fields. I also specify that I would like these in the proper ISO format (i.e YYYY-MM-DD) and any other limits, starting or ending, that I would like to use.


In the actual HTML, the BODY section of it, this is how the date fields are coded:



   1: <form method="POST" action="/showweek">    
   2:                     <p>Show temperature trend for week starting: <input name="week" type="text" id="popupDatepicker" />&nbsp;&nbsp;<input value="Create Week Graph" name="createweekgraph" type="submit" /></p><br />
   3:                 </form>
   4:                 <br /><br />
   5:                 <form method="POST" action="/showextended">
   6:                     <p>Show daily high and low temperatures for the following period:</p><br />
   7:                     <p>Starting date<input name="startdate" type="text" id="1popupDatepicker" /></p><br />
   8:                     <p>Ending date <input name="enddate" type="text" id="2popupDatepicker" />
   9:                     <input value="Create Extended Graph" name="createextendedgraph" type="submit" /></p><br />
  10:                 </form>

You can see that the JavaScript tag #popupDatepicker is matched to popupDatepicker in the body section. Anytime that field is accessed, the date popup will be displayed. Same for the other 2 fields.


So that takes care of date formatting. Next is how to actually display the graphs in an elegant matter, in other words, don’t stick them on an ugly HTML page somewhere and be done with it.


After a lot of searching, I settled for zebra_dialog jquery add in. This add in, like quite a few others, will allow for a window to be opened up in front of a web page, with the background faintly visible. So, in my case, the user fills in a date for a weekly graph for instance, then presses Create Week Graph. Next, the application on Raspberry Pi, creates the graph and returns it to the user’s browser. The browser then first show the graph, and waits for the user to press Ok to close it. Control is then returned to the ‘regular’ web page. If there is no graph to be displayed, then the regular web page opens up, plain and simple.


To get the zebra_dialog to display the graph, this is what is included in the <script type=”text/JavaScript”> section of the HTML page:



   1: $(document).ready(function () {
   2:  
   3:        {{cSlashes}} $.Zebra_Dialog('<img border="0" src="{{cGraphFile}}"><BR><a href="javascript:window.print()"><p style="font-family:arial;color:red;font-size:20px;">Print This Page</p></a>', { 'title': 'Temperature Graph', width: 1000, 'custom_class': 'zclass', animation_speed: 1000, overlay_close: false,'position': ['left + 150', 'top + 30']});
   4:  
   5:     });

The {{cSlashes}} reference is replaced at runtime by the web application with either ‘//’ or ‘’ depending on whether or not a graph needs displaying. The {{cGraphFile}} reference is replaced by the name of the file to be displayed, once again this is done at runtime by the Bottle web application. As you can see, there are quite a few parameters you can specify in order to control the look and feel of the dialog box.


The web page generates surprisingly quickly, even when the longest data range is used (almost two months now). Yet another testament to the Pi’s amazing abilities.