2013-12-08

Django 1.6 source file count and directory structure

The other day i would like to know how many source files are there in Django 1.6, so I run the following command on the django directory.  Basically
it lists all non-empty python source files and count them.

$ ls -Rl django | grep \.py$ | grep -v " 0 " | wc -l
707

I would also like to know Django 1.6 directory structure, so I run the following command on the django directory again, limiting the tree depth to two:

$tree django -d -L 2
django
|-- bin
|   `-- profiling
|-- conf
|   |-- app_template
|   |-- locale
|   |-- project_template
|   `-- urls
|-- contrib
|   |-- admin
|   |-- admindocs
|   |-- auth
|   |-- comments
|   |-- contenttypes
|   |-- flatpages
|   |-- formtools
|   |-- gis
|   |-- humanize
|   |-- messages
|   |-- redirects
|   |-- sessions
|   |-- sitemaps
|   |-- sites
|   |-- staticfiles
|   |-- syndication
|   `-- webdesign
|-- core
|   |-- cache
|   |-- checks
|   |-- files
|   |-- handlers
|   |-- mail
|   |-- management
|   |-- serializers
|   `-- servers
|-- db
|   |-- backends
|   `-- models
|-- dispatch
|-- forms
|   `-- extras
|-- http
|-- middleware
|-- shortcuts
|-- template
|   `-- loaders
|-- templatetags
|-- test
|-- utils
|   |-- 2to3_fixes
|   |-- translation
|   `-- unittest
`-- views
    |-- decorators
    `-- generic

2013-09-26

Checking Android app publish status in Google play

After publishing an Android app in the Google Developer Console, I wonder how long it will take Google to list it the Play store.

At first I manually checked Google Play every now and then.  Eventually I wrote a BASH script to do just that:

appid='com.amgcomputing.website'
data=`curl -s https://play.google.com/store/apps/details?id=$appid`
echo $data | grep -q "What's New"
if [ $? -eq 0 ]; then
    espeak "application is published"
fi


Here is what it does:

1. Ask curl to retrieve the search result from Google Play, using the reverse domain app id as the search key.

2. Ask grep to find a text string "What's New" in the search result.  If found, it is safe to assume the app is now listed.

3. Ask the computer to let me know verbally.

The -s option of curl and the -q option of grep just tell curl and grep not to write any output to the screen.

I then schedule the BASH script in crontab to run every five minutes. Now I will know the moment the app is listed in Google Play.


2013-09-21

How I turn my website into an Android App - Part 3

In part 3 of this blog post, we are going to prepare the release version of the apk and publish it to Google Play store.

In order to publish an app to Google Play, the apk has to be signed and zip aligned. We are going to do both of them.


Step 1 
Use the keytool utility bundled with the Oracle JDK to generate a private key for signing the apk:

Assuming we are still in the amg directory, 

$ keytool -genkey -v -keystore ./amg.keystore -alias amgkey -keyalg RSA -keysize 2048 -validity 10000

where

"./amg.keystore" = the name and location of the key store
"amgkey" = the name of the key to generate

The encryption algorithm is RSA with a key size of 2048 bits, and the generated key is valid for 10,000 days (27 years).

It will ask you for a passphrase to protect the key store, and a bunch of other questions such as your first and last name, company name, etc.

It will also ask you for a passphrase to protect the key, with the option of using the same passphrase you choose for the key store.

Step 2 
Use the jarsigner utility bundled with the Oracle JDK to sign the apk with the generated key:

$ jarsigner -verbose -sigalg MD5withRSA -digestalg SHA1 -keystore ./amg.keystore platforms/android/bin/AMG-release-unsigned.apk amgkey

where

signature algorithm = MD5 with RSA
digest algorithm = SHA1

It will ask you for the passphrase to the key store.  The apk will be signed in place.

Step 3 
Use the zipalign utility bundled with the Android SDK to align the apk, which is basically a zip file:

zipalign -v 4 platforms/android/bin/AMG-release-unsigned.apk amg.apk

where

-v = verbose output
 4 = alignment in bytes

The signed and zip aligned amg.apk is now ready to upload to Google Play.

That's it.  Hope you like this post.

References
http://developer.android.com/tools/publishing/app-signing.html

How I turn my website into an Android App - Part 2

In part 2 of this blog post, we are going to convert my website into an Android app using Cordova command line interface (CLI).

Step 1 
Create a Cordova project to host the website.

$ cordova create amg com.amgcomputing.website AMG

where:
amg = directory to host the project files
com.amgcomputing.website = reverse domain-style project identifier
AMG = display name of the app

Step 2
Since I am only interested in the Android platform in this post, I remove all icon and screen resources for other platforms.

$ cd amg

$ rm -rf www/res/icon/bada
$ rm -rf www/res/icon/bada-wac
$ rm -rf www/res/icon/blackberry
$ rm -rf www/res/icon/ios
$ rm -rf www/res/icon/tizen
$ rm -rf www/res/icon/webos
$ rm -rf www/res/icon/windows-phone

$ rm -rf www/res/screen/bada
$ rm -rf www/res/screen/bada-wac
$ rm -rf www/res/screen/blackberry
$ rm -rf www/res/screen/ios
$ rm -rf www/res/screen/tizen
$ rm -rf www/res/screen/webos
$ rm -rf www/res/screen/windows-phone

Step 3
Add android platform support to the project.

$ cordova platform add android

Step 4
To keep it simple, I am going to delete the higher resolution drawable resources that come with the platform:

$ rm -rf platforms/android/res/drawable-hdpi
$ rm -rf platforms/android/res/drawable-ldpi
$ rm -rf platforms/android/res/drawable-mdpi
$ rm -rf platforms/android/res/drawable-xhdpi

Then I replace the icon resource (96x96 pixels) that comes with the platform with my own version of it:

platforms/android/res/drawable/icon.png


Step 5
Here is the directory structure of my Bootstrap v3 based website.  I built it using the starter-template.

www
|_ assets
|  |_ js
|_ dist
   |_ css
   |_ js

I put a copy of my website in the www folder of the cordova project, replacing the index.html there.

Step 6
I like to set my app at version 1.0, so I change the version name from the default of "0.0.1" to "1.0" in the www/config.xml file.  During the build process, Cordova will automatically update the Android Manifest (AndroidManifest.xml) located in the platforms/android direcotry to reflect this change.

$ sed -i'' 's/version="0.0.1"/version="1.0"/' www/config.xml

The -i'' option is for in place editing without making any backup.

Step 7
Build the project to generate a debug apk:

$ cordova build android

The generated debug apk is located at:
platorms/android/bin/AMG-deubg.apk

You can also test the app by running it in the Android Virtual Device (AVD) created in Part 1 of this post:

$ cordova emulate android

Step 8
Next is to create a release version of the apk:

$ cordova build --release android

The generated release apk is located at:
platorms/android/bin/AMG-release-unsigned.apk

Step 9
Now we are ready to prepare the release apk for publishing to the Google Play store.

See you in Part 3 of this post.

How I turn my website into an Android app - Part 1

This is an account on how I turned a website into an Android app. The website is built with Bootstrap v3.  I am going to use Apache Cordova v3.0.6 (aka PhoneGap) to convert it into an app.  Almost everything is done in the command line, and the operating system in use is Debian 6.x.  Let's see the result first.  

The website:


The Android app:


or you can see the app website here:

    http://amg99.com/amg.html

This post is divided into three parts:
1. Set up the Cordova environment.
2. Convert the website into an Android app.
3. Publish the Android app to Google Play.

Here is part 1 of the post.

Step 1 
Cordova is installed by Node.js. So we need to intall Node.js first.  We will install it locally in the user's home folder so that we do not root access.

$ wget http://nodejs.org/dist/v0.10.18/node-v0.10.18.tar.gz
$ tar -xzvf node-v0.10.18.tar.gz
$ cd node-v0.10.18
$ ./configure --prefix=$HOME
$ make
$ make install

It will take some time to "make" Node.js.  Please be patient.  Eventually, the Node.js package manager (npm) will be installed in the ~/bin folder.  

Step 2 
Install Apache Cordova using the Node.js package manager (npm).

$ npm install -g cordova

Step 3
Download the ADT (Android Developer Tools) bundle from the Android Developer website here.  We need the ADT to build the Android app.

As of this writing, I have downloaded the following zip file for my Debian 6 (32-bit). You will have to agree with Google terms and conditions before you can download the ADT.

   adt-bundle-linux-x86-20130729.zip (Linux 32-bit)

Unzip it and move its content to a convenient location:

$ unzip adt-bundle-linux-x86-20130729.zip
$ mkdir ~/android
$ mv adt-bundle-linux-x86-20130729/eclipse ~/android
$ mv adt-bundle-linux-x86-20130729/sdk ~/android

Step 4
Download Oracle JDK.  We need the JDK to compile the java source files.  OpenJDK may or may not work.  Your mileage may vary.

The Oracle JDK can be downloaded from the Oracle website here.  You must accept Oracle's license agreement in order to download the JDK.

As of this writing, I have downloaded the following file for my Debian 6 (32-bit):

   jdk-7u25-linux-i586.tar.gz (Linux x86)

Untar it and move it to a convenient location:

$ tar -xzvf jdk-7u25-linux-i586.tar.gz
$ mv jdk1.7.0_25 ~/android

Step 5
Add various paths in ~/.bashrc for convenience:

echo 'PATH="$HOME/bin:$PATH"' >> ~/.bashrc
echo 'PATH="$HOME/android/sdk/tools:$PATH"' >> ~/.bashrc
echo 'PATH="$HOME/android/sdk/platform-tools:$PATH"' >> ~/.bashrc
echo 'PATH="$HOME/android/jdk1.7.0_25/bin:$PATH"' >> ~/.bashrc
echo 'PATH="$HOME/android/eclipse:$PATH"' >> ~/.bashrc

Step 6
Install the Android API by starting the Android SDK Manager:

$ android

and follow the on screen instruction to install the Android API. In this writing, I installed Android 4.3 (API 18).

You may also like to create an Android Virtual Device (AVD) for testing purpose.  You can start the Android Virtual Device Manager from the Android SDK Manager by going to the "Tools" menu and select "Manage AVDs...".

We are now ready to move on to Part 2 of this post, i.e. to convert the website into an Android app using Cordova.

Please note that Cordova can build mobile apps in multiple platforms, including Android, iOS, etc.  In this post, we will focus in Android only.

2013-08-28

Django logging setting

It took me some time to figure out how to set up logging in Django.  What I want to do is to log to a pair of rotating files each 1MB in size.

What I come up with is the following code segment to replace the LOGGING setting in the settings.py:

U_LOGFILE_NAME = r'/path/to/log.txt'
U_LOGFILE_SIZE = 1 * 1024 * 1024
U_LOGFILE_COUNT = 2
U_LOGFILE_APP1 = 'app1'
U_LOGFILE_APP2 = 'app2'

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            #'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
    },
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'logfile': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': U_LOGFILE_NAME,
            'maxBytes': U_LOGFILE_SIZE,
            'backupCount': U_LOGFILE_COUNT,
            'formatter': 'standard',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        U_LOGFILE_APP1: {
            'handlers': ['logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
        U_LOGFILE_APP2: {
            'handlers': ['logfile'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

Usage:

import os
log = logging.getLogger(__name__)

log.debug("debug")
log.info("info")
log.warn("warn")
log.error("error")

This post also appears in Django Snippets.


2013-08-01

Add password to a PDF with PyPDF

PyPDF is a pure python module to manipulate PDF.

Examples:

# add password to a pdf
# add_password.py input.pdf password output.pdf


import sys
from pyPdf import PdfFileWriter, PdfFileReader

output = PdfFileWriter()
input = PdfFileReader(file(sys.argv[1], "rb"))
for i in range(0, input.getNumPages()):
    output.addPage(input.getPage(i))
outputStream = file(sys.argv[3], "wb")
output.encrypt(sys.argv[2], use_128bit=True)
output.write(outputStream)
outputStream.close()


# extract pages from a pdf
# extract_page.py input.pdf start_page end_page out.pdf
# page starts from 1


import sys
from pyPdf import PdfFileWriter, PdfFileReader

output = PdfFileWriter()
input = PdfFileReader(file(sys.argv[1], "rb"))
for i in range(int(sys.argv[2]), int(sys.argv[3])+1):
    output.addPage(input.getPage(i-1))
outputStream = file(sys.argv[4], "wb")
output.write(outputStream)
outputStream.close()


# rotate all pages clockwise
# rotate.py input.pdf degrees output.pdf

# degrees must be multiple of 90 

import sys
from pyPdf import PdfFileWriter, PdfFileReader

output = PdfFileWriter()
input = PdfFileReader(file(sys.argv[1], "rb"))
for i in range(0, input.getNumPages()):
    output.addPage(input.getPage(i).rotateClockwise(int(sys.argv[2])))
outputStream = file(sys.argv[3], "wb")
output.write(outputStream)
outputStream.close()

Troubleshooting ssh "Too many authentication failures"

Some time ago all of a sudden, I was unable to ssh into a host anymore with the above error.  I didn't even have a chance to enter my password.

Eventual the cause of the problem was found.  I have multiple ssh keys but none of them is for the target host.  SSH is configure to use public key authentication before password authentication.  Hence ssh tries all my keys without success, hitting the maximum password attempts, and ends with the error "Too many authentication failures".

The solution:

Force ssh to not use public key authentication, and then copy over
the public key for the target host:

$ ssh-copy-id -i ~/.ssh/id_rsa.pub "-o PubKeyAuthentication=no user@host.com -p 22"

Note the use of double quote, otherwise ssh-copy-id will not recognize 
the ssh options.

2013-06-06

Installing MySQL driver in VirtualEnv

If "pip install mysql-python" fails in a virtualenv with the following errors:

1.  missing python.h
2.  missing mysql_config

try install the following first:

$ sudo apt-get install python-dev
$ sudo apt-get install libmysqlclient-dev



2013-05-14

Set up Django 1.5.1 in DreamHost VPS

This post is about setting up Django 1.5.1 in a virtualenv in DreamHost VPS with Apache2 and mod_wsgi 3.4, and is mostly an update to the following post about three years ago:
http://www.wombatnation.com/2010/06/django-on-dreamhost-ps/comment-page-1
Software Versions:

   Apache 2.2.22
   Django 1.5.1
   mod_wsgi 3.4
   PIP 1.3.1
   Python 2.6.6
   VirtualEnv 1.9.1

1. Sign up a VPS account with DreamHost.

2. Create a regular user [user] in the VPS.  The Django project will run under this account.

3. Create an admin user [admin] in the VPS to manage the web server.  This admin account has sudo privileges.

4. Install Python PIP (1.3.1)

   [admin]$ sudo easy_install pip

   PIP is installed at /usr/local/bin/pip.

5. Upgrade VirtualEnv (from 1.4.9 to 1.9.1)

   [admin]$ sudo pip install --upgrade virtualenv

6. Create a virtualenv and install Django (1.5.1) in it

   [user]$ mkdir ~/django
   [user]$ virtualenv ~/django/env
   [user]$ source  ~/django/env/bin/activate
   [env]$ pip install django

7. Create a Django project (demo)

   [env]$ cd ~/django
   [env]$ django-admin.py startproject demo

8. Verify Django project is working

   [env]$ cd ~/django/demo
   [env]$ python manage.py runserver 0.0.0.0:8000

   Visit http://your.domain.com:8000 to verify Django project is up and running.

9. Install mod_wsgi 3.4

   [admin]$ mkdir ~/src
   [admin]$ cd ~/src
   [admin]$ wget http://modwsgi.google.com/files/mod_wsgi-3.4.tar.gz
   [admin]$ tar xzvf mod_wsgi-3.4.tar.gz
   [admin]$ cd mod_wsgi-3.4
   [admin]$ ./configure --with-apxs=/usr/local/dh/apache2/template/sbin/apxs --with-python=/usr/bin/python
   [admin]$ make
   [admin]$ sudo make install

   mod_wsgi is installed at /usr/local/dh/apach2/template/lib/modules/mod_wsi.so

10. Edit httpd.conf

   [admin]$ sudo vim /usr/local/dh/apache2/apache2-psNNNNNN/etc/httpd.conf

   psNNNNNN is your VPS id.

   add the following line in the LoadModule section:
   LoadModule wsgi_module /dh/apache2/template/lib/modules/mod_wsgi.so

   add the following line outside of VirtualHost directive:
   WSGIPythonPath /home/user/django/env/lib/python2.6/site-packages

   add following line inside the VirtualHost directive:
   WSGIScriptAlias /demo /home/user/django/demo/demo/wsgi.py

11. Restart Apache

   [admin]$ sudo /etc/init.d/httpd2 restart apache2-psNNNNNN

12. Edit wsgi.py

   [env]$ vim ~/django/demo/demo/wsgi.py

   add next two lines at the top of the file:
   import sys
   sys.path.insert(0, '/home/user/django/demo')

13. Create a Django app (demoapp) and update urls.py

   [env]$ cd ~/django/demo
   [env]$ python manage.py startapp demoapp
   [env]$ vim ~/django/demo/demoapp/views.py

   Make it look like:
   from django.http import HttpResponse

   def index(request):
       return HttpResponse("Hello world!")

   [env]$ vim ~/django/demo/demo/urls.py

   Add a url pattern:
   url(r'^$', 'demoapp.views.index')

14. Verify

   Visit http://your.domain.com/demo, will see "Hello World!" in browser.

-End-

2013-04-10

Password protect a PDF

The othe day, I wrote a CherryPy application that add a password to the PDF, and run it behind Apache2 with the mod_wsgi module.  The actual adding of password is done by pdftk console tool.

The entire CherryPy application is in one file main.py as below.

The index() method presents the UI to the web browser, asking the user to select the PDF and enter a password.

The upload() method shells out to the console and calls pdftk to add the password, and then returns the password-protected file back to the browser, letting the user to save the PDF to local storage (e.g. the user's hard disk).

The last part of the file deals with hooking it up with Apache2/mod_wsgi.  If main.py is run from the command line, then the CherryPy built-in HTTP server will start, otherwise, it assumes it is run behind Apache2 and a WSGI application is created.

# main.py
# requires cherrypy
# requires pdftk (command line tool)
# dkf 130401 creation

import os
import tempfile

import cherrypy

class PdfPass(object):
    def index(self):
        return """
        <html><body>
            <h2>Add password to PDF</h2>
            <form action="upload" method="post" enctype="multipart/form-data">
            <table>
            <tr>
            <td>Select PDF:</td>
            <td><input type="file" name="pdf" size="60"/></td>
            </tr>
            <tr>
            <td>Password:</td>
            <td><input type="password" name="pass1" value="" size="20" maxlength="40"/></td>
            </tr>
            <tr>
            <td>Password again:</td>
            <td><input type="password" name="pass2" value="" size="20" maxlength="40"/></td>
            </tr>
            <tr>
            <td colspan="2"><input type="submit" value="Add password"/></td>
            </tr>
            </table>
            </form>
        </body></html>
        """
    index.exposed = True

    def upload(self, pdf, pass1, pass2):
        if not pass1:
            return "Password cannot be emply!<br/>Please go back and correct."

        if pass1 != pass2:
            return "Passwords do not match!<br/>Please go back and correct."

        # read in the user uploaded pdf
        temp1 = tempfile.mktemp()
        with open(temp1, "wb") as f:
            f.write(pdf.file.read())
            f.close()

        # call pdftk to add password to pdf
        temp2 = tempfile.mktemp()
        os.system('pdftk %s output %s user_pw %s' % (temp1, temp2, pass1))
        with open(temp2, "rb") as f:
            data = f.read()
            f.close()

        # clean up temp files
        os.remove(temp1)
        os.remove(temp2)

        # deliver the password protected pdf to the user
        cherrypy.response.headers['Content-Type'] = "application/pdf"
        cherrypy.response.headers['Content-Disposition'] = 'attachment, filename="%s"' % pdf.filename
        return data
    upload.exposed = True

if __name__ == '__main__':
    cherrypy.quickstart(PdfPass())
else:
   # cherrypy.config.update({'environment': 'embedded'})
    application = cherrypy.Application(PdfPass(), script_name=None, config=None)


On the Apache side (Apache2 in Debian6), need to modify the /etc/apache2/sites-available/default, adding the following lines inside the <VirtualHost *.80> block.  Then the CherryPy application will be available at http://localhost/<url>.

# must run wsgi in daemon mode otherwise content-disposition may not work
WSGIDaemonProcess cherrypy processes=2 threads=15 display-name=%{GROUP}
WSGIProcessGroup cherrypy
WSGIScriptAlias /<url> /full/path/to/main.py
<Directory /full/path/to>
    Order allow,deny
    allow from all
</Directory>


-End-


2013-03-09

Checking battery life

I often work at the console, and I use the following bash script to check the battery life of my laptop.  It makes use of the ACPI (Advanced Configuration and Power Interface) in my Linux distro to retrieve information on the battery, and does some simple calculation to get the remaining time left to battery depletion.


#!/bin/bash
# check battery charge
rate=$(sed -n 4p /proc/acpi/battery/BAT1/state | sed 's/ \+/$/g' | cut -d$ -f3)
left=$(sed -n 5p /proc/acpi/battery/BAT1/state | sed 's/ \+/$/g' | cut -d$ -f3)
left=$(echo -e $left \* 60 / $rate | bc)
echo $left "mins left"

[Edited 2013-01-27]

I have modified the script to add a check if the laptop is plugged in and charging.  If so, exit gracefully with calculating the remaining time left.

#!/bin/bash
# skip if charging
state=$(cat /proc/acpi/battery/BAT1/state | sed -n 3p | awk '{print $3}')
if [ $state == 'charging' ]; then
    echo 'charging'
    exit 0
fi
# check battery charge
rate=$(sed -n 4p /proc/acpi/battery/BAT1/state | sed 's/ \+/$/g' | cut -d$ -f3)
left=$(sed -n 5p /proc/acpi/battery/BAT1/state | sed 's/ \+/$/g' | cut -d$ -f3)
mins=$(echo -e $left \* 60 / $rate | bc)
echo $mins "mins left"


[Edited 2013-04-10]
The Linux distro in this post is Backtrack 5 Release 3.