2015-12-14

Auto loading user-defined models in Django shell

I would like the Django shell to auto-load all user-defined models on startup without the 
use of a third-party app. Below is that I did.

[00] # FILE .djangoshellrc
[01] # alias m='PYTHONSTARTUP="path/to/.djangoshellrc" python manage.py'
[02] 
[03] import sys
[04] 
[05] from django.apps import apps
[06] 
[07] def load_models():
[08]     try:
[09]         models = [m for m in apps.get_models() if not m.__module__.startswith('django')]
[10]         for model in sorted(models, key=lambda x: x.__name__):
[11]             name = model.__name__
[12]             globals()[name] = model
[13]             print("[+] imported %s" % name)
[14]     except:
[15]         print("[*] %s" % sys.exc_info()[0])
[16]         print("[*] %s" % sys.exc_info()[1])
[17] 
[18] load_models()

Line 00-01: To use this script, create a bash alias to start the Django shell, with 
this script specified in the PYTHONSTARTUP environ variable.  In other words, this 
script will be run on startup as long as the alias is used to start the Django shell.

Line 03-05: All the necessary imports.

Line 09: Ask Django to provide a list of all the app models, with the django built-in 
models filtered out.

Line 10: Iterate the models list after sorting them based on the model names.

Line 11-13: Add the models to the globals dict with the model name as the key,
effectively doing a "from <app>.models import <Model> in the Django shell."

Hope this script is useful to you.

2015-12-07

Comparing changes in a Django form

With the help of the Form.changed_data list, the Form.initial dict, and the Form.cleaned_data dict, Django makes it easy to compare changes in a form.
For example, we can override the form_valid method in an UpdateView to print the old and new values of all the changed fields as below:
def form_valid(self, form):
    for fieldname in form.changed_data:
        print(fieldname)
        print("old=%s" % form.initial[fieldname])
        print("new=%s" % form.cleaned_data[fieldname])
    return super().form_valid(form)

References:
Form.changed_data
Form.initial
Form.cleaned_data

2015-11-30

ReportLab Chinese Support

I need to generate a PDF with Chinese characters in a Django project. Generating a PDF is straightforward as ReportLab does the job well. The hard part is to make sure the user can read the Chinese characters in the PDF.
At the end I settled in using the Chinese font provided by the Adobe's Asian Language Packs. That way, if no usable Chinese font is available, the user can download and install the Chinese font on demand automatically. This approach offers good performance since nothing is embedded in the PDF.
The following is the code snippet in registering the Chinese font for ReportLab to use.
from reportlab.pdfbase import pdfmetrics
from reportlab.pdfbase.cidfonts import UnicodeCIDFont
font_chinese = 'STSong-Light' # from Adobe's Asian Language Packs
pdfmetrics.registerFont(UnicodeCIDFont(font_chinese))
...
canvas.setFont(font_chinese)

2015-11-22

Django Form - Confirm Before Saving (Part 2)

In Part 1 of this post, we create a form which conditionally shows a checkbox for user confirmation. The implementation is to raise an validation error when a condition is met. That does not work well with multiple confirmations, as the checking stops when the first condition is met. We need a way to check all conditions without raising an validation error, with one checkbox to confirm them all. We can do that with Form.add_error()
Instead of raising an validation error like in Part 1 of this post:
class InvoiceForm(forms.ModelForm):
    ...
    def clean(self):
        super().clean()
        ...
       if all((key in self.cleaned_data for key in ['date', 'confirm'])):
           if self.cleaned_data['date'].weekday() == 6 and not self.cleaned_data['confirm']:
                raise forms.ValidationError(MSG001)
We add an validation error when a condition is met:
class InvoiceForm(forms.ModelForm):
    ...
    def clean(self):
        super().clean()
        ...
        if 'confirm' in self.cleaned_data and not self.cleaned_data['confirm']:
            if 'date' in self.cleaned_data:
                d = self.cleaned_data['date']
                if d.weekday() == 6:
                    self.add_error(None, forms.ValidationError(CONFIRM_MSG[1]))
                if (datetime.date.today() - d).days > 14:
                    self.add_error(None, forms.ValidationError(CONFIRM_MSG[2]))
In the example above, a confirmation is required if either the invoice date falls on a Sunday or the invoice date is more than 14 days in the past. CONFIRM_MSG is a dict of confirmation messages such as the following:
CONFIRM_MSG = {
    1: "The invoice date does not fall on a Sunday.  Please check confirm to proceed.",
    2: "The invoice date is more than 14 days in the past.  Please check confirm to proceed.",
}
In the Form.__init__() method, we check for existence of any of the confirmation messages to show the checkbox. Note the use of the Form.non_field_errors() method to retrieve all non-field errors, such as those added in the Form.clean().
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        ...
        confirm_set = set(CONFIRM_MSG.values())
        if confirm_set.intersection(self.non_field_errors()):
            self.fields['confirm'].widget = forms.CheckboxInput()

2015-11-17

Django Form - Confirm Before Saving (Part 1)

How can I ask the user to confirm before saving a form if the form meets certain conditions? I tried different approaches and settled on the following.
This example is about saving an invoice. I would like to ask the user to confirm (by checking a checkbox) before saving an invoice if the invoice date falls on a Sunday. The checkbox will only appear if confirmation is required.
Step 1: Add a checkbox in the form as a hidden input.
class InvoiceForm(forms.ModelForm):
    confirm = forms.BooleanField(initial=False, required=False, widget=forms.HiddenInput)
    class Meta:
        model = Invoice
        fields = ('data', ..., 'confirm',)
Step 2: Check for the condition in the form.clean() method. The required data are retrieved from the form cleaned_data dictionary. Note that it is a straightly a read only access. If the condition is met, raise a validation error thereby invalidating the form.
MSG001 is a string constant used as the error message. For example something like "The invoice date does not fall on a Sunday. Please check confirm to proceed."
class InvoiceForm(forms.ModelForm):
    ...
    def clean(self):
        super().clean()
        ...
        if all((key in self.cleaned_data for key in ['date', 'confirm'])):
            if self.cleaned_data['date'].weekday() == 6 and not self.cleaned_data['confirm']:
                raise forms.ValidationError(MSG001)
Step 3: Check for the error condition in the __init__() method of the form, and dynamically change the checkbox from a hidden input to a regular checkbox on the fly. This effectively adds a checkbox to the form and unless it is checked, the form is invalid.
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        ...
        if any(e == MSG001 for e in self.errors.get('__all__', [])):
            self.fields['confirm'].widget = forms.CheckboxInput()
Step 4: In the HTML template, add conditional logic to show the checkbox either as a hidden input or as a checkbox as appropriate.
{% if form.confirm in form.hidden_fields %}
  {{ form.confirm }}
{% else %}
  {{ form.confirm.label_tag }}{{ form.confirm }}
{% endif %}
Final thoughts:
This approach does not require AJAX or any third party packages. The reason is to minimize Javascript knowledge and external dependency.
This approach uses validation error as a signal to add a checkbox to the form.
This approach modifies a form field dynamically in the form.__init__() method.
Also, in another approach, I tried to to update the form.cleaned_data dict without success. The changes I made are lost somewhere in the call chain. It is evident that I do not have sufficient knowledge in how Django forms work.
Continue to Part 2 of this blog for working with multiple confirmations.

2015-07-20

Abstract methods in Python3

You can use the Abstract Base Classes to implement abstract methods in Python. For example, if you have an Animal class and a Cat class like the following:
class Animal():
    def speak(self):
        print("Animal speak")

class Cat(Animal):
    pass

class Doc(Animal):
    pass

Cat().speak() # returns "Animal speak"
Dog().speak() # returns "Animal speak"
You can turn Animal.speak() to an abstract method with a metaclass and a decorator:
import abc

class Animal(metaclass=abc.ABCMeta):
    @abc.abstractmethod
    def speak(self):
        print("Animal speak")

class Cat(Animal):
    def speak(self):
        print("Cat speak")

class Dog(Animal):
    pass

Cat().speak() # returns "Cat speak"
Dog().speak() # returns "TypeError: Can't instantiate abstract class Dog with abstract methods speak"
Note that although the Animal.speak() method is always shadowed by its subclass method, it can still be called explicitly by using super():
class Cat(Animal):
    def speak(self):
        super().speak()

Cat().speak() # returns "Animal speak"

2015-07-13

Add a button to export the search results in Django - Part 2

Refering to a previous post, it turns out that using str() to serialize the QuerySet query is not a viable solution if one of the selection criteria is a date. The str() will not quote the date value resulting in an invalid SQL statement.
Using pickle to serialize the query solves the problem.
# Replace str()
request.session['search_sql'] = str(Person.objects.filter(**kw).query)

# with pickle.dumps()
request.session['search_query'] = pickle.dumps(Person.objects.filter(**kw).query)
When the time comes to re-execute the query:
# Rather than executing a raw sql statement:
sql = request.session.get('search_sql')
object_list = Person.objects.raw(sql)

# deserialize the saved query and assign it to the QuerySet instead:
saved_query = request.session.get('search_query')
query = pickle.loads(saved_query)
object_list = Person.objects.all()
object_list.query = query
Although less secured, in order to store the query object in the request session, have to use the pickle based session serializer instead of the JSON based counterpart in Django 1.7.
=== settings.py ===
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'

2015-07-06

Set the width of an input field in an inline formset

To set the width of an input field in an inline formset, you can specify the widget and its attributes in the inlineformset_factory call. For example:
 LineFormSet = forms.models.inlineformset_factory(Document, DocumentLine, 
-                  fields=['seqno', 'sku', 'desc', 'qty', 'unit_price'])
+                  fields=['seqno', 'sku', 'desc', 'qty', 'unit_price'], 
+                  widgets={
+                    'seqno': forms.TextInput(attrs={'size':'10'}),
+                    'sku': forms.TextInput(attrs={'size':'10'}),
+                    'desc': forms.TextInput(attrs={'size':'50'}),
+                    'qty': forms.TextInput(attrs={'size':'10'}),
+                    'unit_price': forms.TextInput(attrs={'size':'10'})})
Reference: https://docs.djangoproject.com/en/1.7/topics/forms/modelforms/#specifying-widgets-to-use-in-the-inline-form

2015-06-29

Automatic version numbering with Mercurial hook - Part 2

The pre-commit hook commands in the last post only works in Linux or Mac. I need to run it in Windows as well. Instead of converting it to MS-DOS commands, I found it easier to write an external python script as the hook handler, as it will work in Windows, Linux and Mac.
Step 1 - Modify the hook to run an external script.
# .hg/hgrc

[hooks]
pre-commit.version = python:./script/hghooks.py:increment_version
Step 2 - Create a python module with the handler function.
It basically uses the subprocess module to get the local revision number, strip any ending newline or plus sign, add one to it, and then write it out to the version.py file. Note that we obtain the repo root folder from the repo object passed into the hook handler, so that we don't have to hardcode the path to the version file.
# ./script/hghooks.py

import os.path
import subprocess

def increment_version(*args, **kw):
    repo = kw['repo']
    version = int(subprocess.check_output(["hg", "id", "-n"]).strip().strip("+")) + 1
    setting = "VERSION = %s" % version
    file = os.path.join(repo.root, "dms/dms/version.py") 
    with open(file, "w") as fp:
        fp.write(setting)
    print(setting)
Using an external script makes it not only cross-platform, but also version control friendly because it is not stored in the .hg folder!

EDITED 2015-07-09 In order to take care of multiple heads, the increment_version() is revised as follow:

# ./script/hghooks.py

import os.path
import subprocess

def increment_version(*args, **kw):
    """hg id -n can returns:
    145      # no uncomitted changes
    145+     # with uncomitted changes
    146+145+ # multi-heads"
    The strip() and split() will take care of all the above cases.
    """
    repo = kw['repo']
    version = int(subprocess.check_output(["hg", "id", "-n"]).strip().split("+")[0]) + 1
    setting = "VERSION = %s" % version
    file = os.path.join(repo.root, "pms/pms/version.py") 
    with open(file, "w") as fp:
        fp.write(setting)
    print(setting)
EDITED 2015-07-15
The script is further enhanced by removing the hard-coded file path to the Mercurial project config file, and add a test to make sure something has changed indeed before incrementing the version number. The final version of the script is listed below, and is also available in Bitbucket.
import os.path
import subprocess

def increment_version(ui, repo, **kw):
    """
    An in-process Mercurial hook to increment a version setting.

    FILE .hg/hgrc

    [hooks]
    pre-commit.version = python:path/to/hghooks.py:increment_version

    [hook_params]
    version.file = path/to/version.py

    ENDFILE

    hg id -n can returns:
        145      # no uncomitted changes
        145+     # with uncomitted changes
        146+145+ # multi-heads"
        The strip() and split() will take care of all the above cases.
    """
    has_changes = subprocess.check_output(["hg", "stat"])
    if has_changes:
        version = int(subprocess.check_output(["hg", "id", "-n"]).strip().split("+")[0]) + 1
        setting = "VERSION = %s" % version
        path = ui.config('hook_params', 'version.file', default='version.py', untrusted=False)
        if not os.path.isabs(path):
            path = os.path.join(repo.root, path) 
        with open(path, "w") as fp:
            fp.write(setting)
        print(setting)

2015-06-22

Automatic version numbering with Mercurial hook

Assuming you put your version number setting in a version.py file as VERSION = NNN, you can add the following to the project .hg/hgrc to automatically increment the version number upon each commit. The version number is actually the local revision number before the commit plus 1, although not exactly robust but is generally increasingly upward. Any conflict should be resolved manually.
# .hg/hgrc
[hooks]
pre-commit.version = echo VERSION = $((`hg id -n | tr -d +` + 1)) > ./path/version.py
We use Mercurial pre-commit hook to update the version number in the file version.py. When the hook is run, the current directory is set to the repo's root directory. We get the local revision number with "hg id -n", remove the trailing "+" if any, and then add one to it as the version number. Please note that the hook is pre-commit not precommit. Both hooks exist, but if you use the precommit hook, you will not get what you want. See this post for details.
EDITED 2015-07-06: See Part 2 for a different approach.

2015-05-25

PyQt5 Password Dialog

The following is the complete source code showing how to program a password input dialog in PyQt5. The password dialog is based on QInputDialog, with the QLineEdit.Password set as the echo mode. Note that we do not need the QApplication to provide an event loop in this simple demonstration.
import sys

from PyQt5.QtWidgets import QApplication, QInputDialog, QLineEdit

def getPassword():
    text, ok = QInputDialog.getText(None, "Attention", "Password?", 
                                    QLineEdit.Password)
    if ok and text:
        print("password=%s" % text)

if __name__ == '__main__':
    app = QApplication(sys.argv)
    getPassword()