Python gotchas # 1 : Indentation Tabs and Spaces

Python gotchas - Indentation

or how not to get caught out when you start using python

At first glance python seems very familiar, especially if you have used other procedural languages such as C or Java - but actually Python is different - in some cases very different, and those differences can trip you up as you progress along your python journey. In this series of occasional posts, I am going to cover some of those gotchas.

Indentation, Tabs and spaces 

As you no doubt already know by now, Python uses the end of a line (in most cases) to delimit statements and uses indentation to identify blocks of code in a loop or function, and Python allows you to use either Tabs or spaces to create your indentations and even mix both Tabs and spaces on the same line but there is a hidden gotcha.

Tabs and spaces look alike in most editors - but if you indent one line with Tabs - and the next with spaces - Python will complain with an IndentationError or a TabError even though the lines will seem to be perfectly aligned in your editor.

This can be such a problem that the official style guide (PEP 008) recommends only using spaces for new code, and setting your editor to insert 4 spaces when the tab key is pressed, and you should only use Tabs to stay compliant with existing code which you might be changing. Make sure you get your editor set up before you start coding - as finding and replacing those tab characters might be tricky when you have several 100 lines of code.

The use of indentation provides another thing to trip over if you are not wary. Since Indentation is important for structure the code you can change the behaviour of your code by indenting or un-indenting a line :

Example 1

flag = False
if flag :
    print "Flag is True"
print "Continuing"

Example 2

flag = False
if flag :
    print "Flag is True"
    print "Continuing"

Example 1 and 2 above are identical, apart from one having the last line indented, but Example 1 generates a line of output, and Example 2 doesn't. You can easily see how this could drastically change how your program behaves, and there isn't a error generated, as both of these are completely valid.

Special care should be taken if you copy and paste code into help sites such as stackoverflow which will often remove formatting, make your code invalid, and make it difficult for other people to help you

Python Weekly # 2 : 10 things to learn as a Python Developer

10 things to learn as a Python Developer

Whether you are contributing to Open source projects, or simply creating applications for your own use, there are a number of tools that you really should learn - theses tools will make your life so much easier.

These are not recommendations for beginners - In fact I would strongly suggest as a beginner you don't worry about extra tools until you are confident with the basic language capabilities - including how to write Object Oriented Code, and how to separate your code into modules and Python Packages.

1. An IDE  - A good IDE can shave significant time off your development cycle. There are many good IDEs available : IDLE, Eclipse, PyCharm (to name but a few).  My personal favourite is PyCharm - I will explore why in another post. There are few projects which will insist collaborators using a particular IDE, although of course some work environments may have standard IDEs which you need to use.
2. Choose a unit test framework or two and get familiar with then. The ability to design, construct and maintain unit tests is invaluable in any serious development project, as you will find your code quality increases (as does your confidence in the code), if you have a set of test cases that you can execute on a regular basis. unittest (Python 3.5), doctest (Python 3.5) and nose (if you are using unittest) are a few examples. 
3. A mocking framework : Mocking is an incredibly powerful technique which allows you to replace a module with artificial version, allowing you fine control over your testing, and allowing you to test your code in ways that might otherwise be impossible (or at least very difficult) to reproduce. A common mocking framework is mock (from Python 3.3 onwards this is part of the standard library)
4. Test Driven Development : A very powerful technique which effectively boils down to defining what your code will do before you start writing it. The difference is that instead of defining the code in a document, you define your code in terms of unit tests which will pass once your code is complete. During development regularly run your tests, so you can understand what is left to develop.
5. A Debugger : It is highly unlikely that your code will be developed bug free - so when you find a bug, knowing how to use the debugger will be significant. Certainly using a debugger well is far more efficient than splattering your code with multiple print statements. Most IDEs (see above) will come with an inbuilt debugger, and of course you can use pdb (Python 3.5). Remember once you find and fix the bug - write a test case for it, and include it in your standard test cases that you run.
6. Data Persistence : Any reasonably complex project will probably have a need to store and retrieve data between executions, and for that you will need to decide how that data is stored and retrieved. Options include : csv files (Python 3.5), json (Python 3.5), pickle (Python 3.5) or sqlite (Python 3.5). Personally I tend to use pickle for small low volume objects, and sqllite for larger volume data storage.
7. Documentation : Don't forget to document your code, and do it as you go. Make sure your code is readable, and that the comments and documentation strings explain what your code does and why (in preference to how it works). Also make sure you have good overall documentation on your complete application - README files, user guides etc. It is surprising how quickly you can forget a piece of code that you wrote a few weeks ago - and end up staring at it for hours trying to work out what it does.
8. Virtualisation : You will probably want to ensure that you isolate your code as you test it, to ensure that you don't destroy your working system. If you don't have a separate test machine, then this could be as simple as creating a clean working directory, but you might well need to use virtualenv or even virtualbox depending on what your code does and how far you want to isolate your testing.
9. Change Control : Even if you have no intention of publishing your code, making use of a good change control system is highly recommended - if nothing else it provides you a long term undo for all of your code. If you learn to use long lived branches, you can keep your next development independent of your working version. Examples include : git, mercurial, and subversion (also called svn). If you decide to contribute to Open source projects though, you will need to be familiar with the change control system that they use.
10. Packaging : Even if you are just creating code for your own use, if you want to ensure a clean installation every time, you probably want to look at packaging your code - either using the Python Packaging tools, or Debian Packaging tools (depending on how your code needs to be installed).

Python Weekly # 1 : Notes on developing an import hook

Notes on developing an import hook

Before we start I should give a warning, this article covers import hooks in python 2.7. The hook mechanism has changed from Python 3.5, although the mechanism in Python 2.7 is still supported.

I am a big fan of using JSON to allow configuration of my application, especially where the configuration is more towards the back end functionality. Many of of my applications have variations of either Class or object factories based on JSON data - using JSON dictionaries as templates for the formation of individual classes or instances, and I soon realised that a more "standard" way of approaching JSON would be ideal.

Using the Python import hook mechanism (See PEP 0302), I set out to develop a way of importing JSON file, and automatically generating classes etc from the data within the JSON - an implementation of data as code.

The result is the importjson library - (available as source on GitHub, and an installable package from PyPi - Python Package Index)

This article documents some of what learned during the development process.

Python exposes two types of import hooks - sys.path_hooks and sys.meta_path :

sys.path_hooks

These hooks are best used when there will be a specific entry on sys.path which only needs the special importer - for instance the zipimport module uses this mechanism.

sys.meta_path

These hooks are appropriate when a special sys.path entry will not exist - i.e. where the things to be imported could exist in anywhere.

It was clear that sys.meta_path is ideal for my purposes, since in my applications often co-locate JSON files with my python files - and I am sure this is the case with many other developers. Also using sys.meta_path mechanism would still work if the JSON files are kept separate from the python files.

The Importer Protocol is well documented in PEP 0302, so I will not cover the details here, but I will cover a few important notes for anyone trying something similar - things I tripped over.

Installation

Your importer not only needs to implement the Importer Protocol, but it will also need to install it self onto either sys.path_hooks, or sys.meta_path.

An entry in sys.meta_path should be an object instance which implements at least the find_module method from the Importer Protocol, while an entry in sys.path_hooks will need to be a callable which when passed a path (from sys.path), returns an object which implements at least the find_module method from the the Importer Protocol (the callable can clearly be a class which accepts the sys.path entry as an argument to the constructor).

Reuse and caching

Regardless of whether the Importer is installed on sys.path_hooks or sys.meta_path the instance will be reused, so care should be taken when deciding which data should be stored on a per instance basis. 

Due to the way that Python caches sys.path_hook entries (sys.path_importer_cache), once an Importer successfully finds one module within an sys.path entry the same Importer will be used to load all other modules in the same sys.path entry. This caching make a sys.path_hook type importer not suitable for situations where many types of importable file might exist within a given directory.

Translation of module names

The Importer Protocol consists of two methods which have a very simple signatures :

• finder.find_module(fullname, path=None) 
• finder.load_module(fullname )

The detail of what these methods need to do are documented in PEP 0302, so I wont cover it again here, but I will note :


Both methods are passed the fullname of the module - i.e. the fully dotted name of the module being loaded, so both methods need to be able to translate the fullname of the module into a file name which can be read and converted to a module, and it makes sense for that translation to be consistent (I cannot think of a situation where you want inconsistency here).

In my module I solved this by the find_module method storing the translation of fullname to file name into a class wide dictionary, and the load_module method uses the same dictionary to identify which file to open.

How you do the translation depends in part whether you have a sys.path_hook or sys.meta_path installed Importer :

• With a sys.path_hook importer, the importer instance is initialized with an entry from sys.path (i.e. a file path), and the find_module method is then passed the full qualified name and the path of the most immediate parent module which applies (or None is the module being imported is a top-level module). Using either the path from the initializer, or the path given as the find_module argument, you can identify the path to the specific module.
• With a sys.meta_path importer, the find_module method is passed a fully qualified module name, and the path the most immediate parent module (or None if the module being imported is a top-level module). The importer should use the the path to the parent if provided, but the importer will need to determine how and where to look for top level modules. My JSON importer does it's own traversal of sys.path to find where to search, but there are other options.

And finally :

During the development cycle beware when reloading your importer module, as this could well result in multiple entries in sys.path_hooks or sys.meta_paths, which could lead to inconsistent test results - take care to remove old entries before reloading.

If during development your importer does go completely wrong, and breaks your ability to import other modules, you have a few different options :

• Remove the importer entry from sys.meta_path
• Remove the importer entry from sys.path_hooks and from sys.path_importer_cache
•  If all else fails, exit the interpreter and start again.

Writing your own import hook may seem scary at first, but in fact it is relatively simple - the key thing is coming up with the idea in the first place.