Wednesday, May 28, 2014

How to safely override __init__ or __new__ in Python 2.6 and later

Beginning with Python 2.6, following from bug 1683368, object.__init__ and object.__new__ deprecate the support for arbitrary arguments. The old behavior was deprecated in Python 2.6 and removed in Python 3.3. As a result, any subclass that overrides either of these methods must be aware of the presence or absence of the overridden method in the MRO up to but not including 'object'. I articulated this unexpected behavior in this post, giving a simple trivial example.

If that's confusing, don't fret. The details aren't important, because what I'm presenting here is a technique that's engineered to be safe in all cases without baking in the details of the parent class.

class SomeClass(SomeParentClass):
    def __new__(cls, *args, **kwargs):
        super_new = super(SomeClass, cls).__new__
        if super_new is object.__new__:
            return super_new(cls)
        return super_new(cls, *args, **kwargs)

    def __init__(self, *args, **kwargs):
        super_init = super(SomeClass, self).__init__
        if super_init.__objclass__ is object:
        super_init(*args, **kwargs)
The above example presents a generic, trivial implementation of a subclass of SomeParentClass that overrides both __init__ and __new__. It does not add any custom behavior (as a typical override would), but instead focuses on the technique on passing through the calls of __init__ and __new__ to the parent class in a safe way.

If you're overriding __new__ or __init__, especially in a subclass of a class over which you do not have control over the implementation, it is important that you check that the parent call will not invoke __new__ or __init__ with parameters. Otherwise, you may run into one of the following errors when constructing your subclass:

TypeError: object.__init__ takes no parameters.
TypeError: object.__new__ takes no parameters.

Friday, April 18, 2014

How to write the perfect pull request

As maintainer of several open-source packages, I'm frequently handling requests from contributors to incorporate changes into the codebase. This article outlines the steps to making those contributions eminently acceptable. This article assumes the reader is familiar with Distributed Version Control Systems such as Git or Mercurial and the pull request process in a service such as GitHub or Bitbucket. Reference GitHub's Using Pull Requests or Atlassian's Create a Pull Request for background on those topics.

1. Locate or file a ticket describing the issue. First search the history to see if the issue has already been reported. If you cannot find an existing issue, then create one. It's okay if you create a duplicate - the project maintainers can link the tickets, but doing a little research up front will save everyone time.

2. In the ticket, describe the problem as you faced it. Include how you encountered the issue and in what environment. Try to include enough detail to allow another person to replicate your findings. If possible, distill the problem down to a minimal program that triggers the failure. If a minimal program cannot be created, try to characterize the issue such that another reader can understand the failure. Include error messages, stack traces, and other detail in their entirety.

3. Research the project-specific contribution policy. Many projects explicitly advertise a contribution policy as a CONTRIBUTING file in the root of the repository, or in other documents such as README. If you're new to a project or unfamiliar with its nuances, doing a little research up front could save you some time and trouble.

4. Start small. Unless your name already carries widespread reputation within the community, your first contribution is likely indistinguishable from that of any other contributor. If you start with small changes, you can establish a rapport with the project and avoid wasting precious time and effort.

5. Prepare your fork of the code for contribution. You haven't committed any code to your fork of the code yet, but you're about to do so. Because you've already been troubleshooting the problem, you may have already even drafted a fix. Set aside those changes (using your DVCS tools) and work from a clean copy of the code as found with the project.

6. Avoid making stylistic changes. This step, while a non-operation, is important. Try to avoid applying your preferred style (regardless how authoritative that style may seem) to the code. Adopt the style of the project as much as possible. In rare cases, your editor will automatically make some stylistic changes when saving (such as removing trailing whitespace from lines). If that is the case, either disable that feature in your editor or commit the stylistic changes separately and prior to any substantive changes. Doing so will allow the reviewers to accept or reject the stylistic changes separately and not be distracted by them.

7. Write a test to capture the desired behavior. If possible, write a test in the test suite of the target project that captures the failure or intended new behavior. Often, the distilled program from above can be re-purposed to accomplish this goal. Commit this change prior to any substantive changes. This newly-created test will now fail at that revision, proving that the issue is fixed in subsequent commits.

8. Implement your suggested fix. Try to minimally adjust the arrangement of the code. If the change is more than a few lines fix, consider making several incremental commits working toward the ultimate solution. For example, if the fix includes extracting a method from a larger method, consider doing that extraction as a separate commit. Then, each commit describes the contributor's thought process working toward the solution. Most code review tools will allow the reviewer to see the commits in aggregate or separately, so more incremental contributions give the reviewer better visibility without compromising the big picture.

9. When committing your changes in (5) and (6), consider using commit messages that reference the ticket number. Both GitHub and Bitbucket can automatically link commits to a ticket if the commit message properly references the ticket. Follow the example of prior commits in that project to be consistent. Creating these references means you'll get the best attribution for your contribution and means less work for the reviewer. Use casual references for incremental commits and then use phrases like "Fixes #NNN" for the commit that actually fixes the issue.

10. Create the pull request referencing the ticket. The pull request should include a description referencing the ticket it fixes and any other information that will help the reviewer understand your thought process. Highlight any areas that you think might be controversial or incomplete and give additional justification for those areas here, along with any factors that mitigate those concerns.

 If you do all of these things while creating your pull request, a reviewer will find it hard not to accept your contribution. Not all of these steps are necessary to accept a contribution. Sometimes a test suite isn't available in a project, or the contribution doesn't justify a test. Perhaps you only have enough experience to write the ticket or the test, but not implement the solution. Use your best judgment and contribute what you can. In the open-source world, good project managers understand that we have limited resources and capabilities and will make the best of any contribution.

Finally, don't be offended if your pull request is rejected. Different projects and their maintainers have different standards for accepting contribution. Hopefully, they will provide constructive criticism that will help you improve your contribution. If not, ask for specific feedback on how you can improve your contributions for the future.

Monday, May 21, 2012

Do you keep your Apple ID secret?

As if erasing my operating system this weekend weren't enough. Late last week, Apple also managed to burn two of my passwords for my Apple ID account. Here's the feedback I sent to Apple:

First a brief introduction. I'm a software engineer and director of DevOps at my company. I have a masters degree in computer science with emphasis on secure networks and cryptography.

I recently encountered a fundamental problem with the password policy of Apple IDs. My password was "disabled for security reasons", allegedly due to to many incorrect password attempts, even though I assure you I did not enter the incorrect password even once on any of my devices.

So it seems that someone or something else has triggered the disabling of my account. As a result, I was forced to change my password.

In other words, because someone failed to guess my secure password, you force me to change it to something else. And furthermore, Apple refuses to tell me anything about when or where (physically or virtually) the failed attempts originated.

Do you see the problem with this scenario? Something completely out of my control causes me to have to reset my password (and never use that password again). Every time this happens, as a user, I'm going to be more inclined to choose a less secure password than before, because obviously the security of the password is irrelevant, but if it's going to be changed arbitrarily, I don't want it to choose something hard to remember or mutate.

To make matters worse, I changed my password on Thursday night, but by Friday night, my *new* Apple ID password was no longer recognized.

According to the support representative I talked to today, this issue was my fault because I failed to sign out and sign back in with the new password on each application that uses my AppleID. Let me just briefly list all of the places I've discovered (so far) where I'm supposed to change my password every time my password is arbitrarily disabled and reset:

- App Store (iPhone)
- App Store (iPad)
- iMessage (iPhone)
- iMessage (iPad)
- FaceTime (iPhone)
- FaceTime (iPad)
- iCloud (iPhone)
- iCloud (iPad)
- Game Center (iPhone)
- Game Center (iPad)
- Web Browser (Windows Desktop)
- Web Browser (Windows Laptop)
- Web Browser (Mac Laptop)
- iTunes (Windows Desktop)
- iTunes (Windows Laptop)
- iTunes (Mac Laptop)
- iCloud Control Panel (Windows Desktop) [and it loses the settings when I sign out and sign in]

And there are probably other places I haven't yet remembered. But you see, according to the rep, it's my responsibility to change it in all of the places it exists, or else Apple will disable my account again, and I will be forced to change my password again, and the process starts anew (plus one more burned password).

What makes matters worse is I tried to resolve this situation using online resources, and then e-mail support, and then genius bar, and then phone support, but none of these avenues was able to answer my questions (why did this happen?) or allow me to resolve the situation to my satisfaction (restore my account with the secure password I've used for some time or my new secure password and prevent this situation from happening in the future). In fact, I was told I can expect this to happen again unless I change my Apple ID (and that's the same answer they'd give to Tim Cook). What a terrible experience.

I'm really regretting buying into the whole Apple ID. Please route this message to the appropriate department so these issues can be considered for improvements to the Apple ID password policy.

In the meantime, I'll try to forget the hours of grief Apple has cost me this weekend.

I didn't mention in the letter that I asked what would happen if someone had my Apple ID and was causing my password to get disabled. The representative indicated I would have to get a new Apple ID. That's right folks. Keep your Apple ID secret (the username). Hide your screen when you're prompted to enter your Apple ID password. Because if the wrong people get a hold of that Apple ID, and (ab)use Apple's site to burn your password, you'll be forced to go through the whole mess outlined above.

Which begs the question - does Tim Cook have to deal with this problem if someone gets his Apple ID? I'm not suggesting that anyone should track down Tim Cook's Apple ID and try bogus passwords until his account is locked out, but I suspect if that happened often enough, Apple would probably devise a better solution than what they have now (which is bullsh**).

Goodbye Apple

My MacBook Air is now for sale. Here is the feedback I sent to Apple following my ordeal this weekend, entitled Lion upgrade corrupted my Windows partition:

There are many reports out there of how an upgrade from Snow Leopard to Lion corrupts a Windows Boot Camp partition (usually beyond repair). This happened to me this weekend. I went to the Apple Store to find out what options I had to upgrade my Boot Camp drivers, and he said my only option to get ongoing upgrades for Boot Camp was to upgrade to Lion. So I did, but instead of getting the drivers I needed, I lost my Windows partition.

This is an issue that was reported last year when Lion was released, but Apple seems to have just ignored the problem, allowing their software to corrupt Windows partitions created by the Boot Camp assistant.

This disdain for your Windows users is reprehensible. That you could allow your software to continue to corrupt installations like this for 10 months after the issue was reported is nothing short of irresponsible.

At the very least, the Lion upgrade should detect a Boot Camp installation and warn that it might corrupt the Windows partition. Even better would be to safely resize it rather than writing over it. What good is a recovery partition if it's written over the data you care the most about?

For me, this is the end of my relationship with Apple for my computing platform. I am selling my MacBook Air and will be getting a Windows laptop. While the hardware is nice, it's not worth the grief of actively destructive software.

I do advanced software and systems work. I've done custom partition management on PCs for years. I've upgraded Windows operating systems for years. Still, I've never encountered a situation where an operating system would overwrite a user's partition without extensive warnings and prompts. Apple is now permanently on my black list.

Wednesday, January 11, 2012

How I install Python on Windows

I've wanted for some time now to document the process I use to install Python and develop, but on Windows. First, let me give a brief background.

I tend to use Windows and I've been programming in Python since 1997. I also tend to use a lot of Unix conventions, especially where they don't create a major incompatibility with Windows. I don't use compatibility layers like Cygwin, though. I find they create more trouble than they're worth. So where possible, I use native implementations of everything. I'm a command-line junkie, and I try to automate everything. One of the best things about Python is its cross-platform support. I've never encountered a better general-purpose language for cross-platform development.

So over the years, I've developed some routines to automate the installation and configuration of Python on my systems. Ever since Python 2.6 and Windows Vista, life has been pretty good. Vista added support for symbolic links and Python 2.6 had excellent 64-bit support.

Since then, I've install almost exclusively 64-bit versions of Python. In this article, I will assume one is using Windows 7 x64, although the routines generally translate to x86. The biggest challenge to using 64-bit Windows is access to compiled modules. To the best of my knowledge, it's not possible to compile 64-bit extension modules without the full version of Visual Studio.

Most Python programmers will find that having several different Python versions present is useful if not necessary. I typically install the latest Python 2 and the latest stable Python 3.

As any Python programmer knows, having distribute is a nearly essential part of any Python installation, so the installation routines should include support for installing distribute as well.

So to accomplish all of this, I needed a scripting language that comes pre-installed with Windows. Originally, I used cmd scripts, but found those not quite adequate. Bundled with the fact that PowerShell 2 comes with Windows 7, it became the language of choice. I developed this script. If you run that script in a PowerShell, it creates three functions:
  • InstallPythonMSI $installer, $target
  • get-python-version $version
  • bootstrap-python
InstallPythonMSI is simply a helper function used by get-python-version. It basically executes a Python .msi file so that it installs silently to the target location. It was not easy generating that code to work properly to pass the parameters to msiexec with quotes in the right places.
get-python-version completely automates installing a single Python version to an appropriate location in $env:ProgramFiles. The script assumes 64-bit, but can be easily modified to support 32-bit. It then also installs distribute by executing the bootstrap script directly from the server, enabling support for setuptools-packaged files.
Finally, bootstrap-python sets up some symbolic links and resets the file associations to create an authoritative (default) Python installation. By default, Python installs into C:\Python27 or similar. I've argued that this location is not appropriate and that the proper location for programs (including Python) is $env:ProgramFiles, so that's where I install it (after all, how do you differentiate between 32-bit and 64-bit versions of Python 2.6?). However, it is useful to have a symbolic link from an easy-to-type location. So bootstrap-python creates a symlink at C:\Python which refers to the "active" version of Python. It then installs and uses one of its command-line scripts to set some environment variables (such as adding the Scripts directory to the path and registering .py files as executable scirpts). Then, bootstrap-python resets the file associations by calling this Python script.
What this all means is that running Python.exe or running scripts installed in the Scripts directory will use the active version of Python (whatever C:\Python points to). This configuration also means that one can easily switch the entire system Python environment from one version of Python to another by just repointing a symlink. To automate this process, I've created this Python script, which I call ''. I usually place this script in a separate location somewhere on the search path (C:\Windows would work). The script searches $ProgramFiles and $ProgramFiles(x86) for PythonXX directories and then allows the user to select one of those, then repoints the C:\Python symbolic link at that version.
The end result is one can install multiple versions of Python in both 32-bit and 64-bit versions and rapidly switch the active version. Of course, it's also possible to select a specific version directly using its canonical location (e.g. C:\Program Files\Python32\python.exe).
With these few scripts, I can rapidly install, bootstrap, and switch between Python versions.
I hope sharing some of these techniques will give some useful techniques that I enjoy using to automate deploying Python on my Windows servers and workstations.

Sunday, January 8, 2012

What would incite me to buy yet another Macbook Air

Last year, I bought my first Mac hardware. Since then, I've found the experience to be mostly wonderful. I love the MagSafe power connector, the SSD drive, the multi-touch Trackpad, and the overall engineering of the hardware.

I've managed to work around most of the rough edges running Windows on this hardware, but there are still three issues that regularly annoy me, and which give me great hesitance when considering another Macbook.

Swap Fn and Ctrl Keys. If the Macbook would provide a way to switch this in hardware. My fingers are trained to find Ctrl in the corner. That's where it is on my desktop keyboard. I'd like it to be in the same place on my laptop.

Clumsy Trackpad driver. The Boot Camp driver for the trackpad has some poor behaviors (compared to any other pointing device I've used).

No HiFi Bluetooth Audio support. I recently found there is no A2DP support for the bluetooth adapter integrated with the Macbook Air (or any MacBook on Windows for that matter).

If Apple were to fix these three issues, it would shift the balance in my hesitance and I would rush out and upgrade my laptop now to another piece of Apple hardware.

Tuesday, July 19, 2011

Unix doesn't support spaces in filenames

I was creating a job on my Linux-based Jenkins server today, when I encountered an error with a Python virtual environment. It turned out the problem was that the Jenkins job had a name with a space in it, which means the workspace had a name with a space in it, which means the virtualenv had a name with a space in it, which means the scripts created in that virtualenv had a hash-bang with space in it... and bash doesn't support hash-bangs with spaces in the interpreter path.

Consider this simple example. There appears to be no specification for the hashbang, but after some searching, I did find a page that indicates there is no supported mechanism for specifying a space in the interpreter name.

That's a shame. That limitation then bleeds through to almost every other aspect in Unix programming that it basically means Unix does not support spaces in filenames.