Over the last year I’ve updated my Advanced Python course to be based on a series of modules that can more easily be adapted to the specific needs of any team or group of delegates. There’s a lot more advanced material and the exercises have also been updated. A good engineer’s Python in three days.

• https://agileabstractions.com/advanced/

Typically taught as a three day hands on course that will take you deeper into the Python programming language and ecosystem. This course will take delegates from beginner/intermediate level in Python to Advanced Python experts. The course provides a solid overview of the Python language including some low level details essential to working confidently and fluidly with Python. The focus is on practical programming and the skills learned here can be applied in any field where Python is used.

This course is taught by Michael Foord. Michael has been teaching Python for over a decade and has over twenty years industry experience as an application developer. Michael is a Python core developer and the creator of unittest.mock in the Python standard library, and is the author of “The Absolute Minimum Every Python Web Application Developer Must Know About Security”.

In this course delegates will learn a great deal of Python, from an essential foundation like how assignment works to taking advantage of multicore systems with multiprocessing. Included is networking, from API clients to understanding sockets and how servers work - the request response cycle of the REST API model, language features like closures, generators, context managers and the whole Python object model along with testing with pytest.

For smart programmers this course provides a solid foundation for working with Python along with many advanced language features and concepts and powerful libraries for tackling many common programming scenarios. As well as learning and discussion every section is backed by lab exercises.

Full list of modules available:

• Exceptions and the Call Stack
• Closures and Decorators
• Concurrency
• Context Managers and __init_subclass__
• Iterators, Generators, References and Assignment
• Imports, Modules and Namespaces
• Networking
• The Core Python Object Model
• Testing with pytest
• Floats, Unicode, and Regular Expressions
• List Comprehensions, Generator Expressions & Function Signatures
• Advanced Python OOP

An additional optional module “Data Science Overview” is available on request.

For several of the modules there are previews of the material in a series of “Python Knowledge Share Videos”:

• https://agileabstractions.com/python-knowledge-share-videos/

The materials/slides for these sessions can be found here:

• https://github.com/voidspace/talks

A new one hour Knowledge Share video is online on YouTube.

• List Comprehensions, Generator Expressions & Function Signatures

This session is on working with data using enumerate, zip plus list comprehensions and generator expressions. Along the way we’ll talk about tuple unpacking, set and dictionary comprehensions and even nested list comprehensions.

We finish off with a look at function signatures: required and optional arguments plus positional and keyword arguments and positional only and keyword only arguments.

The slides/presentation can be downloaded here:

• https://github.com/voidspace/talks/tree/main/listcomprehensions

You can see all the sessions, eight so far, on a range of Python topics from generators to concurrency to testing with pytest on YouTube:

• Python Knowledge Share Videos on YouTube

The Open Source Initiative (OST) is the authority that defines Open Source. They’ve published my guide to security for Python web applications as a three part series. *“The Absolute Minimum Every Python Web Application Developer Must Know About Security”.

Never store passwords in plain text. At a minimum you should be storing salted hashed passwords (which is probably what went wrong in the image here), even better use a key derivation function, but best of all delegate password management to a modern framework and keep it up to date. And never implement your own cryptography algorithms.

Data should be encrypted in transit and at rest. Just stealing your database shouldn’t reveal sensitive information because data is encrypted at rest. You can do this with the StringEncryptedType for SQLAlchemy for example. mTLS (Mutual TLS) is how we can keep data encrypted in transit, even on our internal networks. Just getting access to your network shouldn’t reveal any sensitive information because data is encrypted in transit.

These are just a few of the principles from a Defence in Depth, multi-layered, approach to security explored in this article series:

• Essential Python Web Security

  The Defence in Depth approach, important security principles, The OWASP Top Ten, the CVE warning system, authentication controls and security tooling.

• Security and cryptography algorithms: A guide

  Cryptography algorithms, using the cryptography library and the Python standard library. Hashing, encryption, key exchange protocols and public/private key signature algorithms with their use cases.

• TLS and Networking

  Much of our security is network security and much of that comes from TLS. We now understand all the crypto algorithms that make up TLS as a security system and can understand when, how and why to use TLS. We’ll also look at the request->response cycle of HTTP, the abstraction layer developers work at, and the security issues around networking. From firewalls to cookies via LANs and sockets.

(Being able to talk about the Blowfish Cipher and the Diffie-Hellmann Elliptic Curve Key Exchange Protocol is cyberpunk, so these articles are teaching real life skills.)

Thanks to Dr David Cassandra Mertz and PyDanny (Daniel Greenfeld) for improving these articles through technical review and to Nick Vidal (and volunteers) from the OSI for their work editing and formatting the articles.

The Open Source Initiative have published part one of an article of mine. The article is called “Essential Python Web Security” and it’s part one of a series called “The Absolute Minimum Every Python Web Application Developer Must Know About Security”. The subject is Full Stack Security for Python web applications, based on the Defence in Depth approach.

This series explores the critical security principles every Python web developer should know. Whilst hard and fast rules, like avoiding plaintext passwords and custom security algorithms, are essential - a deeper understanding of broader security principles is equally important. This first pots delves into fundamental security best practises, ranging from general principles to specific Python-related techniques.

• Essential Python Web Security: Part 1

Part 2, on Cryptographic Algorithms, will be published soon. When the series is complete it will probably also be available as an ebook. The full document, about fifty pages, can be read here:

• The Absolute Minimum Every Python Web Application Developer Must Know About Security

Special thanks to Gigaclear Ltd who sponsored the creation of this article. Also thanks to Dr David Mertz and Daniel Roy Greenfeld for technical reviews of this article prior to publication.

This two day course covers Full Stack Security with the Defence in Depth approach. It covers important security principles, and mitigating specific vulnerabilities including The OWASP Top Ten, but is focused on secure Python web application development.

• Secure Python Web Application Development

Course Contents

This is a practical and hands on, two day, course. Learn how to use the security tools that come in the Python standard library. Modules covered include:

• hashlib
• hmac
• secrets
• random
• socket
• ssl

Web application frameworks:

• Security features in web application frameworks for API servers and web applications
• How to secure data in Django, Flask and other popular web application frameworks
• Secure deployment practises with containers and application servers (WSGI or ASGI)

Third party libraries for cryptography and secure network access:

• authlib
• cryptography
• httpx and requests
• websocket
• jwt
• OpenZiti for application level zero trust architecture
• certifi for TLS certification verification

Tooling for secure Python development and as part of your CI pipelines:

• uv/pipenv
• pip-audit
• bandit
• ruff
• mypy
• dependabot/renovate
• Security testing

Network security with TLS:

• How, why and when to use TLS
• How TLS works
• mTLS for enhanced security
• Generating self-signed certificates for local development, with the cryptography library

Michael Foord has been a Python application developer for over twenty years, is a Python core developer and the creator of unittest.mock in the Python standard library, and is the author of The Absolute Minimum Every Python Web Application Developer Must Know About Security.

Topics Covered:

Core Python Security Fundamentals

• Security principles and defence in depth strategies with Python frameworks
• Implementing OWASP Top 10 protections in Django, Flask and FastAPI applications
• Security principles and defence in depth strategies
• Principles from The OWASP Web Security Testing Guide
• Threat modelling and the security requirements document
• The principles of least privilege and deny by default
• Zero trust architecture fundamentals
• Building zero trust architecture with OpenZiti’s Python SDK

Cryptography and Data Security

• Hashing, encryption, and digital signatures
• Symmetric encryption and public key encryption
• Secure password storage and management
• Using Python’s hashlib and hmac modules for secure hashing
• Using Python’s cryptography libraries correctly
• Data encryption at rest and in transit
• Data encryption at rest using Django’s encrypted model fields and SQLAlchemy StringEncryptedType

Authentication and Authorization

• Secure session management
• OAuth 2.0 and JWT for authentication
• Oauth2 with the Python library authlib
• JWT handling with PyJWT and managing token lifecycles
• Role-based access control (RBAC), plus alternatives
• Multi-factor authentication
• Managing access tokens and permissions

Secure Coding Practices

• Proven security with modern cryptography algorithms
• Protection against SQL injection
• Input validation and sanitisation
• Using secrets vs random for cryptographic operations
• Sanitizing logs in Python applications
• Django template escaping and Flask/Jinja2 for XSS prevention
• Preventing timing attacks and token prediction attacks
• Cryptographically secure randomness
• Cross-site scripting (XSS) prevention
• Cross-site request forgery (CSRF) protection
• Secure file handling and upload validation
• Preventing TLS downgrade attacks

Network Security

• Networking fundamentals
• TLS/SSL implementation and certificate management
• Secure API design and implementation
• WebSocket security
• Network architecture and segmentation
• Firewalls, routers, network interfaces
• Protocols, HTTP & TLS, with the Python standard library
• Application deployment
• Software Defined Networking

Infrastructure Security

• Container security best practices
• Secure deployment patterns
• Network interfaces and routing
• Building DMZ architectures for Python web applications
• Virtual private networks (VPN)

Security Tools and Testing

• Static analysis with bandit and ruff
• Dependency scanning using pip-audit
• Automated security testing integration
• Container scanning and runtime protection
• Code review practices for security

Operational Security

• Live security alerts
• Statutory duties around security
• Monitoring Python applications for security issues
• Security patch management for Python applications
• Updating and patching strategies

The course includes practical exercises throughout, with participants implementing secure coding patterns, identifying vulnerabilities in sample code, and building secure components.

A practical two day course on the object oriented features of Python. Perfect for programmers with some experience of Python looking to use objects and classes and to understand them. An excellent course for data scientists, devops engineers and those self taught with Python looking to move beyond scripting into programming.

• Object Oriented Programming Theory with Python

Course Contents

Fundamentals: Classes and methods

• Computer architecture and programming languages
• Python as a high-level, object-oriented language
• Objects as abstractions, for thinking
• The class statement
• The explicit self
• The initialiser method __init__
• Bound methods
• Attributes and the built-in attribute access functions
• References and assignment (how Python works)
• Mutable objects (and call by object)
• Object copying

Object Oriented Features

• Class attributes
• Class methods
• Properties
• Private attributes
• Single inheritance
• Inheritance to extend and modify the parent
• The use of super
• Cooperative multiple inheritance
• Mixin Classes
• Attribute lookup and the method resolution order
• The type system: isinstance and issubclass

Inside Python Objects

• The instance dictionary
• Slots
• Class dictionaries
• The descriptor protocol

Python Protocols

• Magic methods and Python protocols
• Operator overloading
• The string representation protocol
• The container protocols
• Implementing custom containers
• Inheriting from collections.abc.MutableSequence

Alternative Approaches

• namedtuples
• dataclasses
• type as a class factory

Object oriented theory:

• History of Object Oriented Programming
• The pillars of OOP
• Abstraction
• Inheritance
• Encapsulation
• Polymorphism
• Object oriented design principles
• Design patterns
• The Liskov Substitution Principle
• Composition versus inheritance

Optional advanced section (third day):

1. Advanced OOP Concepts

   • Interfaces and API design
   • Abstract base classes and protocols
   • Type hints and static typing with mypy
   • Class decorators
   • Decoration via inheritance with __init_sublass__
   • Metaclasses
   • Context managers and the with statement
   • Weak references and destructors
   • The descriptor protocol

2. OOP Design Principles

   • SOLID principles
   • Law of Demeter
   • Liskov Substitution Principle
   • Composition vs inheritance
   • Domain Driven Design
   • Test Driven Development
   • Stop writing classes
   • The hexagon pattern (microservices)
   • The C4 Model for system architecture

📘 Written in 2021.

IronPython in Action was published on the 7th April 2009 and we sold a little over 7000 copies.

Royalties for last quarter amounted to $25.

It took me two years to write thirteen chapters and a couple of appendices, and took Christian Muirhead about the same to write two chapters and an appendix. Jonathan Hartley did the diagrams and illustrations and the worst part was compiling the index.

It took so long because IronPython was still in alpha (!) when we started and it changed several times (including a Silverlight version being released) whilst writing!

After leaving Resolver Systems in 2010 I spent a year contracting on Line of Business apps that ran in Silverlight (Django on the server): Python code running in the browser on the client side. It was glorious.

We even had functional tests on unittest built in to the app.

Work on mock accelerated massively once IronPython in Action was complete. MagickMock was born not long afterwards.

I was also helping maintain the python.org website and adding test discovery to unittest at the time, and speaking at every conference I could find.

It felt like the glory days of the Python community. It’s almost time for PyCon (online) and I’m nostalgic once again.

My first PyCon, the second Dallas PyCon and my first time in the US, there were about 600 attendees. You could almost know everyone.

I shaved my beard to enter Dallas and wore my hair in a pony tail. All I knew was they didn’t like hippies there. It was the nicest greeting at a US airport I’ve ever had.

I went on a road trip with Andrzej Krzywda afterwards trying to find mountains. We found the Ouchita mountains in Oaklahoma and drove back through Arkansas to visit friends of mine in Houston. Along the peaks of the mountains, which are hills really, we found a view called Dead Man’s Vista and we I laughed together at Microsoft.

Not long after this the web explosion happened and Django happened, google adopted Python as an official language and the community started to explode and grow.

That was even before Python became huge as a teaching language and before Python exploded in data science too.

I once paired with Jacob Kaplan Moss at a PyCon sprint and fixed some issue by adding a metaclass to the Django codebase. Which he never committed and found a better way.

That’s the closest I’ve come to deploying a metaclass I think, although I’ve removed a few in my time.

I knew Python had “made it” as a language when one bag stuffing pre-PyCon I met someone who didn’t want to be there. He’d been sent by work. Before that Python was obscure, and only people who really loved it went to PyCon. Which I’m convinced is the secret of Python’s success.

It was built by passion not by money. For the sheer love and the joy of building something beautiful with other people.

I was a Mac user then and had a running joke with Jonathan Hartley about Linux and projectors.

One time he plugged his laptop into the projector prior to his PyCon talk (Testing is a Silver Bullet), tried to fix the x-config from the terminal and rendered his laptop unusable. He did the presentation on mine. The next year Mark Shuttleworth did a keynote talk at PyCon and running some bleeding edge version of Ubuntu also couldn’t plug it into the projector system. Hilarity on my part.

The biggest conference I ever spoke at was a Microsoft one in Brighton where they demoed Silverlight and I demoed IronPython on Silverlight. They didn’t tell me I would be on main stage in front of a few thousand Microsoft devs. I was used to talking to a few hundred at a time!

I had a slide deck built from S5 with reStructured Text markup and a Far Side slide mocking static typing. Which went down a bomb to an audience of C# devs. I still managed, by coincidence, to demo almost the same features of Silverlight as Microsoft bigwig Scott Hanselman who did the keynote.

It was an “interesting experience”, evangelising Python and dynamic languages in “the heart of the beast” as it were. Microsoft went on to step up their involvement with Python and sincere Open Source commitments which they’ve maintained since.

Since I first wrote this Python has finally made it, ranked as the most widely used programming language in the world by TIOBE and PyPL. World number one.

I joined Twitter fourteen years ago and have tweeted over fifty-two thousand times. I follow 1,636 accounts, which is too many, and have 8,670 followers. I use Tweetdeck which is run by Twitter and doesn’t show ads or promoted tweets or mess with tweet order and it lets me use two different accounts.

I use twitter a lot less than I did during my social media and community frenzy whilst I delighted to learn Python, but I still enjoy it.

During that time (2006-2011) I “drank from the firehose”. I read all of slashdot (scanned every headline and read relevant articles), read all of comp.lang.python (every message title - read and replied to many), read all of python-dev (similarly) and all of testing-in-python, blogged almost daily and worked full time as a software engineer commuting to London four times a week and developed mock in my spare time and worked on unittest in the Python standard library. And wrote a book and worked part time doing community liaison and service development for a local charity working with the homeless and disadvantaged. I was Microsoft MVP for three years for my work with IronPython, I spoke at countless conferences and received the Python Software Foundation Community Award for my work running Planet Python and helping out with the Python.org website and mailing infrastructure.

Then in 2011 my first child was born and I started working for Canonical. Three years of large Django web applications then three years of Go and MongoDB and then a year with Red Hat testing Ansible Tower and now four years self employed.

During that time I remembered that the primary drive in my life was spiritual and I started meditating again. One hour a day of mindfulness of breathing. That transformed my life all over again.

I once rode in the back of a beaten up station wagon owned and operated by the creator of the Python programming language whilst sat alongside the creator of Bitorrent, which was written in Python.

I also once had a pub lunch in Oxford with the creator of the Erlang programming language and the creator of the Haskell programming language. We were all three speaking at the ACCU conference. I was speaking on IronPython.

It’s been a fun journey.

Python metaclasses, considered advanced programming and Python “black magick” (*) explained in eight words:

The type of a class is a class.

Here’s what knowledge of Object Oriented theory and type systems permit you to deduce from this:

Using the word “class”, instead of “the type of a class is type” or even “the type of a class is a type, classes are types”, implies that a user defined class can be a metaclass. This is indeed the case, and the point of metaclasses in Python.

The type is responsible for creating new instances. So if the type of a class is a class then we can write classes that create classes. Indeed this is the primary usecase for metaclasses.

(Deeper knowledge of Python, and the two phase object creation protocol, may lead you to deduce that this is done by overriding the __new__ method. If you’re familiar with “type” as a class factory you can probably even guess the signature and that you must inherit from type.)

If the type of a class is a class then the type system will permit a type check for the class against its class. And indeed isinstance(klass, metaclass) returns true.

(And deeper knowledge of Python will tell you that the magic methods, the protocol methods, are always looked up on the type. So we can implement behaviour for class objects by providing magic methods on the metaclass.)

All of this implies that classes are themselves objects. Which is true in Python for everything is an object in Python (and everything is a reference).

And so on…

• Type and class are synonyms in Python.
• type(type) is type

And to further round out the type system, these are also Python invariants:

• isinstance(object, object) is True # object is an object
• isinstance(object, type) is True # but also a type
• isinstance(type, object) is True # type is an object
• isinstance(type, type) is True # but also a type

(*) Like all black magick it is useful for understanding the world but never for actual use. Well, except perhaps in very rare circumstances if you know what you’re doing.

I’ve seen this meme a bunch of times recently. I always reply; what is asserted without evidence may be dismissed without consideration.

Current Gen AI is flawed by hallucination issues, mired in copyright controversy, expensive to run and lacking clear use cases. (Although it’s pretty good at code generation). It’s a massive hype train.

Gen AI, as it is now, was made possible by the invention of “Transformer Architecture” by Google in 2017. We’re seeing fast paced change and development, but all built on that technology.

At some point another quantum breakthrough will change things all over again - and make another step towards AGI. Although it will take several such steps, and order of magnitudes larger models (and multi models), to create anything resembling true AI.

So a huge number of disparate individuals, institutions, governments and companies are pursuing the development of AI. There’s no single cohesive agenda behind it. As new technologies arise we adapt to them, find uses for them, and everyone pursues their agendas with them.

Not particularly special to AI I don’t think.

I’ve been a fan of Agile ever since my first programming job with Resolver Systems back in 2006. I had taught myself programming and there I really learned engineering, how to build software products whilst caring about quality. I became passionate about testing as a way to ensure a minimal level of quality and about agile processes which are able to change quickly.

My only experience of Scrum was for a year at a heavily waterfall shop which layered Scrum for project management on top of heavily waterfall software development processes. It wasn’t a fun experience but I still learned a great deal.

I’ve been working with Gigaclear, as team lead on backend API servers, including One Touch Switch, for about a year now. I enjoy our software development practises and processes; we use a combination of agile for software development, devops (devsecops of course) for software deployment and maintenance, and Scrum for project management. It’s a very effective combination.

I recently attended a course with the Agile Alliance, led by John McFadyen, and became certified as a Scrummaster.

The course was fantastic and very inspiring. At Gigaclear we’re systematically evaluating all our systems, systems architecture and processes. This process and the course have been hugely inspirational and I have an article on software development processes coming shortly…

I’ve been teaching Python in one hour knowledge sharing sessions, some of which I’ve put online on youtube.

This is the link to the playlist of the sessions:

• Python Knowledge Share Videos on YouTube

The slides for each of the sessions, along with some example code, can be found in this github repository:

• https://github.com/voidspace/talks

So far there are eight one-hour sessions (with more planned) on:

• Python Core Object Model
  • Python objects
  • Slots
  • Attribute lookup and the MRO
  • Inheritance, multiple inheritance and super
  • Inside Python objects and classes
• Closures and decorators (functional programming)
  • Functional programming: higher order functions and functions as objects
  • Lambdas
  • Closures: functions that build functions
  • Variable scoping: global, local and nonlocal
  • Decorators: functions wrapping functions
  • Decorator factories (decorators that take arguments)
  • Class decorators
  • Decorator order and using functools.wraps
• Generators and Iterators
  • The iteration protocol
  • Stateful iteration with generators
  • Adding iteration support to objects
  • References, assignment and mutability
  • Identity versus equality
  • Call by object
  • Object copying
• Unicode, Floats and regex
  • Floating point numbers
  • Unicode, encodings and strings
  • Regular expressions
• Concurrency (async, threads, processes, the GIL)
  • The history of concurrency from AmigaOS to a multi-core world
  • Python and the Global Interpreter Lock
  • I/O bound and CPU bound tasks
  • Threads and processes
  • Async programming (green threading, coroutines)
  • Concurrency with threads
  • Concurrency with multiprocessing
  • Looking to the future (Python 3.13): optional GIL (PEP 703) and subinterpreters (PEP 554)
• Testing with pytest
  • virtual environments and pipenv (installing pytest)
  • pytest command line for collecting and running tests
  • Simple test functions and asserts
  • Test fixtures and conftest.py
  • Testing exceptions
  • Test parameterisation for test combinations
  • Test marking for running test subsets
  • Principles of testing (unit tests versus end to end testing, building test helpers etc)
  • Mocking and patching
• Modules and Namespaces
  • Import syntax variations
  • namespaces and variable lookups
  • sys.modules and the import cache
  • Module objects
  • Module level functionality: __dir__ and __getattr__
  • Packages and the filesystem
  • Relative import syntax
  • Module reloading (how to do it and why not to do it)
  • Circular imports, avoiding and fixing
  • Executable modules and packages
• List Comprehensions, Generator Expressions & Function Signatures
  • zip and enumerate, builtin functions
  • Tuple unpacking, iterating over pairs
  • List comprehensions
  • Generator Expressions
  • Set and dictionary comprehensions
  • Nested comprehensions
  • Function signatures: required and optional arguments
  • Positional and keyword arguments
  • Positional only and keyword only arguments

Other Talks

A selection of some of the talks and interviews I’ve given on Python and software engineering across my career.

• TLS and Networking (article, Jan 2025)
• Security and cryptography algorithms: A guide (article, Jan 2025)
• Essential Python Web Security (article, Dec 2024)
• UK Health Security Agency Software Development Practise Conf 2024
• PyCon UK 2023, Metaclasses in 5 Minutes Lightning Talk
• PyCon MEA 2022 How Python Took Over the World
• Test and Code Podcast Episode 145: For Those About to Mock
• PyCon Belarus 2020 How Python Took Over the World
• PyLondinium 2019 The Python Object Model
• Interview on Podcast.__init__ on testing, Mock and the Python community (2018)
• The Role of Abstractions: Lightning Talk PyCon US 2018
• Best Practises for Software Development and Testing (2017)
• PyCon UK Panel 2015
• To the Clouds: EuroPython 2015
• Automated Deployments with Juju: PyCon UK 2014
• Python and Pythons: PyCon NZ 2013
• Testing with Mock: PyCon US 2011
• A Little Bit of Python Podcast (2010-11)
• New and Improved unittest 2: PyCon US 2010
• Michael Foord on IronPython: Hanselminutes 2009
• IronPython in Action (2009, book)
• Michael Foord on IronPython: TechEd 2007

My first blog post in a few years! I have some articles I’d like to publish, and some adventures to share, so I thought it was time to fire up a blog engine again.

My nine year old son, Benjamin, is really into programming with Scratch and he’s keen to play with electronics and learn MicroPython. Which is awesome because there’s almost nothing I would love to do more with him.

MicroPython is an extremely impressive implementation of Python that will run on embedded devices and microcontrollers, as well as bigger tiny computers like the Raspberry Pi.

I’ve dug out an old MicroBit I had, purchased a Raspberry Pi Pico board/kit and also a ZumoBot 2040 robot which uses the same microcontroller as the Pico, to play with.

I’m now starting to get to grips with the basics, using the Thonny IDE.

I have a bunch of Neopixel LEDs, including a long light strip, I’d like to wire up in my living room controlled by a Pico board and an Android App using Kivy. That’s my goal number 1.

I’d like to program the ZumoBot to explore and map my flat. Goal 2.

Meanwhile Benjamin is enjoying playing with electronics (switches, LEDs, potentiometer and now a motor) with the Pico and on his own he’s programming the MicroBit with Scratch (or at least “blocks” which is the Microsoft equivalent). I’ve also done my first soldering in over a decade.

I have a github repository to track my tinkering, but I’d like to write up some recipes and post them on this blog as I go. (The biggest hurdle is I can’t easily create circuit diagrams. Time to explore.)

The github repository and ZumoBot links:

• https://github.com/voidspace/zumobot
• https://thepihut.com/products/zumo-2040-robot-assembled-with-75-1-hp-motors
• https://www.pololu.com/product/5012
• https://github.com/pololu/zumo-2040-robot

For the last year I’ve been working as a team lead for backend API development with Gigaclear a UK rural ISP who own and run fibre internet to rural communities across the UK. This is alongside my training work.

This image shows the main project I’ve been working on since joining Gigaclear, One Touch Switch. A regulatory requirement for all ISPs to allow automated switching between ISPs. When you sign up with a new internet provider your account is automatically ceased with the old provider and VOIP numbers can be automatically ported.

Our OTS project is just part of the Gigaclear One Touch Switch system which interfaces with Salesforce and Netadmin and the website order flow (the Online Buying Journey) and represents an impressive engineering effort. We were one of the first ISPs with a system ready to take part in industry trials a few months ago, both OTS and our underlying systems passed pen testing with flying colours, and the switch on has been smooth.

Something I’m proud to have been part of. My current project is preparing security awareness training materials based on OWASP for our various engineering departments whilst we also undertake a systematic review of all of our systems and processes.

In the diagram I’m team lead for the Sphinx engineering team.

One of the contracts I’ve been working on recently is working with Gurock building a test automation system for a PHP application, their test management app TestRail. As well as building the instrastructure for the application testing and the API testing I’ve once again been involved in the nitty-gritty of testing a web application with Selenium and all the fun that involved.

And actually it has been fun. We’ve had a bunch of issues to overcome and despite the usual pain and trauma and running round in circles we seem to have overcome most of them and have a test suite that is robust against the three different platforms we’re testing against.

For those who don’t know Selenium WebDriver interface allows you to connect your Python tests, or just about any language you care to choose, and test you web application as a “black box” - interacting with it in the same way as your users do. These are “functional tests”, as opposed to unit tests”, that tests the whole application as a whole meets its specifications and rquirements. As will all testing you can’t guarantee that it makes your applcation bug-free, but you can eleminiate whole classes of bugs and gurarantee a minimum level of application quality.

This application is written in PHP, but we’re using Python, py.test and Selenium to automate the tests and the front end is built with jQuery. There are various fun aspects of testing this app that we’ve encountered. A couple of these stem from the fact that like any modern any web application much of the UI updates are done from AJAX calls. This means that there’s no global page state ready event to wait for to know that load has finished and the page is ready to interact with.

One of the plugins in use is the BlockUI plugin. This puts a semi-opaque overlay over the user interface in the browser whilst asynchronous AJAX requests are being made, to prevent other elements of the user interface being interacted with. As the request is an asynchronous one the browser isn’t blocked so our Selenium tests don’t know that the user interface is blocked and it should wait before attempting any more interactions. This causes tests to fail with the dreaded error:

Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: Element <input type="button" class="btn btn-default" data-toggle="modal" data-target="#adduser" data-localize="adduser" value="Add user"> is not clickable at point (1397, 97). Other element would receive the click: <div class="blockUI blockOverlay" style="z-index: 1000; border: none; margin: 0px; padding: 0px; width: 100%; height: 100%; top: 0px; left: 0px; background-color: rgb(0, 0, 0); cursor: wait; position: absolute; opacity: 0.304712;"></div>

The dreaded part is specifically is not clickable at point (1397, 97). Other element would receive the click: <div class="blockUI blockOverlay".

The “blockUI” element is intercepting the click because the AJAX request is not completed, or more to the point a blockUI element is intercepting it. The normal way round this would be to find the “blockU” element and wait for it to no longer be displayed. Unfortunately there’s more than one of them! So this is the code we came up with to wait until none of them are displayed:

from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.common.by import By

class GeneralLocators:

    blockUI = (By.CLASS_NAME, "blockUI")
    busy = (By.CLASS_NAME, "busy")


def any_elements_displayed(elements):
    for element in elements:
        try:
            if element.is_displayed():
                return True
        except StaleElementReferenceException:
            pass
    return False

class BasePageElement(object):

    def wait_for_blockui_to_close(self, seconds=5):
        self.driver.implicitly_wait(0)
        try:
            stop = time.time() + seconds
            while time.time() < stop:
                blockUIs = self.driver.find_elements(*GeneralLocators.blockUI)
                if not any_elements_displayed(blockUIs):
                    return
                time.sleep(0.1)
            raise TimeoutException("Timed out waiting for blockUI to go away")
        finally:
            self.driver.implicitly_wait(10)

We have a similar problem with AJAX elements that don’t block the page, but take several seconds to update, showing a busy indiciator whilst they’re updating. Again, we need to wait for the busy indicators to complete before we ineract with any of the elements. Thanksfully that is similarly easy. Note that we set the global implicitly_wait timeout to zero whilst we’re checking.

    def wait_until_not_busy(self, seconds=5):
        self.driver.implicitly_wait(0)
        try:
            stop = time.time() + seconds
            while time.time() < stop:
                busy = self.driver.find_elements(*GeneralLocators.busy)
                if not any_elements_displayed(busy):
                    return
                time.sleep(0.1)
            raise TimeoutException("Timed out waiting to not be busy")
        finally:
            self.driver.implicitly_wait(10)

It’s well worth noting that with the selenium library in Python, the implicitly_wait value is a global. Setting it anywhere sets it for the rest of the session.

We put all the element locators into classes, like GeneralLocators so that as locators change (inevitable in an evolving user interface) there is only one place to change the locators rather than having them scattered through out our code.

Here’s a few more tricks and trips we’ve discovered along the way. Whilst text boxes have a nice and straightforward .clear() method to clear existing text in them, this dioesn’;t work with a textarea (which confusingly enough has a .clear() method which apppears to do nothing. The right way to clear to a text box is to send a CTRL-A followed by a backspace:

        # CTRL-A plus BACKSPACE are needed for selenium to clear the textarea as .clear() doesn't work.
        self.send_keys_to_element(CustomizationsLocators.add_custom_field_description, Keys.CONTROL + "a")
        self.send_keys_to_element(CustomizationsLocators.add_custom_field_description, Keys.BACKSPACE)

If you want to provide a command line option to run the tests with a headless browser, this little function (firefox only) will do the trick. You could further customize is to switch between browers:

import pytest
from selenium import webdriver
from selenium.webdriver.firefox.options import Options

	def get_driver():
	    options = Options()
	    if pytest.config.getoption('headless'):
		options.headless = True
	    return webdriver.Firefox(options=options)

And these final two are interesting. Uploading files with Selenium. Because the file upload dialog is a native dialog it’s very hard to interact with Selenium (impossible I thin.). However it does come along with a hidden input field that you can enter file paths directly to. So for a normal file dialog this works fine:

    from selenium.webdriver.common.by import By
    file_inputs = driver.find_elements(By.CSS_SELECTOR, 'input.dz-hidden-input')
    input_element = file_inputs[input_index]
    driver.execute_script('arguments[0].style = ""; arguments[0].style.display = "block"; arguments[0].style.visibility = "visible";', input_element)
    time.sleep(0.1)
    input_element.send_keys(filename)

So long as you know, or work out by trial and error, which file input dialog to send the input to it will work fine. The useful thing is that it exposes all the hidden file inputs in the user interace so you can see what you’re interacting with.

This still unfortunately doesn’t work for file uploads by dropzone, some kind of javascript extension. For this you need to base64 encode the file yourself and attach it to the dropzone. Made all the more interesting by the fact that the driver.execute_script api will only take a single line of input. Still, it works!! As horrible as it is, this works!! It takes the base64 encoded version of the file and attaches it to the dropzone element as a blob, with the filename attached as metadata.

    def add_dropzone_attachment(self, locator, attachment_path):
        filename = os.path.basename(attachment_path)
        with open(attachment_path, 'rb') as f:
            content = f.read()
        content = base64.b64encode(content).decode('ascii')

        script = (
            "var myZone, blob, base64Image; myZone = Dropzone.forElement('{}');"
            "base64content = '{}';"
            "function base64toBlob(r,e,n)\{\{e=e||\"\",n=n||512;for(var t=atob(r),a=[],o=0;o<t.length;o+=n)\{\{for(var l=t.slice(o,o+n),h=new Array(l.length),b=0;b<l.length;b++)h[b]=l.charCodeAt(b);var v=new Uint8Array(h);a.push(v)}}var c=new Blob(a,\{\{type\:e\}\s});return c}}"
            "blob = base64toBlob(base64content, 'image/png');"
            "blob.name = '{}';"
            "myZone.addFile(blob);"
        ).format(locator, content, filename) 
        self.driver.execute_script(script)

The locator is the locator of the dropzone area itself, usually something like #attachmentDropzone.

Hopefully all this painfully won information proves useful to someone!

I recently had the great pleasure of having a conversation with Tobias Macey, who produces the Podcast.init podcast. We spent almost an hour talking about testing, the Python community and the mock library amongst other things.

The podcast is now available to listen to:

• Michael Foord On Testing, Mock, TDD, And The Python Community - Episode 171

  Michael Foord has been working on building and testing software in Python for over a decade. One of his most notable and widely used contributions to the community is the Mock library, which has been incorporated into the standard library. In this episode he explains how he got involved in the community, why testing has been such a strong focus throughout his career, the uses and hazards of mocked objects, and how he is transitioning to freelancing full time.

A selection of the questions Tobias asked include:

• How did you get introduced to Python?
• One of the main threads in your career appears to be software testing. What aspects of testing do you find so interesting and how did you first get exposed to that aspect of building software?
  • How has the language and ecosystem support for testing evolved over the course of your career?
  • What are some of the areas that you find it to still be lacking?
• Mock is one of your projects that has been widely adopted and ultimately incorporated into the standard library. What was your reason for starting it in the first place?
  • Mocking can be a controversial topic. What are your current thoughts on how and when to use mocks, stubs, and fixtures?
• How do you view the state of the art for testing in Python as it compares to other languages that you have worked in?
• You were fairly early in the move to supporting Python 2 and 3 in a single project with Mock. How has that overall experience changed in the intervening years since Python 2.4 and 3.2?
• What are some of the notable evolutions in Python and the software industry that you have experienced over your career?
• You recently transitioned to acting as a software trainer and consultant full time. Where are you focusing your energy currently and what are your grand plans for the future?

The photo shows some of the Resolver Systems crew enjoying a meal together at the 2009 EuroPython in Birmingham.

I love the word rigour. It conveys either, or both, strict discipline or something that was really hard work.

I’ve found the rigorous application of theoretical principles a really useful way of learning those principles. Learning what they really mean, and what those principles are good at achieving and what they’re not good at achieving.

I’ve been rigorous in my discipline in meditating. I’ve meditated for an hour a day, generally six days a week, for a number of years now. A dedicated and regular practise of focus and letting go of distractions, which is the substance of mindfulness practise, has made a difference in my life and my understanding of myself.

My trade is as a software engineer, a computer programmer. I taught myself to program by becoming really passionate about it. What you love you learn. I learned the art and craft of engineering in my first professional job, at a small startup in London called Resolver Systems.

There, for the four years I worked there, we rigorously applied the principles of Extreme Programming, a strict variant and really the progenitor of the “agile” movement. The goal is engineering processes that make the development process agile and fluid, able to change direction quickly and able to check that it is continuously delivering value in the work being done, whilst also creating the software in ways that ensure as much as is possible you are creating a quality and useful product.

This includes full Test Driven Design (often now called “test first” in a great misunderstanding of the value of TDD), with a full test coverage from unit to functional (full stack). We had about three to four times more test code than production code. We built a beautiful thing.

It also included full pair programming. So for four years we worked together and thought together and learned together. The product failed, unfortunately, an idea ahead of its time. With Python finally being added to Excel as a scripting language it’s possible that the idea of applying proper engineering principles to the creation of complex spreadsheets may have its day after all.

Contrary to perhaps what we thought in the honeymoon phase of learning “agile” it isn’t a magic silver bullet for curing all ills of software engineering. However, agile processes do stand in stark contrast to the traditional “waterfall” model of software enginering. A strongly waterfall process is very hard to change, which is precisely why they’re such a bad idea. Perhaps we can summarise the essence of agile as “finding engineering processes and practises that are able to evolve and suit the specific needs of the team, product and customers”. Being able to change matters, being stuck is awful.

This post originally appeared on my personal blog A Love Letter to Agile on Unpolished Musings.

This is a video and text of a lightning talk, a five minute presentation, given at PyCon US 2018 in Cleveland. The image is an abstract representation of a concrete apple.

• The Role of Abstractions Lightning Talk

This is an abstract talk. There isn’t time to give examples but I hope that the application to the day to day challenges of the practise of software engineering is clear. The only theory worth a damn is the theory of the practise. This is a talk about the role of abstractions in software engineering.

Programming is all about the use of abstractions. We often say that the fundamental language spoken by the machine is ones and zeros. Binary. This isn’t true. Ones and zeroes are an abstract representation of the fundamental operation of computers. It’s a way of representing what central processors do in a way that can be understood by people.

The actual language spoken by computers is the electromagnetic dance across wires and etched silicon, choreographed by the beating of a quartz crystal at the heart of the machine.

Ones and zeroes are a representation of that dance, understandable by humans in order for us to reason about the behaviour of the system.

That’s a very low level abstraction. Very close to the actual operation of computers, but very hard to work with. The next step up is assembly language where we use mnemonics, symbolic instructions like JMP for jump, to represent these patterns of ones and zeroes. We can also use human recognisable labels for memory locations instead of numbers and allow the assembler to calculate offsets for us. Much easier.

Next we have languages like C and then right at the very top we have Python where each construct, a print statement for example, may correspond to as many as millions of the lowest level operations.

Computer programming is communication in two directions. Programming provides a language the computer understands, and is able to execute deterministically, whilst also communicating with humans so they can conceptualise the behaviour of the system. A programming language is a set of conceptual tools to facilitate that communication in both directions.

The art and craft of software engineering is taking the conceptual tools that programming languages provide and using them to solve real world problems. This is the difference between science and engineering. Science is the theory, engineering is the application.

In order to be able to do this we have to have an understanding of the problem domain. We conceptualise it. We think about it. Software is easy to understand and maintain when the abstractions you build map well to the problem domain. If the way you think about the problem is close to the way you think about your software then you have to do less mental translation between the problem and your code.

Joel Spolsky talks about the law of leaky abstractions. Any abstraction that maps to lower level operations in the system will leak. At some point something will go wrong and you will only be able to fix it by understanding the lower level operations too.

I’ve heard it said, and it rings true, that a good programmer can hold about ten thousand lines of code in their head. So if your system is less than ten thousand lines of code, even if it’s terrible code, you don’t need to build higher level building blocks to hold it in your head.

An all too common situation is that a system becomes too complex to reason about, so an engineer decides to create abstractions to simplify how they think. So they create black boxes, abstractions, in which to place the complexity. These type of abstractions conceal complexity. So now you don’t have to look at the mess you just made.

You can reason about your system with your abstractions, but in order to understand the actual behaviour (at a lower level) you need to go digging in all that dirt.

Instead of concealing complexity a good abstraction will explain and point you to the lower level operations. Good abstractions simplify and reveal complexity rather than concealing it.

We can also use this kind of reasoning to think about product and system design. What user experience are you providing, what’s the user story? Your users also think about the problem domain using conceptual tools. The closer the abstractions your software presents to your user map to the way they already think about the problem the easier your software will be to use.

And here we come full circle. If the way you build your software maps well to the problem domain then it will be easy to reason about and maintain. If the abstractions you present to the user map well to the problem domain then it will be easier for your users to think within your system and it will be more intuitive to use.

So abstractions matter. They’re the raw stuff of our world.

This post originally appeared on my personal blog Abstractions on Unpolished Musings.

It feels funny to be writing a “Hello World” blog entry in a new technical blog so far into my adventures with Python. January 2005 and my first entry in my Python technical blog, which marked the very early days of me discovering Python and falling in love with it, doesn’t seem so very long ago.

This was only a year or so before I started my first programming job, with a small London startup called Resolver Systems. We were building a Windows desktop spreadsheet application, using IronPython, with Python embedded as the calculation engine for the spreadsheet. I can’t find a blog entry when I first started working with the Resolver team, but in December 2006 I wrote a post Happy Birthday Resolver.

Since then I’ve had many more adventures including working freelance building web applications with Silverlight and IronPython in the browser, web application development with Django for Canonical, Go development working on a devops tool called Juju and working for Red Hat Ansible as a test engineer.

Python and the Python community has been very good to me in providing me with friendship, intellectual stimulation, a passion for engineering and a career. My involvement in the community included running Planet Python for many years, helping maintain the python.org website, becoming a Python core developer and helping maintain unittest whilst writing and contributing mock to the Python standard library plus at various points helping organise and speaking at all of PyCon in the US, EuroPython and PyCon UK. I was organiser and compere of the Python Language Summit from 2010 to 2014 and the Dynamic Languages VM Summit at PyCon 2011. I’ve lost track of the various conferences I’ve spoken about; spanning .NET, Python specific conferences and general programming like the ACCU conference. I’ve keynoted at PyCon India and PyCon New Zealand, probably the greatest privileges of my career so far.

I’m not saying any of this to boast (mostly), many of my contemporaries and those who are newer to the Python and programming communities have found as much of a passion and a sense of belonging in the Python community as I found. It’s fun to reminisce because it’s been such an enjoyable trip and one that’s far from over.

Alongside that, since 2011, I’ve done Python training on behalf of David Beazley. Teaching Python, both the introduction course and the super-advanced Python Mastery course, has been the most fun thing I’ve done professionally with Python. This is one of the reasons I’ve decided it’s time to branch out as a freelance Python programmer, trainer, contractor and consultant.

This blog entry is both a “Hello World” for the blog and for my new venture Agile Abstractions. I’m available for contract work, specialising in the automated end-to-end testing of systems.

The training courses I offer are listed here:

• Python and Testing Training Courses

For custom training packages or any enquiries contact me on michael@python.org.

If you’re at EuroPython in Edinburgh this year, or PyCon UK in Cardiff, then hopefully I’ll see you there!

This website is built with Jekyll using the open source Jekyll Now and hosted on Github Pages. It’s a lovely and simple workflow for geeks to build and host websites that include a blog. It reminds me of the heady days of 2006 and my static site generator rest2web.