Lee Willis

March 27, 2017
by Lee
0 comments

Background processing for WordPress

I’m the maintainer of a reasonably well-used WooCommerce plugin that currently generates a full set of product data from a store This is currently done on-demand to make sure the information is up to date. On smallish stores, everything works just fine. It’s even OK on larger stores as long as your hosting is ‘modern’ (think reasonable PHP versions, execution limits, persistent object cache etc.).

However, some client hosting can’t always manage to pull products, build product objects, apply business logic for output, and construct the output for all products within execution time limits / memory limits. This is especially the case for stores which may have large inventories, but not necessarily high levels of site traffic. These people have had to make do with generating multiple, partial feeds which increases complexity for them, and can cause other issues around re-use of output data with multiple integration partners.

There’s a couple of obvious approaches to solving this:

  1. Make the processing less complex
  2. General code optimisations
  3. Pre-generate the output and cache it

I’ve looked at, and worked on, several of these avenues over the years, [1] and [2] have given me some pretty good results, and even proved a learning opportunity in WordPress internals at times…

It’s now reached the point that the only way to eek out more performance would be to stop using WordPress to generate the content, and to pull it out with straight database queries instead. This would no doubt be a lot quicker, and a lot lighter on CPU.

However, it misses out on a lot of the point of WordPress – that plugins can add, update, and remove information programatically.

From my support work on the plugin over the years, it’s clear that many people rely on plugins / hook customisations to get their final data to be what they want. Raw SQL queries would most likely result in incorrect data, so it’s not really an option.

That leaves us with item [3] – pre-generating the output and caching it. Due to the previous scalability concerns though – we can’t assume that we can generate the full output in a single request. Nor do we want to have to invalidate the full output if a single product, or category is changed.

The solution I’ve settled on generates cached output for each item included in the full output. This gives a couple of benefits:

  1. We can generate our final output with a simple SQL query to pull all of the fragments. This is quick. We also don’t have to  worry that information won’t include input from other plugins on the site as full processing was used to build the fragment
  2. We can invalidate, and rebuild single products, or sets of products without having to invalidate everything
  3. We can build up our cached output in parts, we don’t have to generate it all at once

I was left with a requirement for background processing infrastructure that the pre-generation, and batched invalidation could be handed off to.

I’ve been working a lot with Laravel recently. One of its strengths, and something I’ve used on most (if not all) of my Laravel projects is its Queue implementation. It allows you to easily add jobs to one or more queues, and have them processed by workers in the background. You can set up multiple queues, and you’re in control of how many workers you set up, and which queues they’ll process. The library is usable standalone – it can be used outside of a Laravel project. Unfortunately this plugin supports WordPress’ minimum requirements which are still anchored at PHP 5.2, while Laravel’s Queue system requires 5.6.4 minimum making it a non-starter.

Fortunately, I came across and mentally noted the WP Background Processing library from Ashley Rich / Delicious Brains a while ago:

It’s the perfect solution to this problem. It’s simple, and lets you build background jobs that will run automatically through WordPress’ existing cron infrastructure, taking care to balance running as many jobs as possible in an execution with getting through the jobs as quickly as possible. It’s also compatible with PHP 5.2.

Using the library has let me build out some pretty neat invalidation / pre-generation jobs without having to worry about putting together the cron logic myself. Definitely worth looking at if you need to do any background processing in WordPress plugins.

March 7, 2017
by Lee
4 Comments

Custom model events with Laravel 5.4

Laravel’s eloquent ORM has a nice system for tracking, and responding to events on models. This allows you to create an Observer class that handles code that should run when various actions are performed on a model. There are a set of standard events (created, updated, deleted etc.) that can be used to achieve many things. These can be useful if you need to send notifications when a new model is created, or update something when a model is updated.

However, as your models get more complex, you may find yourself needing different events. Imagine you have a content model which has statuses, for example draft, pending review, and published. If you want to perform different actions based on the model status then you can attach an observer to the model, and listed for the created / updated events.

You can then check the new status and fire off the relevant code. This can get messy though, and also doesn’t reflect what happened to the model to move it into it’s current status. For example, is the content in “pending review” because it’s a draft that someone wants to publish, or an article that was published but has been bumped back down for review? The current model status doesn’t tell you that.

Now, you can create your own events in Laravel, and fire them off, however that would mean you have some event listeners in your model observer, and some in a bespoke event listener. Messy.

It’s nicer to be able to handle everything the same way. Fortunately Eloquent’s model event system can be extended to support custom events which you can fire at the appropriate point.

To add a new event, simply set the $observables property on your model.

<?php

use Illuminate\Database\Eloquent\Model;

class MyContentModel extends Model
{
  /**
   * Additional observable events.
   */
  protected $observables = [
    'sentforreview',
    'rejected',
  ];
}

This tells Laravel that there are two new model events that an observer can listen for. In our observer, we can listed to these by defining the appropriate function as we would with any of the standard events:

<?php

namespace App\Observers;

class MyContentObserver
{

  public function sentforreview(MyContent $myContent)
  {
    // Your logic here.
  }

  public function rejected(MyContent $myContent)
  {
    // Your logic here.
  }

}

So far, so good. We have defined our custom observables, and built a method to listed for them. However – the event will never be fired. After all, Laravel doesn’t know when it should be fired. Firing the event is simple though. Within your model you simply call fireModelEvent() and tell it the event you are firing, for example:

<?php

use Illuminate\Database\Eloquent\Model;

class MyContentModel extends Model
{
  public function sendForReview()
  {
     // Do any other updates to the model here.
     // Then fire the event.
     $this->fireModelEvent('sentforreview', false);
  }

  public function reject()
  {
     // Do any other updates to the model here.
     // Then fire the event.
     $this->fireModelEvent('rejected', false);
  }
}

Now, you can have a single model observer that handles standard, and custom events keeping everything nice and tidy.

February 20, 2017
by Lee
0 comments

Product tours with Hopscotch

I’m just about to launch a brand new project to the world. As a small business, part of the work I’m doing on the project is looking at how I can use smart technology solutions to engage with new users automatically, and help them through the process of getting up and running.

If you’ve used modern software-as-a-service products – you’ll be familiar with these sorts of solutions, that can include things like:

  • Online knowledgebase articles
  • Drip-fed email series
  • Online product tours

I’ll probably talk about some others another day, but today I’m looking at “Online product tours”. This is where you have pop-up dialogs guide the user through key features of your app. Not sure what I mean, check out the final result, the welcome tour on MyHill.blog.

I looked at a couple of Javascript based open source packages when I was building the tour:

As you might have guessed from the article title, I finally settled on Hopscotch for my tours. For me, the advantages where the ease with which I could get multi-page tours set up, support for callbacks, and of course the actual user-facing experience.

Shepherd was my first favourite, and I got most of the tour implemented in this initially. However I hit a couple of issues running some code I needed on callbacks – I couldn’t easily get them to run at the right time. The transitions between steps were also a little harsh for my liking, it was easy to get confused jumping between sections of (potentially) long pages. Other than that though it was a solid, good-looking solution.

I really liked the highlighting offered by intro.js, but it didn’t quite work for everything I wanted to do, and the out-of-the-box theming was a little simple for my liking. Other than that though it also seemed fairly nice.

Overall, I think I’d be happy working with any of these three in the future, but Hopscotch was definitely the best match for this project.

January 18, 2017
by Lee
0 comments

Uptime monitoring with Uptime Robot

The rest of this series has mostly covered things you’d use when building a site. Once you’ve finished building though, and your new site has gone live there are plenty more things you need to do. There’s hosting, SSL certificates, CDNs, and a variety of other services that only apply when a site is in production.

If you’re doing things properly you’ll almost certainly want to monitor your live site.

There are a multitude of things that you can monitor.  I’ve touched on monitoring previously where I talked about how I use sentry.io to track application errors. You might also want to:

  • track user behaviour (I wrote about using Google Analytics’ advanced analytics monitoring in a recent post)
  • monitor and analyze server performance (I use Newrelic)
  • analyze deep language internals / code-paths (Blackfire is pretty awesome for PHP).

For me though, the most basic, and essential need is to understand whether your application is up and running. I’ve used Uptime Robot for this for a while. They have a free tier that provides everything you need for day-to-day monitoring. This provides up to 50 different monitors, checked every 5 minutes, with email alerts when sites go down.

Their standard HTTP monitors checks if the specified URL responded with a positive HTTP response. They also offer keyword monitors where it will fetch a page and check for the presence or absence of specified keywords.

Once you need to go beyond that, they have a very reasonably priced Pro plan that gives more frequent checking, complex alerting rules, and SMS alerts. Perfect for higher value sites.

For me, UptimeRobot is a perfect example of a service that does one thing, and does it well.

November 15, 2016
by Lee
0 comments

IP Geolocation with MaxMind’s GeoLite2

IP geolocation is something that’s needed more and more as people publish & trade internationally. Stores want to offer different pricing, currencies, shipping rules to different countries, or provide content in different languages.

Many systems have something already baked in, but working on a project recently I needed to do some simple country-level IP geolocation. My previous iteration had used a third party service, but that slowed things down, and had a tendency to be unreliable – recently going AWOL for a couple of weeks!

There are some free GeoLocation databases provided by MaxMind, but I doubt I’m telling you anything new there, it seems to be fairly well known about. However I’ve never actually used them myself directly, so in I dived.

Firstly I used their public PHP wrapper:

I was able to pull it into my project using Composer really easily, download the database and hook it up simply. Running a geolocation really was as simple as their documentation suggested, I ended up with this in my code:

$reader = new GeoIp2\Database\Reader( GEOIP_DB_PATH );
$record = $reader->country( $this->get_user_ip() );
$country = $record->country->isoCode;

Really nice and simple, and a lot more reliable than my previous solution, and definitely something I’d use again in the future without hesitation.

Image credit: https://commons.wikimedia.org/wiki/File:Red1234.jpg

November 4, 2016
by Lee
0 comments

Assessing software health

First, a question…

When you find a piece of software to fill a specific need on a project, do you:

  1. Add it in straight away while proclaiming “Open source is awesome, on to the next feature everybody!”
  2. Proceed cautiously, taking a good look at the project’s health before adding it in

If you picked option 1, then congratulations; this blog post is firmly for you.  If you picked option 2 then you’re already ahead of the game, but hopefully this will give you some thoughts about how you assess project health.

A lot of the agencies (and freelancers) that I’ve worked for, or with, have got some way of assessing project health when considering software to add to their project. For larger agencies that might be formal, written-down rules. In smaller agencies its often unwritten “best-practice”. For freelancers, it’s commonly just a mental checklist / gut-feel that gets worked through.

As I’ve moved into working more heavily with new frameworks over the past 6 months, I’ve had to do my fair share of searching for new (to-me) software to plug missing holes. I’ve posted about a lot of the solutions I’ve found in my Stuff I’ve Used series.

There were a couple that I had on my shortlist to cover, but I’ve since decided need to be covered differently. The main reason is that while they solve my problem today, I’m not confident with the health of the projects to view them as long term / maintainable solutions. So – I’ll be talking about them in a blog post soon, but I wanted to write down my thoughts on project health first to give that post some context.

So, how do I assess project health?

I work with different technologies, and the exact measures vary depending on the ecosystem I’m working in. For example a module on drupal.org, a free WordPress plugin, or a repo or package on GitHub have different indicators which you can use. The general principles are the same though. I look at each of the following areas before making my decision.

  • Stability
  • Active maintenance
  • Release management
  • Documentation
  • Code quality

For each one you can just weigh things up to get to your gut feel. You can use a fancy scoring matrix if that floats your boat, or if you need to set rules / guidelines for a team. You might also have specific “red flags” in each area that would block you from using a project.

Here’s the things I look for.

Stability

What does it mean to be “stable”? In some eco-systems it might be that there is a release that’s marked as “stable”. Drupal.org for example lets module maintainers tag releases as stable / dev etc. Packagist packages generally follow semantic versioning allowing you to infer stability from the version number. However, as a concept it means that there have been a couple of releases, there is basic usage / installation documentation, and there has been some feedback in the form of issues / feature requests.

Some projects don’t even have a release (common with GitHub repos), or have only an initial release. This generally makes me uncomfortable, particularly because it makes it really hard to assess some of the other areas.

sail_wagon_edit1There’s nothing to say that new software is bad. However, new software is often subject to change, which can make extra work for you as integration / usage changes – or worse make you stick with an old version to avoid the pain.

I don’t think I’d ever red-flag a project based on “age” – after all newer projects haven’t always acquired “feature-bloat” either. However, it’s certainly easier to feel confident about an established project than a new one.

Active maintenance

abandonedThis covers a few things, and as ever there are a few different ways to measure it. If the commit history is available, I’d look at how recent the latest commits are, and how sporadic they are. Of course – no recent commits isn’t always a bad thing – stable software doesn’t necessarily need changing regularly. I’d also look at issues raised against the project (if that’s available), to see if issues are responded to, and/or worked on. For GitHub repos, I’d look at outstanding pull-requests to see if they are merged and/or responded to.

If there are issues / pull-requests backing up, and no sign of those being worked on / merged it would definitely count against the project.

Release management

package-v1-0-0This looks at whether there are releases firstly – I’d hesitate to depend on a project from which you alway just had to go with an arbitrary commit as the version you’re using.

Where there are releases, it’s important to check that they are “usable”. This means a few different things:

  • that the version numbering is sensible (semver preferably)
  • that each release has clear, plain-english release notes – a list of commits since the last release does not count
  • releases are made with a relevant frequency, not too often, not too rarely
  • that releases aren’t regularly superceded by bugfix releases to the release

Documentation

A package that you’re going to pick up and use in your project should have some level of documentation.

Important note to developers: If your software has no releases, and no documentation you don’t have an “open-source” project, you’re just hosting your version control in public.

I’m not going to suggest that you only use software that has pages, and pages of documentation. At the bare minimum though I’d expect a project to have a short document that explains:

  • what the software should do
  • what the software isn’t expected to do
  • how it should used
  • important dependencies
  • basic ‘getting started’ instructions

Code quality

As a developer, I’m fortunate enough that I can generally look at the code and see how well I think it’s written.

I generally check for basic security precautions being taken (proper escaping of SQL to avoid injection, use of tokens to prevent CSRF, and output escaping to avoid XSS). I also check that the software fits the general approaches taken by the eco-system in which it fits. I’ve found that adherence to eco-system best practice is usually a good barometer for whether its been developed by someone who has taken care with their code.

If you’re non technical, or perhaps don’t have experience in the technology you’re pulling in, then if you know someone who is skilled – it’s worth asking them to give it a quick review.

Summary

Hopefully this has given you a feel for some of the things I look at when deciding whether to use a new module / plugin / package. If you’re not doing any evaluation right now – I’d encourage you to start. It saves time in the long run.

Photo credits:

 

October 28, 2016
by Lee
0 comments

Testing Laravel emails with MailThief

The last couple of Laravel projects I’ve worked on have all included important emails being delivered. It was important that the triggering of these emails could be tested. I also needed to make sure that the correct email was being sent, as some processes would trigger different emails based on the data input.

In the first project, I relied on Mockery to test that Mail functions were called, e.g.

Mail::shouldReceive('send')
  ->once()
  ->with(
    'emails.enquiry',
    Mockery::any(),
    Mockery::any()
  );

This allowed testing that the correct email was being triggered. However if you want to test in more detail that that, for example testing that data from input has been inserted correctly into the email content then it gets a lot less straightforward. On the next project I tried to evolve how I was testing emails. I needed to do some more detailed testing on email contents, so I turned to the MailThief package.

This allowed me to easily test not only that emails were being triggered, but specific tests against the email content. Here’s some examples from my tests:

$this->seeMessageFor($company->email);
 $message = $this->lastMessage();
 $this->assertTrue(
  $message->contains('Web enquiry</title>'),
  $message->getBody('html')
 );
 $this->assertTrue(
  $message->contains('0100 000000'),
  $message->getBody('html')
 );
 $this->assertTrue(
  $message->contains('<a href="tel:0100000000'),
  $message->getBody('html')
 );

I find it much more consistent to keep assertions after the actions in my test rather than having them as expectations before my actions. The package has proved so flexible that I’ll be using it for all email-related tests in future.

Thanks Tighten Co!

October 27, 2016
by Lee
0 comments

Image handling in PHP with Intervention Image

If you’ve done much with PHP you’ve probably come across its image handling libraries. Normally this involves using either the GD, or ImageMagick.

These work reasonably well, but there are a number of disadvantages:

  1. The APIs are thin wrappers around the relevant image libraries. Neither of them offer particularly developer-friendly APIs, and the two APIs aren’t equal enough to make switching between them simple.
  2. You have to know which library your server has in order to start developing, or make extra work, and support both.

On a recent project I needed to do image manipulation (thumbnail generation etc.), and came across the Intervention Image library. This is a PHP library that can be used in any PHP project, and offers a much improved API for working with images. The API is agnostic to whether you’re using ImageMagick, or GD, so you can swap between them at will without having to re-code your application.

Next time you’re working with images, check out Intervention Image.

 

 

October 26, 2016
by Lee
0 comments

Activity logging in Laravel

I’ve been working on a new service for the last month or so that enables people to log, track and share walks in the UK’s hills and mountains. The service is based on Laravel, and one of the things I wanted to include on the site was an “activity feed”.

There are two types of feed I wanted to create. The first was for my own benefit. I wanted to see a simple timeline of activity on the site as usage builds up. Secondly, I wanted to be able to log data so that I can create public timelines in the future.

This is all pretty simple to build from scratch, but it turns out that the team at Spatie have already got this covered with a general package for logging activity on a Laravel application.

You can easily log custom activity events wherever you like, it’s as easy as:

activity()->log('My custom message here');

Even better though is that you can get it to log changes to Eloquent models automatically using the handy LogsActivity trait. If you add the trait to your model, then creating, updating or deleting a model will automatically create a log entry. The log entries include the item affected, and the “causer” (normally a user).

Going beyond the basics, you can also define custom messages, choose what data to store along with log events.

The activities are logged as normal Eloquent models, so once you have your data logged it’s easy to pull out the information you need using standard Laravel querying, and views.

October 25, 2016
by Lee
0 comments

Adding Google Maps to your Laravel application

On a recent project, I needed to add some Google Maps functionality to a number of pages. The functionality I was looking for included:

  • adding KML layers
  • adding markers onto the maps with custom info windows
  • layering photo thumbnails onto the maps.

While I’ve used the Google Maps JavaScript API plenty of times before, I was looking to minimise the amount of custom code I used. The project is Laravel-based, so I had a look around for an existing package to handle some of the basic features. I ended up picking the Googlmapper package by Brad Cornford:

With the library in place, I didn’t have to worry about the basics. Instead, I spent my time on custom elements I wanted to add. I logged an issue early on with a bug that came up with my use case, and Brad was really helpful getting it resolved and into a release. Brad is actively working on the library, and a few features I was planning as custom ended up being covered by updates to the library mid-project.

Each time I moved on to another feature I found something in the library to help me. If you’ve got developers who are comfortable with Laravel and PHP, but not necessarily JavaScript, it’s a really nice library that lets you build your solution in PHP.