Random Mutterings: Websockets, Mechanising stuff and looking at Stats

How did it get to March so quickly?
If things have gone to plan, this post was scheduled to go out on Monday/Tuesday and I’m probably already on paternity leave and taking some time of to meet my daughter. Which given the lack of posts it might be hard to notice the difference between taking time off and not.

With Feb sailing by I just haven’t written anything, though I have been doing a little video work. So I thought I would do a round-up/link/idea post with things I have done, played with or just found interesting and cool this month. Mostly it’s stuff I either should have written about but didn’t have time or linked to in Twitter but deserves a bit more.

WebsocketD + WP-CLI == Super Easy Websockets for WordPress?

This was last weeks little distraction for 5 minutes, a month or so I came across WebsocketD a Go application that provides a unix deamon for websockets that you can interact with through STDIN STDOUT piped to it. At the time I thought it was neat and went into the list of things to play with.

Earlier this week Web Operations Weekly (Peter does a range of newsletters well worth checking out) arrived in my inbox and reminded me about the project with a few minutes to kill I installed it on a test server and piped some content in and out following their 10 second tutorial it worked! Actually as is always the way it took a good 20 minutes of working out why adding it to PATH hadn’t worked and spotting the inevitable typo. Ignoring that the 10 second tutorial worked well.

So the next step was to pipe it through to WP-CLI a quick and very basic extension that replicated the 10 second example ensued and it worked. However WP-CLI isn’t really expecting to be running continuously open, so I fudged the process by building a shell script wrapper. The result was pleasing.

So where would you use it?

Practically not in anything you plan to distribute but for custom projects this combination could work quite well. For example a couple of ideas sprung to mind, including instantly appearing comments or posts on the front end. Or for providing a web facing monitor for long running WP-CLI tasks for example triggering and running a backup script.

As it was literally a 5 minute experiment I didn’t test the performance or if it would scale.

But WebsocketD + WP-CLI does appear to be a super easy way to introduce websockets to WordPress setup.

Post Status Launched it’s membership site

In case you have been living under a rock, make.wordpress.orgon how the way WordPress installs are being calculated for their stats were changed to reflect a more accurate representation of the WordPress landscape.

The result, it turns out that PHP5.2 isn’t as common as originally reported, which will come as no surprise to well anyone. The stats seem to be now closer then those reported by organisations like PHP.net, The number of 5.2 installs is still high though and curiously the number of 5.3 installs remains relatively unchanged percent wise. It’s also be made clear ˜dinosaur policy’ remains, and we must wait for a Internet extinction event to occur before the minimum version is increased.

WP-CLI backup script

My recent WP-CLI video series has had at least one person watching it as Alastair from here.

If your interested you can see the original video series

• Getting Started with WP-CLI
• Extending WP-CLI
• Working with Shell Scripts and WP-CLI

In other news Alastair and I have also recorded a screencast on setting up HHVM and Memcached on his server which once we done a few follow up bits I will put up, with our amusing commentary.

The recording of that screencast also put paid to another idea I had which was doing some live, setups, so taking a specific process and screencasting the process live. At one point I even contemplated using a platform like Twitch. The reality is that you spend as much time going, WHY WON’T YOU WORK!!! to make such videos un-useful and a bit of a train wreck so, we won’t be doing that!

PGBrowser for when there is no API

One of my failed projects this month and let’s face it we all have dozens each month, or at least I do, was organising a new membership solution for my Patreon patrons. The idea to give me extra information needed to process their payments and sort out digital rewards and provide a more streamlined experience.

This is one of those tales of woes that you hear surrounding VATMOSS if you have no idea what VATMOSS is I suggesting skipping this stuff to the pgbrowser part.

Basically Patreon may or may not be responsible for handling VATMOSS stuff, unfortunately they are still investigating which has left lots of the creators in legal limbo.

Now I’m pretty sure even if they turn round and say it’s my responsibility, the sort of bits I do would be exempt as the work that is being produced is completely free and available to all and the rewards are not a digital goods. There is a small possibility the having your face on the page  reward might be a digital goods but at the moment that process isn’t automated (and even if it was it’s not really a goods/service) still just to be on the safe side and because of some useful other things I had a plan to continue to use Patreon, but to also generate Patrons an account on my site automatically and keep them in sync.

So the next problem I had was Patreon has no API or webhook or anyway to get your data out except a CSV download.

The good news, they also had a fairly simple login process, and a standardised CSV export (although the data inside is a mess).

So the goal was simple:

• Build a script to login and download the CSV
• Process the CSV and import/update users on my site
• Run Restrict Content Pro or similar to manage users
• Use Ninja forms to collect information I needed/wanted

As a by product, I could also put Patrons on a MailChimp list so I can handle communication using a system that I know reliably reaches people.

How I did this after playing around with WP-HTTP library for a bit to try and get it to play nicely with cookies was to use the PGBrowser which is a pure PHP Mechanise library for simulating user interaction. It will work and manipulate documents like a browser (though it can’t process javascript) so you can easily mock a normal user for example:

function login()
	{
		$url = 'http://www.patreon.com/login_i';
		$page = $this->pgbrowser->get($url);
		$form = $page->form();
		$form->set('email', $this->user);
		$form->set('password', $this->pass);
		$page = $form->submit();
		/**
		 *
		 * Check if user is logged in
		 *
		 */
	return strpos($page->html, '"is_creator":true') ? TRUE : FALSE;
	}

Is going to the page, finding the username and password field and submitting the form.

Getting the CSV was simply:

function get_csv()
	{
		$url = 'https://www.patreon.com/downloadCsv?hid=';
		$page = $this->pgbrowser->get($url);
		
		$lines = explode( "\n", $page->html );
		$headers = str_getcsv( array_shift( $lines ) );
		$data = array();
		foreach ( $lines as $line ) {

			$row = array();

			foreach ( str_getcsv( $line ) as $key => $field )
				@$row[ $headers[ $key ] ] = $field;

				$row = array_filter( $row );
				if(count($row) > 4)
				{
					$data[] = $row;
				}

			}
		if(empty($data)) return false;

		return $data;
	}

The weird little for-loop was because every so often, random extra data was shunted in to the CSV that wasn’t a specific user, so the count was to make sure we were only processing data with at least the right number of columns to be user data that we could use.

Once the data was grabbed, each user was looped through and added as a WordPress user, along with the Patreon information as metadata and then automatically updated Restrict Content Pro metadata for that user, so actually the plugin wasn’t doing much user management or processing.

And you know what it worked, when I built it in January, but sadly when I went to put the finishing touches, it turned out they changed their login system, to be well awkward. Instead of a nice easy to use web form the entire login system now involves javascript. This means my scraper tool doesn’t work it also means virtually all blind readers systems won’t either. The kicker, the signup and taking payment side still does so someone with a blind reader, can have their money taken off them with no way to login and cancel

So this new login means my little solution no longer worked, but just to prove a point, I fired up Codeception and set it to work with a copy of Selenium which used a headless Firefox browser to access the site, login and download the CSV.

However while it’s certainly still technically possible to gain access, the idea of running Firefox and Selenium to gain access to download the CSV to process on my live server doesn’t inspire me and as I felt they have taken steps to deliberately stop general access, I’m thinking that I will simply abandon this idea.

Unfortunately it does leave me in limbo so for time being Patreon Patrons are not being charged while I figure a long term solution. In the mean time a few takeaways PGBrowser is a really useful mocking tool that is super light, and if you design a login solution as javascript it can still be walked around.

On plus side, it does mean if you become a Patron you get to say you are supporting my videos and know I’m currently not charging so cancel before I do. Though that would be mean and I may get a brain wave, or simply put my head in the sand and charge anyway.

However please consider becoming a patron, for more information see patreon.com/tnash

C# Weekly

So not WordPress related but my friend Jack Hughes has set up and sent out episode 1 of C# weekly for those who live in the world of Microsoft, expect J#, F#, and D Flat to follow shortly.

Finally ¦

A few interesting things that caught my attention

Jeff Waugh wrote a nice article about a WordPress Renasssince.

That’s me done for now did you know you can find me on twitter Patreon page.

My posting schedule is going to be a bit off for the next month or so (more then usual), but I will hopefully be getting the Acceptance Testing video series launched this month, especially with my talk at the end of the month at WordCamp London.

If you like this style post which is a bit of an experiment, or if you want any of the ideas in the post turned into full posts then let me know in the comments below and I will add them to the queue.