Lathonez' Blog

End to End testing an Ionic2 project

Mon, 15 Jan 2018 01:48:23 +0000

Updated for Ionic 3.9.2 and Angular 5.0.3

TL;DR - I have an Ionic 2 project on github set up for E2E testing with protractor, dive in, or read on.

The previous post in this series on Unit Testing does a bit of intro that I won’t repeat here. For the purposes of this post, it’ll be useful to have the demo app cloned locally.

Install dev dependencies

npm install --save-dev angular-cli @angular/router connect jasmine-reporters jasmine-spec-reporter protractor serve-static ts-node

Install config files and boilerplate

Into your project’s root:

.angular-cli.json: Angular Cli’s config file
protractor.conf.js: Protractor’s config file
tsconfig.ng-cli.json: Angular Cli’s base compiler config

Into a newly created ./e2e folder in your project’s root:

tsconfig.e2e.json: Angular Cli’s compiler config

For the lazy:

for file in angular-cli.json protractor.conf.js tsconfig.ng-cli.json
do
  wget https://raw.githubusercontent.com/lathonez/clicker/master/${file}
done

mkdir e2e
cd e2e

for file in tsconfig.e2e.json
do
  wget https://raw.githubusercontent.com/lathonez/clicker/master/e2e/${file}
done

Modify existing Ionic config files:

Add jasmine types in Ionic’s tsconfig.json (only necessary on Windows):

    "typeRoots": [
      "node_modules/@types"
    ],
    "types": [
     "jasmine"
    ]

Add the following to your gitignore file:

# e2e
/e2e/*.js
/e2e/*.map

Hook ng e2e into your package.json scripts array:

  "scripts": {
    "e2e": "ionic-app-scripts build && protractor",
    "postinstall": "webdriver-manager update"
  },

A simple e2e test on app.ts

Create a simple e2e test file ./e2e/app.e2e-spec.ts to get us going:

import { browser, element, by } from 'protractor';

describe('MyApp', () => {

  beforeEach(() => {
    browser.get('');
  });

  it('should have a title', () => {
    expect(browser.getTitle()).toEqual('MyApp');
  });
})

Update the webdriver

As we hooked webdriver-manager update into our npm postinstall script, we just need to run npm install. Thus this step is covered by simply installing your project (in the future):

x220:~/code/myApp$ npm install

...

> ionic-hello-world@ postinstall /home/lathonez/code/myApp
> webdriver-manager update

...

[16:38:03] I/update - chromedriver: setting permissions to 0755 for /home/lathonez/code/myApp/node_modules/protractor/node_modules/webdriver-manager/selenium/chromedriver_2.25

Running the tests

As we hooked into our package.json above, we can run the tests with a simple npm run e2e.

x220:~/code/myApp$ npm run e2e
(node:31496) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.

> ionic-hello-world@ e2e /home/lathonez/code/myApp
> protractor

[16:40:33] I/direct - Using ChromeDriver directly...
[16:40:33] I/launcher - Running 1 instances of WebDriver
Started
Spec started
.
  MyApp
    ✓ should have a title

1 spec, 0 failures
Finished in 1.631 seconds

Executed 1 of 1 spec SUCCESS in 2 secs.
[16:40:36] I/launcher - 0 instance(s) of WebDriver still running
[16:40:36] I/launcher - chrome #01 passed

Using Docker?

Testing on docker (or any other headless environment), requires the use of xvfb, a virtual frame buffer implementing the X11 protocol.

apt-get install xvfb

Change the e2e line in package.json to include xvfb-run, ensuring protractor has access to the virtual display:

  "scripts": {
    "e2e": "xvfb-run && ionic-app-scripts build && protractor",
    ...
  }

For more information see issue #144

Contribute

Clickers is a work in progress. If you’d like to help out or have any suggestions, check the roadmap sticky.

This blog is on github, if you can improve it, have any suggestions or I’ve failed to keep it up to date, raise an issue or a PR.

Help!

If you can’t get any of this working in your own project, raise an issue and I’ll do my best to help out.

If you have a general question about e2e testing concepts (e.g. how can I write an e2e test for some-module), see our General Testing Help thread.

Say “Thanks”

If the clicker project helped you out, show it some love by giving it a star on github <3:

Unit Testing an Ionic2 project

Mon, 15 Jan 2018 01:48:23 +0000

Updated for Ionic 3.9.2 and Angular 5.0.3

This blog and associated project have been around for over a year, since the early days of Ionic 2. Recently, Ionic have brought out an official example repository for unit testing.

So why is this blog still relevant? We spent a lot of time and effort migrating the project over to the example setup. We found that:

The repo is not mature and has a number of outstanding issues that make it unsuitable for production
It is meant to be a very lightweight example and will have minimal support from Ionic
It does not use angular/cli for testing, so lacks community support and resources
Ionic are ultimately looking to bake testing support directly into ionic-app-scripts anyway, so the example repo is a stop-gap.

For ~large apps, or anything that needs production support, I recommend this setup. For small / side projects Ionic’s example will probably suffice.

Install dev dependencies

Install the following npm dev dependencies, or simply merge our package.json with your own.

npm install --save-dev @angular/cli @angular/router @types/jasmine @types/node ionic-mocks jasmine-core jasmine-spec-reporter karma karma-chrome-launcher karma-cli karma-jasmine karma-jasmine-html-reporter karma-coverage-istanbul-reporter karma-junit-reporter

Install config files and boilerplate

Into your project’s root:

.angular-cli.json: Angular Cli’s config file
karma.conf.js: Karma’s config file
tsconfig.ng-cli.json: Angular Cli’s base compiler config

Into your project’s ./src folder

polyfills.ts: Pollyfills used by Angular Cli
test.ts: Main entry point for our unit tests. Remove references to ClickerServices and Translate!
tsconfig.spec.json: Angular Cli’s compiler config for spec files

For the lazy:

for file in .angular-cli.json karma.conf.js tsconfig.ng-cli.json
do
  wget https://raw.githubusercontent.com/lathonez/clicker/master/${file}
done

cd src

for file in polyfills.ts test.ts tsconfig.spec.json
do
  wget https://raw.githubusercontent.com/lathonez/clicker/master/src/${file}
done

Modify existing Ionic config files:

Add jasmine types in Ionic’s tsconfig.json (only necessary on Windows):

    "typeRoots": [
      "node_modules/@types"
    ],
    "types": [
     "jasmine"
    ]

Add the following line to the scripts object in your package.json (generating code coverage breaks sourcemaps, so we add an explicit option for it):

  "test-coverage": "ng test --code-coverage",
  "test": "ng test"

test.ts

This file is worth exploring a little further. We’ve created a couple of functions to remove a lot of the boilerplate around an Ionic testbed setup, we’ll be using these in any of our unit tests that create a Angular 2 components.

The following function configureIonicTestingModule takes one or more of your components and sets up an Ionic test bed for them:

  public static configureIonicTestingModule(components: Array<any>): typeof TestBed {
    return TestBed.configureTestingModule({
      declarations: [
        ...components,
      ],
      providers: [
        App, Form, Keyboard, DomController, MenuController, NavController,
        {provide: Platform, useFactory: () => PlatformMock.instance()},
        {provide: Config, useFactory: () => ConfigMock.instance()},
        {provide: ClickersService, useClass: ClickersServiceMock},
      ],
      imports: [
        FormsModule,
        IonicModule,
        ReactiveFormsModule,
      ],
    });
  }

And this is just a wrapper we’ll call from beforeEach to compile your components:

  public static beforeEachCompiler(components: Array<any>): Promise<{fixture: any, instance: any}> {
    return TestUtils.configureIonicTestingModule(components)
      .compileComponents().then(() => {
        let fixture: any = TestBed.createComponent(components[0]);
        return {
          fixture: fixture,
          instance: fixture.debugElement.componentInstance,
        };
      });
  }

This means that instead of needing the above code in each of your component’s spec files, you simply need:

  beforeEach(async(() => TestUtils.beforeEachCompiler([MyComponent]).then(compiled => {
    fixture = compiled.fixture;
    instance = compiled.instance;
  })));

Alternatively, you can write all this inline in your spec files (as opposed to using TestUtils above):

  let fixture = null;
  let instance = null;

  beforeEach(async(() => {

    TestBed.configureTestingModule({
      declarations: [MyComponent],
      providers: [
        App, Platform, Form, Keyboard, MenuController, NavController,
        {provide: Config, useFactory: () => ConfigMock.instance()},
      ],
      imports: [
        FormsModule,
        IonicModule,
        ReactiveFormsModule,
      ],
    })
    .compileComponents().then(() => {
      fixture = TestBed.createComponent(MyComponent);
      instance = fixture;
      fixture.detectChanges();
    });
  }));

Your first unit test

Pick one of your components to write a test for and create a component-name.spec.ts file for it.

A simple skeleton unit test file looks like this, where HelloIonic is whatever component you’re testing.

import { ComponentFixture, async } from '@angular/core/testing';
import { TestUtils }               from '../../test';
import { HelloIonicPage }          from './hello-ionic';

let fixture: ComponentFixture<HelloIonicPage> = null;
let instance: any = null;

describe('Pages: HelloIonic', () => {

  beforeEach(async(() => TestUtils.beforeEachCompiler([HelloIonicPage]).then(compiled => {
    fixture = compiled.fixture;
    instance = compiled.instance;
  })));

  it('should create the hello ionic page', async(() => {
    expect(instance).toBeTruthy();
  }));
});

You’ll also need to add ./ to the templateUrl path in your component .ts file:

templateUrl: 'hello-ionic.html' becomes templateUrl: './hello-ionic.html'

Running the tests

npm test

x220:~/code/myApp$ npm test
(node:17800) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.

> ionic-hello-world@ test /home/lathonez/code/myApp
> ng test

30 10 2016 19:12:00.472:WARN [karma]: No captured browser, open http://localhost:9876/
30 10 2016 19:12:00.491:INFO [karma]: Karma v1.3.0 server started at http://localhost:9878/
30 10 2016 19:12:00.491:INFO [launcher]: Launching browser Chrome with unlimited concurrency
30 10 2016 19:12:00.546:INFO [launcher]: Starting browser Chrome
30 10 2016 19:12:01.732:INFO [Chrome 54.0.2840 (Linux 0.0.0)]: Connected on socket /#-CZqilkRyMTyIn1xAAAA with id 84944674

START:
  Pages: HelloIonic
    ✔ should create the hello ionic page

Finished in 0.776 secs / 0.77 secs

SUMMARY:
✔ 1 test completed

Congrats! You now have unit testing working in your Ionic 2 project.

Test Coverage

This set up outputs lcov coverage to ./coverage in the root folder of your app. If you browse to /path/to/myApp/coverage/lcov-report/ in a web browser, you’ll get an overview of all your tested files. Drill into one and you get line by line info as to what’s covered.

You can also use external tools, I highly recommend codecov.

Debugging Setup

There are a few fiddly steps to get Angular CLI integrated into your Ionic project. If something isn’t working for you, you have missed a step above. This setup works cross platform and is running on a closed source project with 400+ unit tests.

Before raising an issue, please follow these basic steps to debug the setup:

check you are on node LTS
remove and reinstall node_modules
clone a copy of clicker and compare the relevant files with your own using a diff tool (meld)

If you have performed these steps and still have no luck, please raise an issue (see below) - note that we will probably need access to your source code to help out.

Debugging the Tests

Sometimes it’s useful to debug our tests in the Chrome console.

Hit the Debug button and another tab will open. Open the dev console and you can see the output of all your tests, along with any errors which can be debugged as per usual.

Contribute

Clicker is a work in progress. If you’d like to help out or have any suggestions, check the roadmap sticky.

This blog is on github, if you can improve it, have any suggestions or I’ve failed to keep it up to date, raise an issue or a PR.

Help!

If you can’t get any of this working in your own project - follow the Debugging Setup, raise an issue and I’ll do my best to help out.

If you have a general question about unit testing concepts (e.g. how can I write a unit test for some-module), see our General Testing Help thread.

Say “Thanks”

If the clicker project helped you out, show it some love by giving it a star on github <3:

Mitsubishi L300 4x4 Camper Coversion

Thu, 30 Nov 2017 01:48:23 +0000

Bridget the Van

In January 2017 Orla and I were living in Sydney. We needed a car. We were torn between a “sensible” car we could sleep in, like a Forester, or a van project.

We chose bridget and the rest is history. Her name comes from the NSW rego - BR27JG.

Why an L300?

We had the following requirements for a camper:

Not have terrible fuel consumption (no V6 / V8)
4x4
Short Wheel Base so we could park it in town (and it could get places)
Big enough for a permanent bed / sofa + kitchen
Be on NSW plates

Needing a 4x4 with good fuel consumption removes a bunch of contenders:

Standard vans used in conversions lack 4x4 (Hiace / Econo / straight Mitsi’s / Kias)
Any 4wd big enough to be seriously useasble (without a roof tent). A troopy would have been ideal but any we could find were V6 or above. Ditto patrol / cruisy. Hilux too small.
Post-factory 4x4 (sprinter / transit) conversions are usually LWB and always over budget

The only other van we found that satisfied all of the above was a T4 syncro bus.

Reasons we picked bridget:

Proper 4x4 (hi / lo range gearbox + chasis off a Pajero) with 23” Rugged T/As all round. Bridget could go anywhere.
Only 30k on her since recon’d engine

Retrospectively the conversion would have been easier on a T4 (as it’s a T4 and there are so many examples), but we’re glad we went with Bridget.

Past Life

We bought Bridget off a mechanic in Hornsby. He’d built her out of two 4x4 L300s he salvaged. Originally he wanted to put a turbo diesel in her, but that didn’t fly for some reason, so we were left with the 2.4 petrol EFI.

Aside from the engine recon and sticking 30k on the clock, he did the following:

Bull bar
Navara Spot Lights
Tints
Roof Bars
Tough dog shocks
BF Goodrich Rugged Terrain 29”s
Rust Proofing

Remedial Work

Before we could get cracking with the conversion, there was a bunch of stuff that needed sorting out, mostly electrical:

Radio wasn’t wired properly (A/T missing)
Spot lights were in a separate switch mounted on the steering column
Driving lights were wired straight into the headlights (no relay), which was burning out the circuitry in the lighting stalk
Horn was broken
Indicator fuse kept blowing from anderson plug on the tow bar
No central locking
Original insulation on the floor was starting to melt / degrade

We took the dash off, sorted out all the electrical issues, installed remote central locking and some reversing sensors for good measure.

Next we started scraping the tar smelling insulation off the back floor, a few days of scraping (and much inhaled acetone) later we were done.

Insulation

To keep the van cool in the summer and warm in the winter, it’s heavily insulated.

Floor: 9mm ply atop thermalbrane
Walls: 7mm ply packed with earthwooll
Roof: 3mm ply under thermalbrane

Floor / Wall Covering

For the floors we used Wood-look lino instead of laminate for $$, water resistance and less height.

For the walls, we used grey marine carpet. It’s hard waring, inoffensive and something we could get to stick to the plywood.

Wheel Arches

We boxed the wheel arches in 7mm ply, which we covered in the same marine carpet as the walls. We put hinged lids on top so we could use what limited storage they offered, in 12mm pine DAR stained and varnished.

We’ve used the same wood finish all over the van.

Engine Box

The engine in the van is under the front seats, so you get a bulge where the firewall of the engine comes into the back of the van.

We to boxed this off to get everything as square as possible. This also added a bit of storage where the bulge tapers away from the middle of the engine, plus allowing us to stuff it with insulation to keep things cooler.

Boxing method ditto wheel arches.

Sofabed

When working on the conversion we had one thing in mind - we wanted the van to be really comfortable and an environment you’d be happy spending some time in. Most of the vans we see on the road are very basic. The ones that are done well often have the bed permanently in “bed” mode, and really close to the roof. This is great for storage but doesn’t make for a nice space to chill out.

We designed the sofabed to be as low as we could whilst still being comfortable to sit on and offering enough room for the house battery and electrics to fit underneath. We also wanted the top of the seat / mattress to line up with the engine box, so you could use the extra space to stretch out.

The bed is made of two halves, each of which have offset slats.

In “Sofa” mode, the two halves are pushed together and there is no space between the slats. The two halves together make the base of the sofa, while the back is the wall of the van.

To turn the sofa into a bed, you pull one of the halves out towards you, which leaves a gap between the slats (same as a normal double bed). There’s a small leg that screws into the floor near the passenger door to offer extra support (this area takes a lot of stress when you get in and out of the van).

The whole bed is hinged and lifts up via a pneumatic stay to give access to storage underneath.

For the mattress we took advice from the foam shop we went to who are specialists. The custom made foam mattress (including the upholstery) is our single most expensive item in the conversion.

The bed has the same dimensions as a double, but is much longer with the engine box and kitchen top taken into consideration.

Roof / Door Panels

These are mostly an aesthetic item, though the roof panels are also insulated with thermalbrane.

We removed the existing van panels and cut duplicates out of 3mm mdf, replacing the push-tacs with new ones.

The door panels are covered in hard wearing outdoor furniture material, the roof panels are lino.

We also replaced the original grey roof slats with the stained pine.

Kitchen

The kitchen was the most fiddly / technically challenging part of the conversion.

We wanted the worktop to be about as high as a standard kitchen counter top. As the van is lifted this didn’t give us a lot to work with, but it did tie in with being the same height as the engine box.

We also wanted to be able to have a “lid” on top of the kitchen, so when not using it you can lounge about the van as if it isn’t there.

Due to the above constraints we were limited in the fixtures available to us:

Fridge: The fridge is 20 litres and cools to about 20 degrees below the ambient temperature. The fridge is hidden in the worktop via a push-catch and is controlled by a digital control panel on the front of the kitchen.
Sink: The sink is about 15 litres, plenty big enough to do washing up. The faucet swings up and down to fit under the closed kitchen lid.
Water: The water tank is 30L, installed where the spare wheel usually lives (spare is now on the roof). A self-priming pump drives the water flow to the sink.
Grey water: We sacrificed a waste water tank for more drinking water storage. The waste water drain is behind the rear left wheel, we just use a bucket. On reflection we’re glad we did this than have less drinking water capacity. We fill the tank every 3 days as it is.
Cupboards / Drawers: There are two cupboards and two drawers in the kitchen. We found the storage to be sufficient (we have no other kitchen storage on the roof or otherwise).
Stove: We sacrificed an inbuilt stove, for kitchen storage. Additionally we didn’t want to be cooking right in the back of the van anyway (cooking smells mean stinky van). We just use a portable butane stove (small cans).

Cupboards

To make the van feel less “van” like inside - e.g. cover up the bare white metal and square things off, we built three internal cupboards, two on either side of the kitchen and one above.

We installed cornicing which we use to hide the light strips and cut a couple of pieces of cladding to cover remaining van metal.

Lighting

There are four different lights in the living area:

Original main cargo light with a super bright LED instead of the original bulb, wired to the house battery. Can be switched to come on via opening the door or flicking the light switch. We only really use this when we’re cleaning.
Two “Warm” LED light strips (one on each side, separately switched). These give a lovely background light for relaxing in the evening. Use these all the time.
Two “White” LED spot lights above the worktop (switched together). We use these only when cooking.

House Electrics

As leccy is required for lots of various jobs along the way (kitchen / lighting), this went in over time.

The setup has been working well for us with a couple hours driving a day to charge the house battery.

120 amp/hour AGM house battery
Ctek Dual DC-DC charger. These things are legit.
100w solar panel permanently mounted on the roof
2400w (max) / 1200w stable sine wave AC inverter, with 2x 240v power outlets
2 * 2.5 amp USB outlets
Bluetooth battery monitor. Provides real time house battery voltage + low voltage alarm

Roof Storage

The van came with a beast of a steel roof rack. We extended it to build a cage which can house various waterproof items:

2x Tables
2x Chairs
2x Stoves
Parasol
10 litre backup water
Waste water bucket

Additionally we use 2 galvanised steel toolboxes (about 64L each) to store stuff we don’t use every day but don’t want to get wet (bags, shoes, tools, etc).

We also fitted a 2.5 x 3m awning which is great for those sunny (or rainy) days.

Curtains

I found a guy on ebay making ultra low profile curtain rails to measure out of steel / nylon. These are excellent. We got some block-out curtains from Spotlight that keep the van nice and dark.

We also put a roll-up mozzie-screen between the kitchen cupboards so we can sit with the back door open without any unwanted visitors.

Misc van stuff

We’ve made the following improvements to the “van” / driving area, in addition to the camper conversion.

Retro steering wheel / gear knob
Tacho (so weird driving stick without a tacho!)
Dash carpet
Marine voltage meter / USB / 12V socket
Added extra engine insulation (earthwool)

Copying a row from one database to another

Fri, 26 Aug 2016 12:34:23 +0000

How can I copy a row from our dev database to our production database?

Most engineers would give an answer for this, varying depending on their background and the DBMS. The common ones are:

Use the tool that came with your DBMS! mysqldump or pg_dump in the context of this post.
Select / copy to outfile
Use raw sql with a connection to each db
Try sqlworkbench
Turn on replication
Roll your own script

The purpose of this post is to convince you that there’s a better way, a python tool we’ve written called CloneRow.py. It currently supports mysql and postgres. If there’s a decent python lib for your DB we’ll write an adapter (open an issue).

CloneRow.py

We came up with a one liner that adheres to the unix tools philosophy. To copy a row in my_table from development to production, where the my_table.my_column equals my_filter:

CloneRow.py dev prod my_table my_column my_filter

DEV:~$ CloneRow.py dev prod my_table my_column my_filter
2016-08-27 05:29:18 Reading configuration..
2016-08-27 05:29:18 The following columns will be ignored: id last_updated
2016-08-27 05:29:18 attempting to connect to local..
2016-08-27 05:29:18 connected to root@local:my_db - Database version : 90503
2016-08-27 05:29:18 attempting to connect to prod..
2016-08-27 05:29:20 connected to lathonez@prod:my_db - Database version : 90502
2016-08-27 05:29:20 getting local row..
2016-08-27 05:29:20 getting prod row..
2016-08-27 05:29:21 checking encoding..
2016-08-27 05:29:21 source encoding UTF8
2016-08-27 05:29:21 target encoding UTF8
2016-08-27 05:29:21 finding deltas..
2016-08-27 05:29:21
2016-08-27 05:29:21 |--------------------------------|Data Changes|--------------------------------|
2016-08-27 05:29:21   The following columns will be updated on prod
2016-08-27 05:29:21     -updated_column_one
2016-08-27 05:29:21     -updated_column_two
2016-08-27 05:29:21 |------------------------------------------------------------------------------|
2016-08-27 05:29:21
2016-08-27 05:29:21 backing up target row..
2016-08-27 05:29:21 backup file can be found at /tmp/my_table-my_column-my_filter-1472275758838.backup on this machine
2016-08-27 05:29:21 dumping update sql to disk..
2016-08-27 05:29:21 update sql is available for inspection at /tmp/my_table-my_column-my_filter-1472275758838.sql on this machine
2016-08-27 05:29:22 Row has been cloned successfully..
2016-08-27 05:29:22 Type 'r' to (r)estore from backup, anything else to exit

2016-08-28 09:19:39
2016-08-28 09:19:39 |---------------------------|Manual Rollback Steps|----------------------------|
2016-08-28 09:19:39   To rollback manually, run the following steps on this machine
2016-08-28 09:19:39     psql --host 127.0.0.1 --port 5432 --user my_user --pass my_pass
2016-08-28 09:19:39     begin;
2016-08-28 09:19:39     delete from my_table where my_column = 'my_filter';
2016-08-28 09:19:39     -- if more than one row has been deleted above run `rollback;`
2016-08-28 09:19:39     copy my_table from '/tmp/my_table-my_column-my_filter-1472275758838.backup';
2016-08-28 09:19:39     commit;
2016-08-28 09:19:39 |------------------------------------------------------------------------------|
2016-08-28 09:19:39
2016-08-28 09:19:39 operation completed successfully, have a fantastic day
2016-08-28 09:19:39 housekeeping..

Features:

Copy a single row from one database to another, given a column and a filter to apply
Fail-safe operation, with automated and manual rollback procedures provided
Ignore columns that you’ll never want to copy (serials, ids)
Display which columns will be modified in the update (deltas)
Allow the user to check the target system before ‘committing’ the changes
An ssh_config like connection alias system
Ignore schema differences where possible (whilst warning the user)
Check that encoding matches on source and target
Hint at schema changes required to bring source and target tables in line

If the above looks interesting to you, why not head on over to github and give it a try?

Still need convincing? Read on.

Use Case #1: Releasing to prod (INSERT)

We have a configuration table for each of our clients, where one row contains many config items for a single client. Currently this table is 39 columns wide.

When developing for a new client, you add the configuration it needs into the config table. When you’re done, the config table represents a perfect view of that client’s configuration, if you could only get it onto prod.

The standard procedure was: “open phpmyadmin and paste everything in, column by column”. Not only is this error prone but also extremely time consuming.

We eventually wrote a simple mysqldump wrapper script for this. I expect there are lots of similar things out there.

Use Case #2: Releasing to prod (UPDATE)

You’ve just finished updating an existing client on prod. Said config table is frequently updated on production after the initial release.

Best case scenario: It was an easy change with only one option updated. You remember what that option is.

Typical scenario: The change started simple, dragged on for a number of days (if not weeks), involving multiple config options. You vaguely remember which ones you changed.. or do you?

Again, the standard procedure was: “open phpmyadmin and paste in whatever you changed”

If you remembered the columns you’d updated and they hadn’t been changed on prod since you started dev, you’d be ok. If you’d forgotten (and had to blat the whole row), or there had been updates in the interim, you’d be screwed.

Use Case #3 Updating Development (UPDATE)

Similar case to above, though obviously not as mission critical (though your work is on the dev database and what happens if you overwrite it?).

The ideal workflow:

start dev: update development database with the configuration row from prod
finish dev: pull the prod row back in again, so we can see if any changes have happened in the interim
release: merge config changes (from above) if necessary in the dev database, then push that row to prod

Clearly without proper tooling this was never going to happen. When possible, we’d refresh the entire dev database from prod, though more often than not there were loose changes on in the dev database which prevented this.

CloneRow.py vs import / export

By import / export, I’m referring to the following from the intro, which all work in broadly the same way: export a row from one database, import it into another.

dump tools
copying / selecting to outfile
using raw sql with a connection to each database

dump tools:

# run dump command
mysqldump --host myhost.com --user my_user --port 3306 --no-create-info --databases my_db --tables my_table --where my_column=myfilter > dump.out
# copy dump file to the target server
scp dump.out myhost.com:/tmp
# connect to target database
DELETE FROM my_table WHERE my_column = 'my_filter';
# load dump sql
SOURCE '/tmp/dump.out'

copy / select to outfile:

# select to outfile
SELECT * INTO OUTFILE LOCAL 'dump.out'
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n'
  FROM my_table
  WHERE my_column='my_filter';
# copy the file to the target server
scp /exportdata/customers.txt myhost.com:/tmp
# connect to target database
DELETE FROM my_table WHERE my_column='my_filter';
LOAD DATA LOCAL INFILE '/tmp/dump.out';

raw sql:

DELETE FROM target_database.my WHERE some_id = 123;
INSERT INTO to_database.to_table SELECT * FROM from_database WHERE some_id = 123;

These are all overly complicated for your average developer (in my experience).
They’re time consuming. Even if you do this on a daily basis and get good at it, there are multiple steps in each.
They’re prone to error. What happens if you get one of your where clauses wrong?
They don’t do any comparison of data.
They don’t allow an easy rollback

CloneRow.py vs phpmyadmin (etc)

Opening two phpmyadmin windows (one for source and one for target), does give you a kind of visual diff, however:

It is time consuming
It is error prone particularly for wide tables
There’s no easy roll back

If using import / export, you lose the visual comparison and risk updating the wrong data.

CloneRow.py vs sqlworkbench

Sqlworkbench is extremely powerful and versatile; it’s basically an abstraction over JDBC, allowing you to write sql-esque commands on top of the adapter. You can also define connection profiles which remove a lot of the headache from repeated operations.

However, I found it to be a lacking in dbms specific nuances. Jack of all trades (NOT the unix tools philosophy!):

The dump / export format was inconsistent and couldn’t be easily used with the import. This was solved easily enough with sed in the wrapper script, but not ideal.
The postgres JSON type seemed to be unsupported. The JSON output by sqlworkbench wasn’t the same as in pg_dump, more to the point it was invalid JSON. The issue was around escaping quotes.

I recommend looking at sqlworkbench, as it may be appropriate for your use case and it’s a great tool for the right job. I see it as analagous to a [python database adapter][http://initd.org/psycopg/], you have to do a lot of scripting around it.

An sqlworkbench script looks like this:

-WbCopy
 --sourceProfile=$[source_con]
 --targetProfile=$[target_con]
 --mode=update,insert
 --keyColumns=$[column]
 --sourceTable=my_db.$[table]
 --targetTable=my_db.$[table]
 --sourceWhere=$[column]='$[filter]'
 -;

Which you invoke via with Java (in a wrapper script):

java -jar sqlworkbench.jar -vardef='vars.def' -script='export.script'

CloneRow.py vs Replication

Production to Development replication handles Use Case #3 (Updating Development) extremely well. However:

It can be time consuming to set up if you do not have the skill set.
It puts strain on the prod database. It can be difficult to justify replication out of prod, when all you gain is updating development or test environments.
If you have developers working on their own databases, replication isn’t an option.
Whilst it solves Use Case #3, it does not address #1 or #2.

CloneRow.py vs rolling your own script

The vast majority of solutions out there will be script(s) written in house. CloneRow.py started off that way. Rolling your own script has a clear advantage that, it can do literally whatever you need it to do. However, it does have some disadvantes:

You have to actually roll it. We spent a lot of time getting CloneRow.py from zero to where it is now.
There will be bugs. There are bugs in CloneRow.py, but the advantages of using an open source tool used by others is, that we can identify the bugs and fix them together.
You have to maintain it. In my experience such tools are written by a single engineer; other users have very little idea of how it works, or what to do when it goes wrong. This is fine as long as the author is still around when it goes wrong or needs updating, and has the time.

Contribute

This blog is on github, if you can improve it, have any suggestions or I’ve failed to keep it up to date, raise an issue or a PR.

Securing a docker registry behind Apache

Wed, 20 Jul 2016 12:34:23 +0000

We recently went to setup a docker registry on our production build server. We wanted the quickest / easiest way to get the registry going but didn’t know where to start. This information is all out there but not in one place (that I found).

Basically if you’ve already got apache running on the target server, getting everything going is incredibly simple. If you don’t run apache this guide will not useful for you.

Securing the registry

TLS is mandatory when running a remote docker registry. We used letsencrypt to obtain our certificates.

If you don’t already have one, create an apache vhost with a subdomain for your registry. This is just so certbot will prompt you to generate the certificate for it.

<VirtualHost *:80>
    ServerAdmin dev@example.io
    ServerName  docker.example.io
</VirtualHost>

Install and run certbot. It’ll prompt you for the necessary info and configure apache with the certifcates it receives from letsencrypt.

./path/to/certbot-auto --apache

Virtual Host Entry

Thanks to R.I.Pienaar for this. I was banging my head against the ProxyPreserveHost line until I found it on his blog.

Now you’ve got the certs, update your registry’s vhost. Main things to note:

using the SSL Cert we’ve just got from letsencrypt
Proxying any requests to the registry through to http://127.0.0.1:5000 where the registry will be running
Basic auth using htpasswd

If you don’t want auth (e.g. username:password) for your registry, remove the <Location> entries.

<VirtualHost *:443>

        ServerName docker.example.io
        ServerAdmin dev@example.io

        SSLEngine On
        SSLCertificateFile /etc/letsencrypt/live/client.example.io/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/client.example.io/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/client.example.io/chain.pem

        ProxyPreserveHost on
        ProxyPass / http://127.0.0.1:5000/
        ProxyPassReverse / http://127.0.0.1:5000/

        <Location />
                Order deny,allow
                Allow from all

                AuthName "Registry Authentication"
                AuthType basic
                AuthUserFile "/opt/registry-htpasswd/.htpasswd"
                Require valid-user
        </Location>

        # Allow ping and users to run unauthenticated.
        <Location /v1/_ping>
                Satisfy any
                Allow from all
        </Location>

        # Allow ping and users to run unauthenticated.
        <Location /_ping>
               Satisfy any
               Allow from all
        </Location>

</VirtualHost>

Authentication

Skip if you don’t want the username / password authentication for your registry.

mkdir /opt/registry-htpasswd
cd /opt/registry-htpasswd
htpasswd -bc .htpasswd username password

Storage

It’s good practice to specify a storage folder for your registry.

mkdir /opt/registry-data

Running the Registry

As we’re using Apache for TLS and auth, running the registry is straightforward:

docker run -d -p 5000:5000 \
    --restart=always \
    --name registry \
    -v /opt/registry-data:/var/lib/registry \
    registry:2

Using the Registry

docker login docker.example.io
docker push docker.example.io/example_image:latest

Contribute

This blog is on github, if you can improve it, have any suggestions or I’ve failed to keep it up to date, raise an issue or a PR.

Unify DI Boilerplate

Sat, 18 Jun 2016 20:34:23 +0000

When setting up dependency injection for my demo project I follwed this excellent blog post.

This post demonstrates how to unify boilerplate necessary for dependency injection away from your test specs into a single file.

What boilerplate?

Below is a very simple test spec with dependency injection component we’re testing. As you can see, it’s 99% boilerplate, lots of imports along with the beforeEach and beforeEachProvider functions.

import {
  beforeEach,
  beforeEachProviders,
  describe,
  expect,
  injectAsync,
  it,
}                               from '@angular/core/testing';
import {
  ComponentFixture,
  TestComponentBuilder,
}                               from '@angular/compiler/testing';
import { provide }              from '@angular/core';
import { Page2 }                from './page2';
import { Utils }                from '../../services/utils';
import { ConfigMock }           from './mocks';
import {
  Config,
  Form,
  App,
  NavController,
  NavParams,
  Platform,
}                               from 'ionic-angular';

let page2: Page2 = null;
let page2Fixture: ComponentFixture = null;

describe('Page2', () => {

  beforeEachProviders(() => [
    Form,
    provide(NavController, {useClass: MockClass}),
    provide(NavParams, {useClass: MockClass}),
    provide(Config, {useClass: MockClass}),
    provide(App, {useClass: MockClass}),
    provide(Platform, {useClass: MockClass}),
  ]);

  beforeEach(injectAsync([TestComponentBuilder], (tcb: TestComponentBuilder) => {
    return tcb
      .createAsync(Page2)
      .then((componentFixture: ComponentFixture) => {
        page2Fixture = componentFixture;
        page2 = componentFixture.componentInstance;
        page2Fixture.detectChanges();
      })
      .catch(Utils.promiseCatchHandler);
  }));

  it('initialises', () => {
    expect(page2).not.toBeNull();
    expect(page2Fixture).not.toBeNull();
  });
});

For a few files this is fine. In a larger project with tens of spec files, maintaining this boilerplate became a headache for us; Angular 2 and Ionic 2 were evolving and the imports and providers kept changing.

It’s also not great having to read ~50 lines of boilerplate at the top of each file before you can see what’s being tested.

Unifying the boilerplate

We set out to remove as many of the imports as we could, as well as outsourcing the beforeEach functions into a single file.

Here’s the what we ended up with:

import { provide, Type }                              from '@angular/core';
import { ComponentFixture, TestComponentBuilder }     from '@angular/compiler/testing';
import { injectAsync }                                from '@angular/core/testing';
import { Control }                                    from '@angular/common';
import { App, Config, Form, NavController, Platform } from 'ionic-angular';
import { ClickersMock, ConfigMock, NavMock }          from './mocks';
import { Utils }                                      from '../app/services/utils';
import { Clickers }                                   from '../app/services/clickers';
export { TestUtils }                                  from './testUtils';

export let providers: Array<any> = [
  Form,
  provide(Config, {useClass: ConfigMock}),
  provide(Clickers, {useClass: ClickersMock}), // required by ClickerButton
  provide(App, {useClass: ConfigMock}),        // required by ClickerList
  provide(NavController, {useClass: NavMock}), // required by ClickerList
  provide(Platform, {useClass: ConfigMock}),   // -> IonicApp
];

export let injectAsyncWrapper: Function = ((callback) => injectAsync([TestComponentBuilder], callback));

export let asyncCallbackFactory: Function = ((component, testSpec, detectChanges, beforeEachFn) => {
  return ((tcb: TestComponentBuilder) => {
    return tcb.createAsync(component)
      .then((fixture: ComponentFixture<Type>) => {
        testSpec.fixture = fixture;
        testSpec.instance = fixture.componentInstance;
        testSpec.instance.control = new Control('');
        if (detectChanges) fixture.detectChanges();
        if (beforeEachFn) beforeEachFn(testSpec);
      })
      .catch(Utils.promiseCatchHandler);
  });
});

injectAsyncWrapper wraps injectAsync, executing the callback passed in when TestComponentBuilder has completed. The main win from this is that you don’t need to import TestComponentBuilder in each of your specs.

The heavy lifting is done in asyncCallbackFactory, which takes the following arguments:

component: The coponent class we’re currently testing
testSpec: A reference to the test spec. This is used so we can set the component (instance) and fixture variables in the test spet without needing further boilerplate.
detectChanges: Should we invoke detectChanges() against the component fixture in beforeEach? This is useful in cases when you need to do initial / bespoke setup on the component instance before detecting changes. Usually this would be true.
beforeEachFn: Sometimes you need to run an additional function before each test, nothing to do with DI, your standard beforeEach. Before this change you’d have this code in-lined inside createAsync callback.

This diff is a good example of the above arguments in use after the boilerplate has been removed.

Resultant Spec

As we import from our unified diExports file, most of the boilerplate has been removed:

import { beforeEach, beforeEachProviders, describe, expect, it }          from '@angular/core/testing';
import { asyncCallbackFactory, injectAsyncWrapper, providers, TestUtils } from '../../../test/diExports';
import { Page2 }                                                          from './page2';

this.fixture = null;
this.instance = null;

describe('Page2', () => {

  beforeEachProviders(() => providers);
  beforeEach(injectAsyncWrapper(asyncCallbackFactory(Page2, this, true)));

  it('initialises', () => {
    expect(this.instance).not.toBeNull();
    expect(this.fixture).not.toBeNull();
  });
});

You can see the full commit of the above change here.

Contribute

Clickers is a work in progress. If you’d like to help out or have any suggestions, check the [roadmap sticky][clicker-issue-38].

This blog is on github, if you can improve it, have any suggestions or I’ve failed to keep it up to date, raise an issue or a PR.

Help!

If you can’t get any of this working in your own project, [raise an issue][clicker-issue-new] and I’ll do my best to help out.

Excluding files from a Cordova build

Sun, 28 Feb 2016 11:34:23 +0000

Quick post to explain how to remove files from a Cordova (also works for Ionic2) project at build time.

See the original issue on github.

Why would I need to do this?

Your project lifecycle creates artifacts in www/build that you do not want going into the .apk.

In our case we needed to delete the output of our test build phase from www/build/test.

Don’t touch the builder

Originally it seemed the correct approach would be to override the builder configuration. However that brings with it a few problems:

The builders are platform specific (e.g. if you build Android and iOS you’ll need two changes)
There are different builders for Android - ant or gradle - depending on which version of Cordova you’re running
Currently the override does not work in gradle

Use the Cordova build hooks

We went for an after_prepare Cordova hook. cordova build is shorthand for cordova prepare then cordova compile. cordova prepare copies your assets over in prepraration for compilation.

We can therefore make use of the after_prepare hook to remove these assets from platform/**/assets so they don’t make their way into the .apk, whilst leaving the output in www/build intact.

If you add your hook into ./hooks/after_prepare, no changes to config.xml are required - more info.

Node is recommended for the hooks to keep your project cross-plaform. Here’s what we came up with, inspired by an existing hook in Ionic’s app base

#!/usr/bin/env node

var del  = require('del');
var fs   = require('fs');
var path = require('path');

var rootdir = process.argv[2];

if (rootdir) {

  // go through each of the platform directories that have been prepared
  var platforms = (process.env.CORDOVA_PLATFORMS ? process.env.CORDOVA_PLATFORMS.split(',') : []);

  for(var x=0; x<platforms.length; x++) {
    // open up the index.html file at the www root
    try {
      var platform = platforms[x].trim().toLowerCase();
      var testBuildPath;

      if(platform == 'android') {
        testBuildPath = path.join('platforms', platform, 'assets', 'www', 'build', 'test');
      } else {
        testBuildPath = path.join('platforms', platform, 'www', 'build', 'test');
      }

      if(fs.existsSync(testBuildPath)) {
        console.log('Removing test build from assets after prepare: ' + testBuildPath);
        del.sync(testBuildPath);
      } else {
        console.log('Test build @ ' + testBuildPath + ' does not exist for removal');
      }

    } catch(e) {
      process.stdout.write(e);
    }
  }
}

When running cordova build or ionic build, Cordova will invoke your hook after it has prepared your code for compilation, removing those pesky files.

Contribute

Clickers is a work in progress. If you’d like to help out or have any suggestions, check the roadmap sticky.

This blog is on github, if you can improve it, have any suggestions or I’ve failed to keep it up to date, raise an issue or a PR.

Help!

If you can’t get any of this working in your own project, raise an issue and I’ll do my best to help out.