Wilco van Esch

The European beewolf

2026-06-09T00:00:00Z

This is a European beewolf.

If you saw it making low passes around your garden, you might hurry out of its way, or think "is that a fat wasp or two mating ones?" before moving on with your life.

But there's a lot of drama happening.

This beewolf happened to be carrying a paralysed honey bee. The bee had been on its way to drink nectar from flowering oregano plants, where its friends were already gorging themselves. It didn't notice the beewolf, who had gotten close enough to smell the bee's cuticular hydrocarbins and confirm that the target was indeed a bee. As the bee went to settle on one of the pink flowers, the beewolf launched itself down at the bee and grabbed it before it even had a chance to land.

As the beewolf wrapped its legs around the honey bee, the bee tried to frantically stab at her assailant with her venomous stinger. Bees have been flying out in the open for millions of years, they can do so because they can defend themselves. But the beewolf has her own defence, a thick and smooth chitinous exoskeleton. They honey bee struggles in vain, while the beewolf slips her stinger in the articular membranes behind the bee's front legs, paralysing it.

At this point, the bee becomes a passive observer. Physically passive, anyway. We have a hint of what may be going on in the tiny bee's brain. In lab settings, honey bees exposed to simulated predatory attacks show neurochemical shifts which mirror that of mammals when they are experiencing acute anxiety or depression. It may look dead, but it's still seeing, smelling, detecting touch. Only its muscles are paralysed.

The bee is brought to the beewolf's burrows. Not a bustling wasp nest. The beewolf is a solitary wasp, and digs burrows in sandy soil. You could mistake these for ant tunnels. For a little while, the beewolf hovers over the entrance. It takes a bit longer than usual, because someone has moved the bucket that was four inches to the west of it. The beewolf is lining up the visual orientation of objects near the nest to how they were arranged when she left.

Minutes later, she drops the bee nearby and searches the entrance by close inspection. The bee might sense it's free, but it's still immobile. The beewolf finds the entrance, recognising the shape of the sand plug she'd used to close it. Returning with the bee, she opens the entrance and slowly drags the bee down the dark tunnel to the branching burrows where her larvae are wriggling in their humid brood cells. This is where the bee's journey continues for up to two weeks in lonely, unseen horror.

Inspecting input field placeholder CSS in DevTools

2026-06-02T00:00:00Z

Placeholder text is not shown in the standard DOM. To find it through DevTools's element inspection, you have to enable the Shadow DOM.

To do so:

Chrome & Safari

Open DevTools
Click the Settings icon (at time of writing: a cog icon in the top right)
On the Preferences tab (at time of writing this is open by default), scroll to Elements
Check the User agent shadow DOM checkbox
Close the settings pane
Inspect the input field
Expand the input field and expand the #shadow-root node below it
Click the pseudo-attribute inside it

Firefox

Open DevTools
Click the Settings icon (at time of writing: triple dots in the top right)
Click Settings
Under Inspector, check the checkbox for Show Browser Shadow DOM
Inspect the input field
Expand the input field and expand the #shadow-root node below it
Click the pseudo-attribute inside it

Resolving a unique id constraint violation in PostgreSQL

2024-07-11T00:00:00Z

If you're working with PostgreSQL and get a unique constraint violation, the first question you have to ask yourself is whether you've executed any SQL queries outside of your application.

Possible error messages

Express app using prisma as ORM:

{"message":"Failed to create category","error":"\nInvalid 'prisma.category.create()' invocation:\n\n\nUnique constraint failed on the fields: (id)"}

Or directly in PostgreSQL:

ERROR: duplicate key violates unique constraint

Problem

You have a table with existing rows of data.

You're trying to add 1 or more additional rows.

PostgreSQL thinks the next available row has id=#, let's say id=4.

However, you already have a row with id=4. Perhaps you earlier manually updated the database via psql or the PgAdmin UI or some batch job ran, but now you're trying to add data through an application you're building.

Therefore, conflict!

Solution

Update the sequence. The sequence is a number generator, providing unique identifiers for your table rows.

Begin by starting a shell with psql for your database and then let's update the sequence for the affected table.

First, figure out what the highest id currently is, so we know where the sequence should be starting.

Example for a Category table:

SELECT MAX(id) FROM "Category";

If we get a 4 from this, we'll want to start the sequence at 5:

ALTER SEQUENCE "Category_id_seq" RESTART WITH 5;

Next time you're inserting data, it will start at row id=5.

Notes

This is assuming you're auto-incrementing your ids. Example for a prisma schema:

model Category {
  id        Int        @id @default(autoincrement())
}

You'll want to do all your future updates via the application you're building.

Differences in test automation for desktop versus web

2024-06-17T00:00:00Z

What can you expect when you're used to writing test automation for web applications and may now be involved in writing tests for desktop applications?

Desktop automation

You may have to deal with a splash screen. Solution: poll until you can access the main application window.
Windows versus macOS. Different UI controls and ability to access them. Test frameworks may only support one OS.
Custom UI controls and third-party UI controls. Test frameworks may offer partial or no support.
No headless mode. For headless execution you could use a virtual machine or a display server.
No implicit wait between application controls, you likely will have to implement your own polling.
For element selection, you'd be looking for AccessibilityId or element type + name rather than data-test attributes.
There may be web controls within the desktop application, with data exchanged between desktop and web.
The installer may have many permutations, for different languages, OSes, and different package contents.
Desktop UI automation frameworks are very limited compared to web frameworks, and the most maintained ones are rather pricey.
Slower pipeline execution, due to needing to render all the actual resource-intensive UI and go through native OS APIs.

Note that for many of these considerations, the solution might not involve the desktop UI test framework itself, but rather a separate test harness or set of unit or integration tests.

The economic reality in Lego themes

2024-06-08T00:00:00Z

Some Lego sets (70 from my count) contain a price list of some kind, indicating how much Lego currency a minifig can expect to pay for an item.

Let's call this Lego currency "Legu" for convenience, faintly referring to the ecu (Wikipedia), the pre-cursor of the euro. In the Lego games, the currency is called "Studs", but the bills and coins in actual Lego sets do not resemble these Lego game studs, so we'll invent "Legu" for the purposes of this article.

I've identified all in-universe pricing in Lego sets from 1949 to June 2024 and made the full list of in-universe prices available on Kaggle, the data science platform.

What can we learn from these prices?

We will discover that:

Lego themes interact with each other through trade
The Duplo theme dominates the agricultural sector
Hidden poverty is rampant in the City theme
Most City residents sleep in a prison cell or hospital ward
The Castle theme is a Mad Max-like dystopia
Talent shows are all-important in the Friends theme
Ice lollies are everything to ninjas
Non-perishables only arrive with new sets

^{_{Lego set 41704 Main Street Building by The Lego Group, used under Lego’s Fair Play Policy for non-commercial purposes.}}

Lego themes interact with each other through trade

Prices of items that are sold (with price list) in multiple Lego themes:

Item	Price	Theme
Apple	5.00	Elves
Apple	3.00	Castle
Apple	3.00	Friends
Apple	2.00	Duplo
Apple	1.99	Juniors
Carrots	3.00	Friends
Carrots	1.99	Juniors
Croissant	7.00	Friends
Croissant	1.00	City
Croissant	1.00	Elves
Cupcake	2.00	Elves
Cupcake	1.00	Friends
Ice cream (one scoop)	10.00	Ninjago
Ice cream (one scoop)	2.00	Friends
Ice cream (three scoops)	20.00	Ninjago
Ice cream (three scoops)	3.00	Friends
Ice lolly (green)	3.00	City
Ice lolly (green)	1.00	Friends

There's too much variance within a very small set of items to definitively determine the most expensive themes to live in, but if we take this list of items that have prices in multiple themes, and we make the available data more meaningful by excluding themes that only have a price for one item (Fabuland, Disney, Creator) and by excluding the uncommon non-food items, and we rank the themes by whether they're more expensive or less expensive than other themes for the same item, we end up with this ranking:

Ninjago (most expensive)
Castle
City
Elves
Juniors
Friends (least expensive)

If we now compare that to the number of agricultural production sets (livestock farming, poultry farming, fruit & vegetable production) in each theme, we get:

Theme	Cost ranking	Number of "farms"
Ninjago	1.00	0
Elves	1.33	0
City	1.50	5
Friends	1.71	4
Castle	2.00	0
Duplo	3.00	26
Juniors	3.00	0

Why are food prices low in the Elves and Juniors sets despite having no agricultural production?

The Elves have more than meets the eye. They grow pumpkins, they grow candycanes, their magical trees grow bizarre fruits we humans don't even have names for, and they have magic. The Elves will not grow hungry. What they don't have is regular apples, for which they're willing to pay a high premium.

The Juniors theme is a different story. They are a mess. They have pirates, dinosaurs, Spiderman, all sorts of wacky characters. All of them eat regular food, but there is no food production. Instead, the Juniors theme has a close relation to the Duplo theme, in that both are targeted at younger children. That means bigger blocks. But most importantly, it means anarchic creativity. Take a look at set 10422 3-in-1 Space Shuttle Adventure.

^{_{Lego set 10422 3-in-1 Space Shuttle Adventure by The Lego Group, used under Lego’s Fair Play Policy for non-commercial purposes.}}

The space shuttle has a winch and a propeller on the outside. It is not skyworthy. But, like the Warhammer 40k Orks (StackExchange), there is a cheerful playful belief that things will work regardless and — for Duplo figures and Orks alike — that looks to be true.

Meanwhile in Juniors, you have 10676 Knights' Castle (Brickset), where a catapult is brought to a castle siege despite the castle having an open gate that cannot be closed, and you have 10764 City Central Airport (Brickset), featuring a jet without a pressurised cabin. Duplo and Juniors are aligned, friendly, and thus trade comes easy.

The Duplo theme dominates the agricultural sector

The Duplo theme has many, many farms. Much of that production goes towards their own massive world, including the many zoos, but some of it goes towards Juniors. Duplo and Juniors are united against other themes in preserving their psychic gestalt of bringing imagination to life, as opposed to the more straight-faced and dangerous themes (especially Ninjago, with its relentlessly training ninjas and many barely contained threats), so Duplo limits trade with themes like Ninjago, City, and Castle as much as possible.

This necessarily gives Duplo huge power over those themes that are dependent on food imports.

A different way of looking at the Lego themes

But is this how it works?

So far we've considered each theme as comprising a collection of a single instance of each set in its theme. However, in the wild, Lego sets usually appear in people's homes, in all sorts of configurations. There might be only 1 or several sets from a theme, and sets from other themes, or there might be bricks from multiple themes all mixed together and forming new creations.

That means a price sign showing a croissant costing 1 Legu might be placed in a space bakery with no grain-supplying planet to be seen anywhere. If The Lego Movie (2014) is canon, then the Lego world is a manifestation of the imagination of whoever plays with the Lego. Anything could be real. The amount of agricultural production is irrelevant, trade agreements are irrelevant, inflation is irrelevant, prices are just numbers.

That would mean these in-universe price lists I've collected are irrelevant. A child (or adult!) can decide whether someone needs to pay 1 Legu for a bouquet of flowers, or 1,000,000 Legu, or whether they can pay with a hug, or can trade for a live crocodile, or whether no-one has to pay for anything.

In that case it would still be interesting to think about where the prices come from. In the Platonic ideal of the Lego sets as designed and placed in various Lego themes, why are the prices what they are?

So, let's think about what else must or could be true in the Lego universe as designed.

Hidden poverty is rampant in the City theme

City sets have no panhandling, no dilapidated buildings, no vacant storefronts. There are no obvious signs of poverty. However, City has rampant crime. There are 2 prisons, 3 prisoner transports, 4 police stations, 5 mobile police labs, and... 3 houses.

^{_{Lego set 60419 Police Prison Island by The Lego Group, used under Lego’s Fair Play Policy for non-commercial purposes.}}

There are also 4 hospitals, suggesting high levels of serious and/or chronic illnesses. There are also no schools at all, except for — notably — the Police Training Academy. There is low education and low chance of advancement in the City theme, with the vast majority of residents working jobs that — judging by the human world — have low pay.

All signs point towards a world where poverty is a huge problem, but enormous amounts of public funds are put towards suppressing the symptoms instead of improving the minifigs' circumstances.

Most City residents sleep in a prison cell or hospital ward

As of June 2024, there are a total of 547 City residents. There are 3 houses. That clearly cannot fit, or it would get extremely uncomfortable. So where do the other residents sleep, once they're done with their jobs and crimes? The only sleeping arrangements available for them are prisons and hospital wards. Surely they can't all live in prisons and hospitals, since there are so many roadworkers, mechanics, binmen, etc. doing work all across the Lego world? Well, the enormous fleet of police vehicles and ambulances suggests a reality where City residents are brought to their jobs in the morning and returned to prison and — for as far as they're ambulant — the hospital in the evening.

It's a sad reality. Of course, if The Lego Movie is correct, the minifigs only need to sleep when a child (or adult) is playing with them and decides that they need to sleep. If no-one is playing with them, they do not sleep where they are, they simply are lifeless. Therefore, no sleeping arrangements are needed.

The Castle theme is a Mad Max-like dystopia

Food is extremely scarce in the Castle theme due to low agricultural production. Powerful food hoarders vigorously defend their castles against constant attacks by famished groups. The market village, with a small amount of food available, requires armed guards. Where the Duplo and Friends themes are the safest, the Castle theme appears to portray class inequality in its most extreme form, a fever dream of unending violence between the haves and have-nots.

Talent shows are all-important in the Friends theme

Why do the Friends characters seem so intensely dedicated to music (Andrea) or sports (Stephanie) or art (Emma)? The answer is talent shows:

Item	Price	Category	Set	Theme	Subtheme	Year
Baking show prize	100,000.00	Prize	41393	Friends	Baking	2020
Talent show prize	100,000.00	Prize	41368	Friends	Music	2019

That's 100,000 Legu. What can you buy in the Friends theme for 100,000 Legu?

You can buy:

33,333 pizzas with tomato, cheese, olives, and extra mozzarella
25,000 flower bouquets
20,000 vintage handbags
10,000 jet ski rides
2,000 sunglasses

That's more than 90 years of having pizza every single day.

Now the obsessive pursuits of Lego mini-dolls make more sense.

Ice lollies are everything to ninjas

For that half of the 198 items that were easiest to compare to real-life equivalents, I've compared the in-universe prices to the prices in Billund, Denmark (or, when not available, in nearby towns or in Copenhagen), based on their local supermarkets, the local sushi shop, the local pizzeria, and so on.

Example: an avocado costs 8.17 krone (DKK) in Billund, and 5 Legu (LGU) in HeartLake City.

Let's first look at the top 5 items which are cheapest in the Lego world, relative to the real world:

Item	Category	Theme	Price (Legu)	Price (DKK)	% of real price
Flowers	Home & Garden	Castle	2.00	200.00	1.00%
Flowers	Home & Garden	Friends	4.00	200.00	2.00%
Yacht ride	Recreation	Friends	15.00	750.00	2.00%
Bicycle	Recreation	Friends	100.00	4,079.00	2.45%
Chips (en-US: fries)	Food	Friends	1.00	40.00	2.50%

Flowers are dirt cheap. Friends has loads of them available, but Castle does not. Perhaps it was a one-time sale in the Medieval Market Village, due to the danger of having anything worth having outside of an actual castle. Boats and bicycles are also very common in the Friends theme. Chips are a bit more puzzling. The potatoes likely come from the farms, but potatoes are rarely seen in Lego sets. So far, from analysing every food item that has ever occurred in any Lego set up to 2015 so far, I know that potatoes have appeared twice, both times in 2002 on a plate that also features cauliflower and peas.

Now let's look at the top 5 items which are most expensive in the Lego world, realtive to the real world:

Item	Category	Theme	Price (Legu)	Price (DKK)	% of real price
Ice lolly (red with chocolate dip)	Food	Ninjago	10.00	3.00	333.89%
Ice lolly (orange or purple)	Food	Ninjago	7.00	3.00	233.72%
Carrot	Food	Friends	2.00	1.00	200.00%
Apple	Food	Elves	5.00	2.75	181.82%
Ferris wheel ticket	Recreation	Creator Expert	100.00	60.00	166.67%

We've already talked about apples in the Elves theme. They can't grow them themselves and import them from Duplo at a premium. The price of carrots in Friends is an odd outlier. Carrots are in fact produced in the Friends theme itself, there's no reason they should be this expensive. It's possible that they're just incredibly delicious to Friends mini-dolls and that production can't be increased to meet demand since that would require a new farm set to be introduced to the theme. Another possibility is that carrots have become part of a speculative frenzy akin to the Dutch tulip mania (Wikipedia) of the 1630s.

Next we have the ferris wheel. Either it's a rare joy the Lego minifigs are happy to pay a premium for, or there's a taxation on fun (unlikely in a Lego world), or Creator Expert has higher prices overall. Let's check that.

Prices in the Creator Expert theme:

Item	Price	Category	Set	Subtheme	Year
Sushi (hosomaki)	3.00	Food	31131	5 In 1	2022
Sushi (onigiri)	6.00	Food	31131	4 In 1	2022
Noodles with egg, onion, and leek	8.00	Food	31131	3 In 1	2022
One ferris wheel ticket	100.00	Recreation	10247	Fairground Collection	2014
Two ferris wheel tickets	200.00	Recreation	10247	Fairground Collection	2014

To compare this to other themes, sushi also appears in Friends and Ninjago. The prices are slightly cheaper than in Ninjago, and slightly more expensive than in Friends. Creator Expert does not appear to be unusually expensive.

Now let's compare the ferris wheel ticket price to other attraction prices across the Lego world.

Item	Price	Theme
Jet ski (rent)	8.00	City
Jet ski ride	10.00	Friends
Movie ticket	50.00	Advanced Models
Ferris wheel ride	100.00	Creator expert
Rollercoaster ride	30.00	Friends
Speedboat (rent)	6.00	City
Surfboard (rent)	4.00	City
Ticket (amusement park)	5.00	Fabuland
Yacht ride	15.00	Friends

For the price of a ferris wheel ride in Creator Expert costs you could go on a rollercoaster 3 times in Friends, or have 10 jet ski rides.

It's a nice ferris wheel, to be fair, and tall! According to Lego, the ferris wheel is 60 cm tall, whereas a minifig is 4 cm tall. That means it's as high as 15 minifigs stacked on top of each other.

For comparison, the classic ferris wheel in Tivoli, Copenhagen is 18.7 m high and the average height of a Dane (male or female) is 1.74 m. That means the ferris wheel is as tall as ten and three quarter Danes (the people, not the dog) stacked on top of each other. What an image.

So the Lego ferris wheel is relatively higher. Perhaps the ferris wheel is just worth the price. It's also possible that it's just convenient to charge 100 Legu, since 100 Legu bills are the only Lego bills and the coins of 10, 20, 30, 40 Legu were discontinued, but not many items are actually charged at 100 Legu. The only ones are:

Item	Price	Theme
Bike	100.00	Friends
Camera	100.00	Friends
Christmas tree (small)	100.00	Advanced Models
One ferris wheel ticket	100.00	Creator Expert

In the case of the Friends theme, the sets show that mini-dolls can actually pay by phone through contactless payment, so bills aren't that relevant for them.

Now let's get back to the ice lollies. These are, relatively, the most expensive. We can see that's the case in the Ninjago theme, but what about themes that are not dependent on food imports?

Item	Price	Theme
Ice lolly (blue)	2.00	Friends
Ice lolly (champagnebrus or pink)	2.00	Friends
Ice lolly (champagnebrus, lime with chocolate dip)	2.00	Friends
Ice lolly (champagnebrus, lime with chocolate dip)	1.00	Friends
Ice lolly (green)	3.00	City
Ice lolly (green)	1.00	Friends
Ice lolly (pink)	3.00	Friends
Ice lolly (pink)	2.00	Friends

With an average price of exactly 2 Legu, that means with a typical Danish price of 3 krone, it seems like ice lollies are sought-after in the Lego world, but absolutely life-affirming in the Ninjago theme. Ninjas love ice lollies, who knew?

^{_{I asked ChatGPT to generate an image of a ninja eating a typically Danish champagnebrus ice lolly.}}

Non-perishables only arrive with new sets

There are farms, but no factories. Where do the cameras (100 Legu) come from? What about the nail polish (12 Legu)? Or the hairspray (3 Legu)? There is only one possible explanation. Perishable items have to be produced, but non-perishables arrive in the Lego world with new Lego sets. The items that are introduced are the only ones the Lego residents will have at their disposal. You can imagine how delighted the existing residents are when new items arrive.

The problem is that this also means that new characters can arrive that are identical to existing characters. For example, Friends sets have a cast of characters which appear in many sets. This would be less of a problem in a child's way of playing, where either the clones get thrown aside or the clones become twins or friends that simply look alike. But in the Lego world's platonic ideal it wouldn't make sense.

Conclusion

It looks, more and more, like The Lego Movie is indeed the reality of how the Lego world works. Price lists are created to roughly reflect real life, enough to engage the imagination, nothing more.

What matters is not that a Danish champagnebrus ice lolly costs 1 or 2 Legu depending on whether you're at a water park or a funfair, what matters is what crazy shenanigans happen at the water park or funfair.

This entire exercise of categorising and analysing price lists was a President Business activity. The full dataset is publicly available on Kaggle, but this President Business is going to build something with his nearest tiny human.

The paradox of efficiency in risk-based testing

2024-05-10T00:00:00Z

But Wilco, how can risk-based testing be more efficient when it requires an analysis of what areas of functionality could have high risk or high impact?

This is a very reasonable question.

Risk-based testing

Risk-based testing means you focus the testing effort where it's most likely to pay off, to identify issues we're going to care about most, since they're most likely to occur and most impactful when they do.

Is it really efficient?

Risk-based testing seems to imply a comprehensive upfront analysis of where issues are likely to occur and what impact they are likely to have when they do occur. Indeed it's not an uncommon approach and ends up with a satisfyingly thorough and structured overview which, however, undoes the efficiency the risk-based testing could have had.

Alternatives

It doesn't have to be that way. You could of course try and reduce the thoroughness of the risk assessment by - for example - identifying all scenarios you can think of ahead of time and get the person most intimately acquainted with the area of functionality to give a simple Low/Medium/High/Critical estimation of risk and the person most intimiately acquainted with the users to give their simple Low/Medium/High/Critical estimation of impact, but in the dynamic environments I like to work in it gives more return on investment to be less formal and use the strengths of experienced test specialists and collaborative teams:

In conversation with the team (whether it be a mobbing/ensembling/teaming/pairing session or a Scrum ceremony or something else), ask questions like "what absolutely must not fail?" (impact) and "what do we feel least confident about?" (risk).
Use your prior experiences and intuition as a test specialist to gain suspicions that you can then investigate.
Use the increasing real information about the system under test while testing to refine your informal risk and impact assessment and guide your continued testing.

Add a second occlusion area in a tileset tile in Godot

2024-04-27T00:00:00Z

Tested with Godot 4.2.1.

In a collision layer, you can paint a second collision area with the 'Add polygon tool'. However, in an occlusion layer adding a second collision area removes the first one.

To work around this, you can add a second occlusion layer:

In TileSet -> Paint, paint the first occlusion area on the tile on Occlusion Layer 0
Save the changes to the scene
Under the TileMap node's properties, open the Tile Set properties
Under Rendering, click Add Element
In TileSet -> Paint, paint the second occlusion area on the tile on Occlusion Layer 1
Save the changes to the scene

The early modern period of European history

2023-09-03T00:00:00Z

There is no weirder, wilder time and place than Europe from about 1400 to 1800 CE.

Poland-Lithuania has an elective monarchy. They choose their king, and not just from a single family. The high drama of inheritance is subverted by gathering thousands of nobles together (who themselves did inherit their titles) and having them play out lots of small dramas until they select one of the candidates to rule them. I imagine myself sitting on an earthen wall watching the wheeling and dealing, the outfits, the tents and wagons and horses. Much more fun than watching a king's sour-faced response to having sired another daughter.

The Holy Roman Empire contains an absolutely dizzying number (roughly 1800 at one point) of bishoprics, principalities, duchies, free cities, and other semi-sovereign authorities. Many are non-contiguous, with enclaves and exclaves littering the empire. Microscopic states with different measurements of weight and distance, different times, different coinage, different laws, different privileges. Somehow it keeps working for centuries. It's a bizarre, wonderful, and powerful mess. With so many noble families, it's also no surprise you find many Germans in foreign dynasties. Georg Ludwig, ruler of the Electorate of Hanover, becomes George I, King of Great Britain and Ireland. Sophie von Anhalt-Zerbst, from the ruling family of the Principality of Anhalt, becomes Catherine the Great, Empress of Russia.

Castile and Aragon band together under the power couple Ferdinand & Isabella, who complete the Reconquista - pushing the last Caliphate out of the Iberian peninsula - and fund Columbus' expedition to find a western trade route to Asia, which will fail "happily", soon leading to the Pope dividing the world into land that's known, land that belongs to the Portuguese, and land that belongs to the Spanish. The age of colonisation and vast trade empires starts here.

Instead of a united Italy, there are the rich city states like Florence, Milan, and Bologna, as well as the pragmatic and hypercapitalist maritime trade republics of Genoa and Venice, along with the influential Papal States, and the contested Kingdom of Naples, all huddled together south of the Alps. Art, literature, and science flourish. So does political warring and scheming.

The Reformation focuses Europe's lively religious debate, including a Thirty Years War during which an estimated 20% of the European population dies prematurely (though mostly to disease and famine). Henry VIII invents the Anglican Church so he can divorce his wife.

The Mongol Empire has shrunk back and disintegrated into smaller independent hordes, with the Russian principalities regaining independence only to get absorbed by the Grand Duchy of Moscow into what will become almost a reverse Mongol expansion.

Denmark, Sweden, and Norway have an on-again, off-again relationship in the form of the Kalmar Union, with the balance of power shifting increasingly away from Denmark and towards Sweden except for a (for Sweden) disastrous Great Northern War.

Everywhere in Europe, medieval kingdoms and other small territories combine into larger entities which become global powers. The Lowlands revolt against Spain and form the United Provinces of the Netherlands. England, Wales, and Scotland from the Kingdom of Great Britain. The Kingdom of France expels the English from the continent and incorporates its last feudal territories. None of them are satisfied with this success and start covering the world's oceans with cannons on wooden platforms.

France and England fight the Hundred Years War for hundred-and-sixteen years, with chivalrous knights fighting chivalrous knights, archers fighting men-at-arms, light cavalry fighting buildings, and England's War of the Roses would end up inspiring a brutal and unsatisfying hit TV show. However, the real game of thrones is played by Austria's Habsburg dynasty, who use every trick in the book to expand their dominions.

The Ottoman Empire moves from dramatic siege to dramatic siege, puts the final nails in the coffin of the Byzantine Empire (the last remnant of ancient Rome, though Greek), gets on the bad side of Vlad Dracula, assimilates Christian child slaves into its armies as its elite household troops, and in a relentless drive to the heart of Europe knocks on the doors of Vienna, where the wave finally breaks and rolls back.

That's not even mentioning the Hanseatic League, the Burgundian Succession, the French Revolution, the Enlightenment, and so much more. These are dull words hiding volumes of personal and collective tragedies and successes, the long term consequences of which we face every day. You could throw a dart at a map of Europe, pick a random year between 1400 and 1800, and you're guaranteed to land on a town or region at a time when they're going through a decisive and fascinating part of their history.

No, you shouldn't NOT use unit tests

2023-08-18T00:00:00Z

How valuable are unit tests?

After more than a decade of "they're useless!" and "they're essential!" articles and discussions, we're no closer to an answer. That's not because it's one of those nuanced practices where we need to consider the context of the application, the codebase, the specific functionality, the team's skill, the business objectives, the stage the business is in, and so on. We should, but that's not where people disagree. Rather they seem to disagree due to miscommunication.

Semantics

A unit does not necessarily mean a class or a function or method. In a sociable unit test, you can test a functional unit that might be a class method which depends on several methods in other classes instead of mocking them out.

In some discussions, two people have been arguing for that exact kind of test that's fast and meaningful and maintainable and easy to debug while one of them calls it integration testing and the other calls it unit testing.

Purpose

Sometimes the argument against unit tests is that they don't give sufficient confidence about the system. That's not what unit tests are for. That's why you don't just have unit tests, but tests at all levels depending on the balance of how much information they give you about the system versus how costly it is to gain that information and act on it.

Return on investment

One person will claim unit tests are a waste of time and another that they've saved them so much time. Who is right? Well, both of them.

You can easily make unit tests a maintenance horror if you write as many of them as you can. You can also make them save time by writing them for functional logic that has to work and for coverage of many permutations of input.

You can make unit tests a waste of time if you don't know what a good test looks like. You can make them save time by writing fast, isolated, deterministic, highly readable tests for loosely coupled, testable code.

Conclusion

I see unit tests as valuable and I think anyone would if they shared an understanding that unit tests can be light integration tests, if they understood the comparative value versus other types of testing, and if they've come to understand the difference between a good unit test and a bad unit test.

However, I suspect that in another decade the same argument will still be had.

How to disable spell check in Word online (Office 365)

2023-08-11T00:00:00Z

Disabling spell check in Word online is not very intuitive.

If the display language is English:

Sign in to Office 365
Open any Word document
Navigate to the Review tab
Hit the arrow next to the Editor icon* to open the dropdown menu
Hit Set Proofing Language...
Check Don't check spelling or grammar
Hit OK

*The icon looks like an "abc" with checkmark, previously looked like a pen, and may have changed again when you read this.

Als de weergavetaal Nederlands is:

Log in bij Office 365
Open een Word document
Ga naar de Controleren tab
Druk het pijltje naast het Spelling en grammatica icoon* om het menu te openen
Druk Controletaal instellen...
Vink Geen spelling- of grammaticacontrole uitvoeren aan
Druk OK

*Het icoon ziet eruit als een "abc" met vinkje, en was daarvoor een pen, maar zou ondertussen weer veranderd kunnen zijn.

Collaboration techniques in test & development

2023-08-04T00:00:00Z

3 Amigos
Bug bash
Ensemble programming
Ensemble testing
Feature kick-off
Mob programming
Pair programming
Pair testing
Software teaming
Swarming
Variations

3 Amigos

Product + Dev + QA discuss work increments

Does not involve the whole team
Make the acceptance criteria SMART
Uncover missing or extraneous acceptance criteria
Make risks explicit which are not covered by the acceptance criteria

Bug bash

The whole team performs exploratory testing together

Could choose to do one before a major release
Should not be done on untested code or code with many known issues
Clearly communicate the scope of the bug bash
Assign people to areas of functionality or type of risk
Make it double as a fun, cross-functional bonding experience

For a variation, see Ensemble testing.

Ensemble programming

The whole team working on one thing at one keyboard

Advantages

Best of all your abilities
Better code quality
More diverse design options
Issues resolved more quickly
Knowledge transfer
Shared course of action
Continuous code review
Higher satisfaction
Builds better communicators

Disadvantages

Can be slower than the whole team working in parallel
Requires interaction with many others
- active involvement
- good communication
- empathy
Doing it remotely is difficult

What a Driver can do

Ask to hear from voices they’re not hearing much from
Point out code that’s long / complex / misnamed / duplicated
Ask for more (or less) detail about what to type
Suggest an improvement for the ensemble’s consideration
Try to use a new IDE feature, shortcut, or tool that could help
Celebrate a good result, elegant solution, or helpful contribution
Give extra support to contributions of more junior team members

What a Navigator can do

Filter ideas and suggestions into concrete things to type
Suggest unit test scenarios and using Red, Green, Refactor
Find & share relevant information from documentation & web
Point out repetitive tasks that could be automated
Get ideas from the group, especially from quieter voices
Amplify the perspectives of non-programmers
Move a stalled discussion forward by making a bad suggestion
Listen actively, ready to pitch in when needed
Ask questions to make sure you know exactly what’s happening
Draw out a possible solution or alternatives on a whiteboard
Record notes about things to remember or investigate

Resources

A day of Mob Programming - Woody Zuill (YouTube)
Remote mob programming guide (external website)
Mob Programming - the Good, the Bad and the Great. (external website)

Ensemble testing

The whole team joins in a test activity

Compared to ensemble programming

Also involves the whole team
Involves the same techniques
Can be used as an exploratory bug bash
Can be used to improve or write test automation

Feature kick-off

Whiteboard technical solution / discuss how to test

Compared to ensemble programming

Does not involve the whole team
May whiteboard possible solutions
May define a back-end/front-end contract
Generate test scenarios and how to test them

Mob programming

See Ensemble programming

Pair programming

Two developers working on one thing at one keyboard

Advantages

Best of both your abilities
Better code quality
More diverse design options
Issues resolved more quickly
Knowledge transfer
Shared course of action
Continuous code review
Higher satisfaction
Builds better communicators

Disadvantages

Can be slower than working on two things in parallel
Requires interaction with another human being (active involvement, good communication, empathy)
Synchronous, requires a call or sit-together

How to start

Set expectations low. It’s like paddling a tandem canoe. It’s easier to keep a canoe straight with two people paddling, but it will take time to learn to work well with another person.
Be physically at the same desk. You can pair program remotely just fine, but preferably not on the first try.
Agree on how you’re going to do it before you start doing it. Which pairing style? Switch roles? Stop when? Pause when? Split tasks and return to pairing when?

Resources

You Must Be CRAZY To Do Pair Programming - Dave Farley (YouTube)

Pair testing

A developer and a tester testing at one keyboard

Advantages

Efficient test-fix-retest cycle
Creates developer-tester rapport
Spreads test techniques & thinking
Knowledge sharing (development)
Knowledge sharing (product)

Disadvantages

Requires interaction with another human being
- active involvement
- effective communication
- empathy
Synchronous, requires a call or sit-together

How to start

Try the strong pairing technique first, where the person most skilled in testing provides test scenarios, ideas, questions to the person at the keyboard.
Pair early, before even code reviews have been started - it will make up for potential retesting time after code review comments
Measure cycle time for similarly sized tickets when they are versus when they are not pair tested

Resources

Styles of Pair Testing - Maaret Pyhäjärvi (YouTube)

Software teaming

See Ensemble programming

Swarming

Temporarily have the whole team working on one urgent item

Compared to ensemble programming

Also involves the whole team
Temporary
Less structured

Variations

For pairing on large tickets

You can go async, for example for bigger bug fixes or compatibility testing, and return to working together.
You can loosely pair, keeping audio open (or sitting at adjacent desks) until converging again.

For pair programming

The Navigator telling the Driver exactly what to type versus the Driver explaining what they’ll do next
Regularly switching roles versus not switching roles
“For an idea to go from your head into the computer, it MUST go through someone else's hands” versus 2 keyboards, 1 problem
Fixed schedule of breaks and rotation versus freestyle
One person writes tests, the other makes them pass (using the Ping Pong approach or with fixed roles)

For ensemble programming regularly

You can do it sometimes, such as one day per Sprint or for specific tickets that involve the whole team.
You can do it always, with individuals stepping out regularly for relief from so much interaction.

Getting time to add unit and integration tests retroactively

2023-06-23T00:00:00Z

In How to handle technical debt I explained how teams that have a handle on repaying technical debt actually make it work. Now I want to go into more detail specifically on getting the following done:

Move time-consuming or frustrating manual regression checks to unit and integration tests
Move time-consuming or unreliable end-to-end UI-level checks to unit and integration tests

Context

A cross-functional product team that has been active for some time and spends a non-trivial amount of time in their development cycles running UI-level checks and/or performing manual regression checks. The team wants to be more efficient and replace these with earlier, faster feedback. However, they are not able to just go and do it. Some teams are and don't need any of this help. Others do, such as teams that spend 100% of their capacity on feature work.

Summary

Here's what you do:

Identify only the worst check you want to move to a lower level of testing.
Describe the technical task in your issue/project management system.
Justify it by comparing time saving to the time investment.
Chase the task's planning/inclusion into the team's development cycle.
Rejoice, because now you do have the time to move tests to lower levels.

Details

Identify it

Here's what a team might be tempted to do: document & evaluate all their current manual regression checks and/or end-to-end UI-level checks for how long they take to execute versus the value they bring, discuss for each check whether they can be adequately covered by unit and/or integration tests, create a plan that specifies when and how each test should be moved to what level, create a workgroup to start handling all that work, and plan regular status meetings. If you have all the time in the world and you enjoy bureaucracy, do this.

Or, do the worst check first, enjoy a slight improvement to everyone's lives, then do the next worst check, repeat.

You also don't have to fully analyse the test suites to find out what that worst check is. For manual regression checks, the person doing the checks will have a keen feeling for which one is the most time-consuming or the most frustrating to set up the test conditions for compared to how likely it is to lead to meaningful issues. For end-to-end UI-level checks, simply check the pipelines and look for the test scenario that takes the longest.

You do want to give a little thought to whether the tests you're moving to lower levels of testing are actually covering meaningful risks, whether there are more meaningful risks that are currently uncovered, and whether the tests you're moving are in fact already covered adequately at lower levels. That should inform which ones you actually move on to the next step for.

Describe it

Create a task. Name it something like "Create unit/integration tests for {functionality}".

Justify it

In the task you can add the justification. This will be useful when promoting it with the team, such as in a backlog grooming / refinement session. So what is the justification?

The most persuasive argument is going to be time saving.

Time saving is obviously attractive to a business. Time is money. It's also easy to wrap your head around and a value that can be expressed in numbers. In your company you might even need to express it in numbers. So that would mean estimating - over some period of time such as a year - the amount of time writing and maintaining the new unit and/or integration test(s) and subtracting it from the amount of time you currently spend running and maintaining the checks that will be replaced. Don't spend a lot of time on this or you're just reducing the time saved. All you need is a rough estimate.

Other major benefits of moving tests to lower levels of testing, such as earlier feedback and more reliable test results, are less easily estimated and less persuasive when you produce the estimates for a single task. If anything, you're going to take away the persuasive force of your justification, so only mention these benefits by name or leave them unsaid until you need them.

Chase it

Once the task has been agreed to be valuable enough, you just have to ensure you or someone else in your team grab it from the backlog, or formally plan it into a sprint if that's how you work.

Rejoice

The team didn't have time! The team wanted the time! Now the team has the time!

The buts

But these new unit or integration tests don't test everything a tester picks up on.

We're only taking explicitly defined test scenarios and automating those. Those already didn't fall under the other things a tester picks up on. Those more interesting risks are the ones the team can continue to focus their brainpower on.

But the time saving can never weigh up against the value of feature work.

That's true if the team works on validated customer requests and on fast market tests of new ideas. There are 2 ways out: 1) You could be wrong and the task is more valuable, because the rest of the work isn't, 2) Promote the long term instead, as not doing one of these tasks might not make a great difference, but never doing them will keep bottlenecks in place and cost the company real opportunities due to a slower time to market. If the effort as a whole is valuable enough, we must do the parts.

But I don't know what unit or integration test should be written instead.

It's enough to know what test scenario is currently covered by a UI-level regression check and/or manual regression check that deserves replacing. Describe that scenario. Then the team can discuss how to adequately cover it at lower levels, or you get ahead of it by sitting with a developer and thinking it out together. You could even rubber duck it with an LLM.

But I care more about a UI-level check that's unreliable than one that's slow.

I'd hope you'd be in a position where immediate problems like a flaky test that makes the pipeline fail when it shouldn't can be tackled immediately by the team. If not, then sure, create a task for the check that would have most return on investment for being moved to lower levels rather than necessarily the slowest one.

Questions to ask during Refinement sessions

2023-05-06T00:00:00Z

Let's assume a common context of Story tickets in Jira being discussed in a Refinement session.

What kind of input can you, as a member of the team, give to make this ceremony worth the time?

If you can't surmise the answers to these questions from the ticket, ask them aloud:

Justification

What value will users get from this?
Is it worth the opportunity cost?
Is this an idea we want to prototype or a request from paying customers?

Measuring success

After release, how will we know we achieved the expected value?
Will we beta test this change through a dark launch?

Specifications

What does [unclear acceptance criterion] mean?
Could we rewrite [unclear acceptance criterion] as [better wording]?
Could we clarify [free text specifications] through a scenario outline table, through Gherkin DSL, and/or through representative examples (Specification by Example)?
Have we covered all relevant happy path scenarios?
Have we covered all relevant unhappy path scenarios?

Readiness

If we tried to start this today:
- would there be any dependencies or blockers?
- would we need to complete any investigation (spike) first?
- do we need any alignment with:
  - the support team who will write help centre articles for this?
  - the marketing team who will update marketing materials for this?
  - the translation team who will provide translations for this?
- could we benefit from whiteboaring and deciding a technical solution?
- would we have all supporting documentation (e.g. lofi UI/UX designs in combination with a UI component library) available?

Testability

To test this, do we need:
- a special test environment and/or mocking solution for a third-party integration?
- production-like test data which we do not currently have?
- instrumentation which we do not currently have?
- to provide the ability to change the application's state?
- to expose one or more custom data attributes?
Which scenarios are we going to cover at the level of an isolated unit test, which at the level of a sociable unit, which at the level of an integration test, and which through a UI-level test?

Why and how to apply pair testing

2023-04-25T00:00:00Z

What

Two people (such as a developer and a test specialist) working together (at one desk or in a call) to perform exploratory testing (not just verification of requirements) on a code change together before the developer picks up a new ticket.

Why

Removes waste in the test-fix-retest cycle
Uncovers implicit requirements more quickly
Reduces context switching
Creates developer-tester understanding and rapport
Spreads test techniques & thinking
Knowledge sharing on the development process
Knowledge sharing on the product

How

Once a code change is ready to be reviewed (and it even pays to do this before code reviews or during coding), the developer should invite a test or product specialist to join them in a call or at their desk. Then the developer (the Driver) demos the change and the test specialist (the Navigator) offers test scenarios and asks questions. The developer executes the tests. Together they fix and retest, with the developer typing and the test specialist guiding.

The Driver and Navigator roles can be switched regularly on a timer. I don't say should, because when teams deliver small chunks of work that can be reviewed in quick sessions, you may not need role switches. You should, as ever, fit your approach to what works for your context.

Traditional pairing

The Driver is on the keyboard.
The Driver generates test ideas.
The Navigator reviews and asks questions.

Strong-style pairing

The Driver is on the keyboard.
The Navigator generates test ideas.
The Navigator reviews and asks questions.

When in doubt, try strong-style pairing. In most cases that will give you more alert and productive sessions.

Practical example

Developer on Slack: "hey man :) want to look at my changes?"

Tester on Slack: (opens a call with the developer)

Developer on Zoom: "As you can see here, if you select service X and press Connect you go into a connection flow through Y."

Tester on Zoom: "What if I already connected X through Y?"

Developer on Zoom: "Then the Connect button should be disabled."

Tester on Zoom: "Should it not change to a Disconnect button?"

Developer on Zoom: "When no service is selected yet, we don't know which service to disconnect."

Tester on Zoom: "But when we select a service, could we not show a Disconnect button for that service?"

Developer on Zoom: "That's possible."

This potentially leads to a brief calling in of a Product Owner or UX specialist, and to further tests and learning. Just the short fictional exchange above might already have been several rounds of playing Jira ticket tennis, perhaps for several days, if it wasn't done through pair testing.

Speech recognition could not start

2023-02-04T00:00:00Z

Problem

When you activate speech recognition in Windows, you get an error like this:

Speech Recognition could not start. Please change the recognizer language in the speech recognition control panel under advanced options

Cause

Having the right language pack installed and selected does not mean you also have its speech features.

Solution

For your display language, install the speech pack:

Go to Settings (not Control Panel)
Go to Language
Select the language you're using as your Windows display language
Scroll to the Speech section
Hit Download

Once downloaded and installed, activate speech recognition again. You may have to completely shut it down and restart it a second time before it works.

Peer dependency conflict from the rollup-plugin-terser package

2023-02-04T00:00:00Z

Problem

You get an error like this during your build step's npm install:

npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR!
npm ERR! While resolving: rollup-plugin-terser@7.0.2
npm ERR! Found: rollup@3.2.2
npm ERR! node_modules/rollup
Failed during stage 'building site': Build script returned non-zero exit code: 1 (https://ntl.fyi/exit-code-1)
npm ERR!   dev rollup@"3.2.2" from the root project
npm ERR!   peerOptional rollup@"^2.78.0||^3.0.0" from @rollup/plugin-node-resolve@15.0.0
npm ERR!   node_modules/@rollup/plugin-node-resolve
npm ERR!     dev @rollup/plugin-node-resolve@"15.0.0" from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer rollup@"^2.0.0" from rollup-plugin-terser@7.0.2
npm ERR! node_modules/rollup-plugin-terser
npm ERR!   dev rollup-plugin-terser@"7.0.2" from the root project
npm ERR!
npm ERR! Conflicting peer dependency: rollup@2.79.1
npm ERR! node_modules/rollup
npm ERR!   peer rollup@"^2.0.0" from rollup-plugin-terser@7.0.2
npm ERR!   node_modules/rollup-plugin-terser
npm ERR!     dev rollup-plugin-terser@"7.0.2" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR!
npm ERR! For a full report see:
npm ERR! /opt/buildhome/.npm/_logs/2023-02-04T13_38_32_938Z-eresolve-report.txt
npm ERR! A complete log of this run can be found in:
npm ERR!     /opt/buildhome/.npm/_logs/2023-02-04T13_38_32_938Z-debug-0.log

Cause

In your project you're requiring the latest version of rollup (3.2.2 at the time of writing), whereas one of your other dependencies (the library rollup-plugin-terser) requires a rollup major version of 2.

Solution

Replace the no longer maintained rollup-plugin-terser with @rollup/plugin-terser. Specifically, create a new branch with the following changes and merge the changes:

npm uninstall rollup-plugin-terser
npm i @rollup/plugin-terser --save-dev
In your rollup config, replace import { terser } from "rollup-plugin-terser"; with import terser from "@rollup/plugin-terser"; (removing the curly brackets matters, as the new package uses a default export).

What are the differences between load, stress, spike, and soak testing?

2023-01-23T00:00:00Z

What do you want to learn about the application?

Each of these questions has a performance test type associated with it:

What is the maximum load the application can handle? ➡ stress test
What are the performance bottlenecks? ➡ load test
How does the application respond to a sudden peak? ➡ spike test
Does performance degrade under continuous usage? ➡ soak test

These terms are not used consistently from one company to the next. You might have someone do a shorter duration soak test and refer to it as stress testing or load testing interchangeably. You're more than welcome to treat the below explanations as the only correct way to refer to these types of performance testing. 😊

Test types

All these tests have in common that you'll need to identify common user flows and simulate them through test scripts.

Stress test

In a stress test you don't just burden the application with a high load, you drive the application to failure.

Decide how you will measure when the application has reached its limit, for example: >= 5% of HTTP requests fail
Ramp up the number of concurrent users until you reach that failure threshold
Note the number of concurrent users at that point

Now you know what load the application can handle before things start going terribly wrong.

Load test

Example approach:

Calculate (see the formula further down the page) the number of concurrent users / virtual users
Set ramping to gently go up to concurrent normal usage and gently down again after 1 hour
Monitor throughput during the test to identify any degradation
Head to observability tools to determine the cause of bottlenecks (repeated transactions? slow SQL queries?)
Run the test again, but this time configure it for concurrent peak usage

Now you're aware of specific bottlenecks you can resolve.

Spike test

Example approach:

Base the spike test configuration on the load test scenario configured for normal usage
Edit it so that after 10 minutes of testing it will go to peak usage with a 5 minute ramp-up, duration, and ramp-down
Edit it so that after 30 minutes, it will go to peak usage with a 2 minute ramp-up, 5 minute duration, and 2 minute ramp-down
Edit it so that after 45 minutes, it will go to peak usage with a 1 minute ramp-up, 5 minute duration, and 1 minute ramp-down
Start the test and monitor error rates, response times, and throughput during those spikes
Head to observability tools to determine the cause of hiccups

Now you know how well the application handles sudden spikes in activity.

Soak test

Example approach:

Base the soak test configuration on the load test scenario configured for normal usage
Extend the test duration to 12 hours, and add in a realistic day/night gentle ramping up and down
When the test has concluded, determine whether any errors or bottlenecks appear later in the test which didn’t occur in the other tests

Now you know whether there are any performance issues which you'd otherwise only discover once a release has been in production for a day.

Concurrent usage calculation

People often overestimate the number of concurrent users they need, because they don't take into account that if you have thousands of users per hour, they don't all spend that full hour actively operating the application.

This is what you need from your user analytics:

For your most representative usage level and peak usage level, what were the number of unique sessions per hour?
Potentially add another level for an optimistic upcoming peak usage level, such as twice last year's peak usage.
The average session duration for the same time period you got the unique sessions per hour from.

The formula:

Concurrent users = (Unique Sessions Per Hour * Average Session Duration in seconds) / 3600

I was taught that multiplication has priority over division, but I've added parenthesis for clarity.

Conclusion

You don't necessarily have to use all of these test types. Simply said they can teach you:

The number of concurrent users at which you expect HTTP requests to start failing for 5% or more of your users
Hardware and software bottlenecks which keep you from performing better at normal and peak usage
How well you’re prepared for sudden peaks in usage (e.g. Black Friday)
What kind of issues you could expect to pop up when the excitement of going live dies down after a few hours

Prerequisites

Have a production-like environment that includes all the expected infrastructure
- Should take into account the potential cost of any quotas you might exceed
- Your infrastructure & InfoSec teams should be warned in advance of the tests (especially stress tests)
- For stress testing, you (or the responsible team) should be ready to restart a failing environment
- The tests shouldn’t be run when anyone else is using the environment
Have a tool which allows you to configure load scenarios AND drive virtual users (e.g. k6)
Design and configure the load scenarios well ahead of time and (safely) test the configuration
Have the load scenarios reflect user flow(s) through the application which you expect to be the most common
- This should include (semi-random) waiting times to reflect real user behaviour, e.g. sleep(math.random(2,4))
- You should be realistic (real flows, realistic behaviour), but not obsessive (all flows, detailed user personas)
Have the ability to monitor (and review logs of) hardware and software metrics
- The performance testing tool used will typically already report the response times, throughput, etc.
- However, you will also want to know what happens to the server (CPU usage, RAM usage, disk space)
- You will also want to understand response times at a deeper level (what’s slow: db queries? frontend rendering?)

Unable to start ssh-agent, error 1058

2023-01-16T00:00:00Z

Problem

When attempting to run eval $(ssh-agent):

unable to start ssh-agent service, error :1058

Cause

The ssh-agent service is disabled.

Solution

Start PowerShell as an Administrator.

Then run Set-Service ssh-agent -StartupType Manual.

Now try eval $(ssh-agent) again.

What are mutation testing, approval testing, and property-based testing?

2022-10-07T00:00:00Z

To-the-point definitions of some less-talked-about types of testing.

Mutation testing

Automatically generate "mutations" of an application under test and run your test suite against it in order to evaluate how capable your test suite is of identifying resulting issues. It serves to test the quality of the test suite (specifically its fault detection and code coverage), not the quality of the application under test.

Notable tool: Stryker

Approval testing

Instead of asserting against expected behaviour, capture any differences between the current output and the previous output and if there are any differences have them evaluated by a human reviewer. An example would be Percy visual regression checks. Think of approval testing as blacklisting changes and assertion-based testing as whitelisting changes.

Notable tool: ApprovalTests

Property-based testing

Instead of writing a specific test case asserting that the application under test has a certain property for a certain input, assert that the property remains valid for a wide variety of inputs. An example of a property in this context could be that the result of a particular calculation should never be a negative number. The test would check that this is true for a wide variety of automatically generated inputs to the calculation.

Notable tool: Hypothesis

Mistakes to avoid when scaling up an engineering discipline

2022-09-30T00:00:00Z

When you're a successful tech start-up experiencing rapid growth, it's tempting to think of scalability as hiring more people, renting more office space, and maturing your HR processes. Where scale-ups often go wrong, understandably, is to not make any changes to the things that made them successful as a start-up. That can cause some problems, sometimes fatal problems.

Not changing leadership roles

The CTO fails to switch from technical excellence to technical leadership and the CEO or COO fails to switch from being a product & sales lead to providing a strong and market-savvy direction and ensuring they take along the right people for that journey.

What a company needs from its senior management changes dramatically as it scales up and so do the skills that match those new requirements.

As a founder team, that either means learning skills you don't yet have or putting a new C-suite in place. Both very difficult things that make it tempting to double down on what worked in the startup phase and stumble along hoping your competitors don't do any better.

Hiring too carelessly

Hiring people primarily on technical skill is not much of an issue until you start working with cross-functional product teams. Then the way in which people communicate, the ways they do (or do not) collaborate, and the way they adjust their approach to the context of a fast-moving scaleup, all become crucial to whether the teams are able to swiftly deliver high value work.

That means the hiring process will need to include people who are especially capable of asking and evaluating questions which will identify the right fit for a highly collaborative working environment. Asking "can you collaborate well?" won't do it.

It also means not hiring too many permanent employees at once, even if you think you need a lot of capacity fast, as a large influx of new employees can have a disproportionately detrimental effect on productivity and the organisation's culture.

Neglecting infrastructure & DevEx

When you move fast and add more and more developers, small pains in the local development environment, CI/CD pipeline, and reliability are going to become huge headaches that also become increasingly more difficult to resolve.

More employees come in, they all suffer the same problems, they all try to be helpful and create workarounds or suggest solutions, and both the problems and the discussions around solutions are unending and involve more and more people. Structural solutions barely have a chance to gain momentum as they have to compete with delivering value to the customer.

You need technical & product leadership to agree and make a firm choice that internal quality also matters and guarantee a continuous margin to address it. Insufficient expertise should be hired for. Then, these small problems can be addressed before they turn into large ones.

Blindly applying a scaled Agile framework

Using the rulebook of a scaled Agile framework like SAFe can be disastrous in a dynamic, fast-changing environment, especially if the practitioners are convinced a priori that any possible friction or failures would come from not following the rules closely enough.

Sure, if you have multiple cross-functional product teams, dependencies are going to need to be negotiated. Having everyone sit in a room for a two day planning meeting is one way to do that. It demonstrates how people can be made to do wicked, unnatural things if there's enough peer pressure. There are other ways, however. Reducing dependencies is one. A simple Scrum of Scrums is another.

How to approach load and stress testing

2022-07-16T00:00:00Z

Update: In 2023, I've written a more up-to-date article which complements this one.

Definitions

Load test

Examining behaviour of the system under sustained typical or high load.

Stress test

Determining the point at which the application (or a % of requests) fails.

Deciding the environment

Never use: any environment whose owner is not expecting the extra load.
May use: a staging environment matching the production environment sufficiently for your purposes, which may include auto-balancing and auto-scaling configurations.
Ideally use: the actual production environment at a time of consistent low usage.

Deciding concurrent usage

Load testing

Sessions Per Hour x Average Visit Duration (seconds) / 3600.

Make this calculation for normal load (your most typical business day) and peak load (highest traffic in the last year, but add a margin on top depending on what you're expecting to gain from any extra spend on marketing and paid search).

Stress testing

You don't need to know concurrent usage. You will find out the maximum concurrent usage your application and its infrastructure can handle.

Deciding what to test

If you already have baseline results from a previous test which you want to compare against for new changes to the application or infrastructure, use the exact same test scenarios.

If not, either analyse the user flows of actual traffic for a non-typical event (for example previous year's Black Friday traffic for preparing for this year's Black Friday) or analyse user flows for the most crucial journeys through the application.

In the test scenarios (for example defined in JS for k6) define page views and events for the user flows with realistic wait times in between. You could make these random within a range, to reflect the variance of real users.

You'll have to make a judgment call for how closely you'll make the user flows and data reflect the variation and user quirks of your actual traffic, by considering how much that extra realism is worth in comparison to the time spent defining it and testing it does what it should.

Deciding how to test

Load testing

In the tool you're using to drive traffic, add up and down ramping which the application can easily handle and spawn virtual users to the concurrent usage results you got in "Deciding concurrent usage" and have those virtual users execute the journeys you defined in "Deciding what to test". Keep this running for an hour (or longer if you'd like to make this a soak or endurance test).

Review relevant metrics which were gathered during the test and look for bottlenecks. What resources were under most stress? Disk, CPU, RAM, the network?

Stress testing

In the tool you're using to drive traffic, configure a ramping up which will not end. Decide what you will consider the breaking point of your application. For example, 10% of HTTP requests fail with a 500 error. Then either automatically (if the tool allows it) or manually stop the stress test when you reach that point.

Mark the number of concurrent virtual users at this point, and review the most burdened resources (for example, CPU usage) and significant metrics (for example, throughput).

When chasing efficiency is a false economy

2022-07-13T00:00:00Z

In software engineering we're often — and rightly so — looking for ways to be more efficient, to be faster than competitors, to reduce waste. Some of those ways work out really well and some turn out to be false economies where the time or other costs we initially save end up costing us more in the end.

Here are seven examples, with charitable titles to show why they might seem like a good idea.

Anticipating needs

When you're working on a feature to integrate a payment provider, it's tempting to think "we should build this feature to accommodate any future payment providers we may add".

This could be smart, if you suspect that it's going to take six weeks to develop and you know the business needs multiple other payment providers to be added six weeks from now.

If the need is not immediate, though, resist the temptation. Keep it simple. That will get you feedback quicker and it will prevent changing circumstances from making all that work obsolete.

See also the XP principles Do The Simplest Thing That Can Possibly Work and You Aren't Going To Need It.

Parallel work streams

If you have a team of five developers, wouldn't you get the most utilisation out of them by having each developer plugging away at their own piece of work?

Not necessarily. Pair programming, pair testing, ensemble programming, and ensemble testing have immense benefits for immediacy of feedback, the quality of delivered work, knowledge sharing, and team cohesion, against a cost of having multiple members of a development team working on the same thing.

To the uninitiated that cost is all that's visible. "How is it more efficient to have two developers deliver one thing instead of delivering two things?" Firstly, the cycle time is faster when collaborating. Think one thing delivered instead of two almost delivered. Secondly, the efficiency of team members working in isolation is illusory. With isolated work you miss out on the benefits of pair programming, pair testing or ensemble (or mob) programming or testing (PDF).

Deliver value now, refactor later

You really do want to deliver value quickly and unless you're in a safety-critical context (health, manufacturing) or compliance-driven context (government, finance) you can worry about making it better later.

In practice, there are three complications which make this a false economy if they're not accounted for. One, teams don't do this for one development cycle and then do the actual refactoring and iterating later. "Later" never comes, or at best some of it comes in ad hoc fashion over time. Second, hastily coded solutions tend to make a codebase less easy and safe to change. Third, when returning to the code later on you (or by this time, someone else) have to spend time getting fully reacquainted with the implementation and its context.

Consider this and then decide "yes, actually all of that cost is worth incurring for getting this code change out now" (could be a hotfix, could be preventing imminent churn of a major customer, could be a huge revenue builder) or "no, we can prevent all that cost by spending a bit more time now without losing anything".

A fully loaded sprint

A team is heading into a new time-boxed development cycle and their main goal is to deliver the most requested feature by customers, one that's not very exciting but removes a ton of clicks and data entry. However, the team's projected velocity is double what they need to deliver it. They fill the rest of their capacity with various bits and bobs from their backlog. Research, bug fixes, library updates, small feature requests.

A nice full sprint should lead to lots of good things delivered, but does it? What typically unfolds is that the primary goal doesn't feel primary even to the one or two persons doing work related to it. Questions, obstacles, and risks do not get identified or resolved as quickly as they could have.

Contrast that to a sprint where everyone is working towards the primary goal before attending to new work. It feels like a top priority, all communication and agile ceremonies are geared towards completing it, anyone not directly writing code is getting stakeholder input or mitigating risks. Probably fewer tickets will get completed, but the primary goal is actually delivered and with more confidence that it will indeed be what all those customers had been waiting for.

I'm not suggesting a team should work on only one thing per sprint, or even necessarily that the whole team should be working together on one thing at a time, but what I would suggest is that if a sprint has a genuine primary goal that it's treated like a primary goal instead of only being paid lip service to in the form of a "Sprint Goal" written down where no-one looks at it and a lot of the rest of the sprint being filled with low priority work.

Don't Repeat Yourself

Don't Repeat Yourself is probably one of the most well-known principles in software engineering. It makes intuitive sense. Don't code the same thing in multiple places when you could abstract it away into one place and call it from anywhere else. Then in the future you only have to make changes in that one place.

Unfortunately what this can lead to is premature abstraction or indiscriminate abstraction. Coming from the world of software testing, I've seen this a lot in UI test suites.

In premature abstraction, a UI interaction might be abstracted away into a helper function despite it only being used once. That's too early. You're now incurring the cost of the test being harder to debug for a benefit that does not yet exist.

In indiscriminate abstraction, everything is abstracted by default. Take for example the Page Object Model. It can result in beautifully formatted and easy to read specs, but when you're debugging someone else's tests it's not enough to read the specs, you'll now have to continually navigate under the hood between the specs and various supporting files to understand what's going on and whether the issue is in the specs, page selectors, global selectors, page helper methods, global helper methods, or the UI.

This doesn't mean "never abstract" or "never use POM". It means, consider the tradeoffs.

Repay technical debt when it hurts

Some companies reserve a 20% (or higher) "technical debt" margin for all types of technical tasks outside of feature delivery. The argument against this goes that people will then just look for things to do, things that aren't necessarily more valuable than backlog items delivering on the business vision. An efficient way of working would be to do only the most important thing right now.

The practical outcome is leaving technical improvements to be undiscovered or languishing on a backlog. Individually it's hard for these improvements to compete with revenue-driving user-facing features, but over time the codebase, the test framework, the local development environment, and the CI/CD pipeline become increasingly harder to work with and the application slower and/or less secure for end users.

You can be better off actually eating the cost of a guaranteed lower team capacity in order to gain the long term benefit of easy and rapid change and a reduction in cross-functional risks.

Hire more developers

You have three product development teams. You need double the work done. Being a leader who understands that growth requires investment, you greenlight a swift expansion of the teams to a total of six.

This never seems to work. Similar to Brooke's Law, doubling the number of developers doesn't double the amount of work done (or even maintain the current pace). The number of dependencies increase, the number of communication lines increase, there is lots of onboarding to be done, and either you have teams of new joiners who will be slower than existing teams for some time or you mix all the teams and slow everyone down (by less) with severe disruption to the existing team dynamics.

It may be counterintuitive when aiming for fast growth, but integration of new developers should be done slowly and deliberately in order to actually achieve the intended increase in pace (and have it be sustainable pace, too). The trick is to do it before you need them, even if that means lower utilisation until that need arises. If it's too late for that, the focus should be less on manpower and more on making tough decisions on priorities.

Unable to boot to desktop after upgrading Raspbian to Buster

2022-03-22T00:00:00Z

When upgrading Raspberry Pi 3's officially supported OS Raspbian from a version based on Debian 9 (Stretch) to a version based on Debian 10 (Buster), you could find yourself unable to boot to the desktop.

Assuming the boot order is still set to boot to desktop (you can check this with sudo raspi-config), a likely cause is not fully updating the sources lists after the upgrade to Buster.

On the command line, try starting the X Window environment:

startx

At this point, you will likely get some error output containing the following:

Fatal server error:
(EE) no screens found(EE)

There's another thing you can check to determine that we're on our way to the right solution. Open the X Server log with nano /var/log/Xorg.0.log and hit CTRL+W to search for fbturbo. You may then find the message Failed to load module "fbturbo" (module does not exist, 0). Hit CTRL+X to close the file.

To fix the issue, update your sources lists to find packages compatible with Buster, update your packages accordingly, and remove the fbturbo driver (that last step was required for me, I have no explanation for why):

sudo nano /etc/apt/sources.list
Look for stretch and if you find it, replace it with buster
Hit CTRL+X and save the change if you made one
sudo nano /etc/apt/sources.list.d/raspi.list
Look for stretch and if you find it, replace it with buster
Hit CTRL+X and save the change if you made one
sudo apt update && sudo apt full-upgrade
sudo mv /usr/share/X11/xorg.conf.d/99-fbturbo.conf ~

Now you should be able to reboot and be presented with the desktop, or run startx to get there.

How does a test specialist fit into a DevOps and Continuous Delivery team?

2022-01-21T00:00:00Z

Context example

Let's assume a feature team (or a stream-aligned team) which is independently able to deliver value in some form, such as feature updates in a section of an application. The team has strong expertise in back-end and front-end development, software architecture, product management, and UI design. The team automatically merges any changes which pass the pipeline stages directly into the main trunk. Work in progress is hidden from end users through feature toggles.

How does a test specialist fit into this team?

Software quality in DevOps and CD

If the whole team is responsible for software quality, do you need a test specialist? Everyone should build the skill and claim the time to challenge explicit requirements, discover implicit requirements, cover cross-functional requirements, write test automation at the right level, perform skilled exploratory testing, and improve testability and observability. If a team has no trouble doing this and doing it well, they probably don't need a test specialist. In practice, teams without test specialists lack in all of the above. Teams with test specialists sometimes also lack in many of the above. If you are that test specialist you'll never lack for things to do and improve.

So what might a test specialist practically do, on an average day?

Pairing & ensembling

Feedback should come as close to when the code is being written as possible, to reduce delays and to reduce context switching. There are a couple of ways of going about this.

Ensemble programming (also known as mob programming) means completing the entire development cycle for a discrete piece of work as a team. Feedback can't get more immediate than this. Instead of thinking of a solution on your own and having it be examined much later, the whole team is there to discuss or whiteboard it together. Every line of code is written together, getting continuous collaborative code review. A test specialist should be there to challenge the solution, to generate test ideas, to give guidance on how to make the automated test scenarios meaningful.

When the team does not work with ensembling, a test specialist can give the same input on a smaller scale through pair testing. This can happen when a merge request has passed code review — to make it so retesting is reduced — or as soon as a merge request is created — to keep feedback closer to when code is created — or before code has been written — to challenge or rubber duck the solution before it has been embarked upon. As with ensembling, pairing has additional benefits of accelerated knowledge sharing and resolving questions without delay.

Shifting left & shifting right

There are many more opportunities for improving software quality than a "Ready for Testing" column would suggest. Imagine a team is discussing a feature which would allow a warehouse's working hours to be set. A developer asks what the granularity should be for each working day's start and end times. A discussion follows, comparing the consequences of choosing 1 hour to 30 minutes to 15 minutes. An agreement is reached and the team moves on. A tester could have been there to ask a question like "what if the user wants to set lunch hours in addition to just start and end times?" One simple question could prevent a great deal of headache later on.

It wouldn't have to be just one question, of course, nor would they have to be about completeness of requirements. It could also be asking why we think there will be return on investment from this feature. It could also be asking whether we can get more information on the user needs from the users directly. It could also be rewording the requirements to be more lucid, revealing new possible scenarios or edge cases as a side effect. It could also be asking if there's a better way to solve the underlying problem the users want solved. It could also be asking how we're going to prove we've done a good job. It could also be asking about cross-functional risks, to help prevent performance issues or security vulnerabilities and so on.

Test automation & testability

A test specialist could inform and/or do the scripting of test automation. When a tester writes test automation it's typically in the form of UI-level tests which exercise the entire stack, as lower level tests live closer to the code and are more easily created by those also living close to the code. If UI tests are considered appropriate for the team, the test specialist can implement and maintain these tests and optimise them for running as swiftly and as reliably as they can and ensure the information they provide is of value. Aside from UI-level checks, they could also write custom test tooling wherever it would provide more benefit than it would cost to create and maintain.

The tester could also inform the test automation by deciding with the team which happy and unhappy scenarios would make sense to cover at the unit versus integration versus UI level. If the test strategy is for UI-level tests to cover critical user journeys, then they could analyse these journeys. They could also help developers in determining what is a useful unit tests, what is a useful integration tests, are they testing what they are trying to test, what could they add, what could they throw away, what could be more efficient?

Observability & user advocacy

If you have a team with developers, a product owner, and a UX/UI designer, then you have a technical perspective, a business perspective, and a user perspective. With a test specialist you have someone who combines all three in one person. You might argue that everyone in a team should have all three perspectives, which I agree with, but in practice it's difficult to get out of the detailed technical perspective when you're knee deep (or neurons deep) in code unless you're participating in a pairing or ensembling session. A tester should think beyond correctness, about what else will matter to users and weigh those things against the cost of developing and testing against them. A tester should also challenge assumptions about who the user is and what the priorities of the business are.

Then, when a code change is deployed, the team should not stop caring. A test specialist can help ensure that a change continues to be monitored — regardless of whether they ultimately are the one to do so or not — to see what the uptake is, to see if any unpredicted issues occur, to bring back customer feedback to the team, to set up dynamic threshold alerts to be automatically notified if key metrics become worrisome, to create an environment where the team gets the information it needs to make good follow-up decisions after a change.

Exploratory testing

When testing is imagined as verification of explicitly defined functional requirements, then there is neither much joy nor value in testing. Testing should go far beyond that and exploratory testing is one way how. This doesn't change in a DevOps or CD environment. Just because it works best to not have a gated testing phase doesn't mean an end to exploratory testing. As soon as there's a branch for new change you could serve the application locally to test the change in the actual user interface, or you may be able to trigger a test environment for the branch, or you may do it in a pairing session, or you may do it in production. Wherever you do it, there need to be opportunities to uncover unknown unknowns.

This is where a test specialist could use their imagination, their metacognitive abilities, their experience of what they've seen before, and the skill they have hopefully been able to hone for years to efficiently discover all kinds of unexpected functional and cross-functional issues and risks. Their mind should be buzzing with questions and they should be running mini-experiments to answer the ones that they think matter most in the moment, leading to more questions and more mini-experiments. This is skilled work only a human can do. It's also difficult to teach, but valuable for everyone in a team to gain some ability in.

Continual process improvement

Code is not created in a vacuum. The cadence of delivery and its results might be affected by many things. How are business opportunities and user needs communicated and refined? How do teams align with each other? How do teams align internally? How do teams manage and prevent technical debt? A test specialist is perfectly placed to guide such processes to be efficient yet effective. Their broad involvement across the entire software development cycle, their drive for improvement (or, less charitably, their fixation on imperfection), their knowledge of how to deliver software with high quality and all the processes and human dynamics that affect it, it all puts them into a good position to make positive changes.

A lot of this might sound like something for an agile coach to do and/or for the whole team to do through retrospectives. It's certainly not necessary to have these activities done by a test specialist. It should also not be expected that one person has sole responsibility for improving ways of working, but if you have a team who aren't already making regular measurable improvements and no agile coach or a not especially skilled agile coach then the test specialist's role in this becomes crucial.

Quality coaching

A test specialist can help a team form a quality culture where the end goal is not just to deliver as much and as quickly as possible, but to do it in a way whereby you can demonstrate that something of value was done, to do it as a team in which everyone helped to make the end result the best it could be, and to do it with as little waste as possible.

A great coach is someone who seeks to make themselves unnecessary, but is able to intervene at just the right moment with just a few words or a simple question to let a team discover a better way of doing things or bring out the best in an individual. A tester has likely already learned great diplomatic skill due to having to make unpleasant news less annoying many times over, has learned humility through being thought of by some as disposable or redundant, and could leverage their social skills to give a team motivation and even enthusiasm in being critical, writing good tests, clearly structuring code, collaborating well, and thinking of ways to grow justified confidence in their own work.

Conclusion

These are some of the ways in which a test specialist can be of value in a "modern" software development team using DevOps and Continuous Delivery practices. Is it too much for one person? Yes. They will have to do more of some and less of others, prioritising whatever the team needs more of at the time. Or you might go as far as having multiple testing roles, such as one focused on test automation, test tooling, and optimising pipelines, and another focused on whole-team testing, shifting left & right, exploratory testing, and testability and observability. Either way, there's a still a lot more for test specialists to do than merely translating acceptance criteria into UI-level tests.

How to prevent a team from feeling overburdened

2022-01-14T00:00:00Z

Context: Software development teams with an Agile/Scrum-like way of working.

Say you've got a pretty good situation going. Executive-level managers tell the feature teams they have full autonomy and responsibility for their work. The teams are told the company goals, given some problems to solve that could help reach those goals, but the teams are free to come up with better solutions, prioritise their efforts, anything they think is best to reach the company goals. Within the teams, the product owners are fully on board with this as well, with team members having the autonomy to decide what goes into their sprints.

Yet, the feature teams almost universally complain that the workload is excessive, that they feel like they're always just pumping out features, always rushing, that there's no time to do proper design thinking or refactoring, that there's no room to develop themselves at this company.

How could this possibly be? The teams have been told they are autonomous, they've been given the space to plan their own work, yet they feel overburdened.

Imagine an airline which holds safety as its highest priority. It does all it can reasonably do in terms of safety procedures, audits, reports, training, corporate communications. Yet, lots of minor incidents are happening and even a near disaster. When digging deep, you find that the experience of personnel is not that safety is the highest priority. They feel that the informal communications and performance evaluations are geared towards the number of on-time departures and arrivals and the amount of time a plane is out of action. Sometimes when a situation feels trivially worrying, maintenance personnel and pilots alike decide to forge ahead. After all, it would be hard to explain to management why so much time was lost in extra checks for something that most of the time will turn out to be a non-issue.

The point is you can invest lots of time and effort into communicating how you want people to behave, but unless your incentives and your casual interactions don't line up with it and do so consistently, it won't matter.

How do you fix it?

Decide if you actually do trust the teams. If not, you have other problems to fix first.
Be consistent in your communication. You could encourage teams in just the right way for a year, but one "it feels like there's very little output" or "is my favourite feature X done yet?" can undo all of it.
When anyone still complains the pace is unsustainable, ask them what's keeping them from changing that pace. At this point the solution should require only a conversation and the answer will make it clear who need to have that conversation.

Edit a file on startup in Windows

2021-07-23T00:00:00Z

Example use case

Maybe you need a value in a Docker config.json to be updated and persist, but the value resets whenever Docker is closed down.

Solution

Create a startup script

In any text editor or IDE, create a startup.ps1 file.

In the file, add the following (while replacing with the values relevant to you):

powershell -Command "(gc \\wsl$\Ubuntu-20.04\home\wilco\.docker\config.json) --replace 'oldvalue', 'newvalue' | sc \\wsl$\Ubuntu-20.04\home\wilco\.docker\config.json"

gc: Get-Content (get the contents of a file)
sc: Set-Content (write to a file)

Place this file in a convenient location, such as your user directory or C:\tools.

Schedule a startup task

Open the Task Scheduler (Windows key + start typing task, then select it)
Hit Create Basic Task.. (under Task Scheduler Library)
Give the task a name you will recognise later
Hit Next
Choose When I log on
Hit Next
Ensure Start a program is selected
Hit Next
For Program/script, enter PowerShell.exe
For Add arguments, add -ExecutionPolicy Bypass C:\YOURSCRIPTLOCATION\startup.ps1
Hit Next
Hit Finish

-ExecutionPolicy Bypass: you're allowing PowerShell to do anything in this script, nothing is blocked or warned about.

And there you go. Check the file has the old value, reboot, and you will find the file is now updated.

It is the best of times and the worst of times for a test specialist

2021-07-11T00:00:00Z

Finally!

You've arrived in an organisation that embraces DevOps culture, Continuous Delivery in a microservices environment, agile in the truest sense, and is filled with clever and driven people. This is it. This is how you deliver software (in some contexts), this is what will make you proud of the work your team delivers and the role you as a test specialist have played in it.

Right?

Maybe not.

Maybe your team has a test engineer building a large, slow, unreliable test suite of end-to-end UI-level tests. Developers are adding some unit and integration tests without collaboration with the test engineer and this is all that Shifting Left means to the team. Exploratory testing is something that is unheard of or is considered to mean manual ad hoc testing that still needs to be automated. Performance, security, resilience, and other quality concerns not explicitly covered in acceptance criteria are ignored or addressed only when an incident or enough customer complaints happen. The quality of the monitoring, alerting, and logging are afterthoughts. Standups are used to justify time spent instead of progressing work. Refinements lead to no changes to the acceptance criteria or priorities. Each team member works on their own ticket. There is no mob programming or pair programming or pair testing.

It's the best of times for a test specialist, because lessons you've learned over time are now being promoted on a large scale. It's the worst of times for a test specialist, because they're being implemented in the worst way possible if at all.

It's the worst of times for a test specialist, because great testers are becoming mediocre developers. It's the best of times for a test specialist, because some developers are now very open to learning better testing.

It's the worst of times for a test specialist, because your expertise is no longer considered essential. It's the best of times for a test specialist, because you can turn all this around.

How to make standups more useful

2021-07-07T00:00:00Z

Practical examples of how to recognise a less effective standup and how to improve:

"Who wants to start?"

Crickets chirp, time drags into eternity.

Bad: wait for a volunteer to be the first to speak
Better: call out a name, or pass around a microphone or other token
Best: go from right to left on the board, whoever's assigned to the ticket should speak

The three questions

Instead of having individual contributors justify their time, focus as a team on what needs to happen.

Bad:

From person to person:

"What did you do yesterday?"
"What will you do today?"
"Are there any impediments in your way?"

Better:

Go from ticket to ticket (from closest to done to less close to done):

"This is our current progress."
"This is what we still need (to do)."

Best:

Go from ticket to ticket (from closest to done to less close to done):

"This is what we still need (to do) to get this done.
"Who will explicitly commit to (code reviewing, rubber ducking, pair testing)?"
"I'm ##% confident we'll get this done today."

And "are we still aligned with the priorities and does anything in To Do need unblocking?"

All of this helps keep people focused throughout and to think like an actual team.

Waiting to discuss obstacles

The (correct) notion that obstacles can be discussed in Standups and Retros means that some people wait for those occasions.

Bad: Waiting for a ceremony to discuss being blocked or impeded by something.
Better: Raising urgent blockers or impediments to a Scrum Master immediately.
Best: Sorting out an urgent blocker or impediment with the relevant team members or stakeholders immediately.

Note: there are times when raising an impediment with a Scrum Master is a better use of everyone's time than solving it yourself with the relevant people who can help. Use your best judgement.

Diversions

It's easy and natural for discussions to occur and for those discussions to take too long and not need everyone's attention.

Bad: Using standups for freeform discussions on technical implementations, design walkthroughs, process improvements.
Better: Nipping in the bud any meandering discussions and encouraging them to be continued afterwards in a smaller group.
Best: Nipping discussions in the bud except where a problem could be most easily and quickly solved by the whole team right then.

Being dead serious

Every day's experience in agile software development teams affirms how much it matters for a team to be happy and communicate well with each other in order to do their best work.

Bad: Discourage the wasting of time with joking around and silly distractions.
Better: Encourage positive socialising among team members during standups.
Best: Start the standup with casual banter, during the standup encourage/praise/thank others, end on an optimistic note.

Get the number of items in a dropdown via DevTools

2021-07-05T00:00:00Z

Say you have a page containing one or more dropdowns with lots of options and you want to know exactly how many options are being displayed.

Chrome

Visit the page
Open DevTools (F12)
Open the Elements tab
Right-click the dropdown -> Inspect
Find the select or ul element which contains the option or li options
Right-click that element -> Copy -> Copy selector
Open search (CTRL + F) and paste the element
Add > option (if the element is a select) or > li (if the element is a ul)

You will now see 1 out of ## where ## is the number of dropdown options.

Firefox

Visit the page
Open DevTools (F12)
Open the Inspector tab
Right-click the dropdown -> Inspect
Find the select or ul element which contains the option or li options
Right-click that element -> Copy -> CSS Selector
Open search (CTRL + F) and paste the element
Add > option (if the element is a select) or > li (if the element is a ul)
Add > option (if the element is a select) or > li (if the element is a ul)

You will now see 1 out of ## where ## is the number of dropdown options.

Fix incorrect time in WSL Ubuntu and Docker

2021-07-04T00:00:00Z

Problem

The system time in WSL is out of sync with your Windows system time. Docker containers launched from WSL suffer from the same incorrect time, which they inherit from WSL. I've observed this in WSL2 Ubuntu in Windows 10.

Solution

We can use ntpdate to update the date & time according to some source we trust. Generally running an ntpd daemon is preferred over using ntpdate, but it's quick and does the job and I'm not worried about issues with running services getting confused about living in a different time all of a sudden.

ntpdate -v

If this command gives Command 'ntpdate' not found, run sudo apt install ntpdate.

Then:

sudo ntpdate time.windows.com

Note: Other NTP servers can be found on the NTP Pool Project website.

Use date to verify the date & time are now correct.

How to use Chrome in Cypress on WSL

2021-07-01T00:00:00Z

Problem 1

Can't run because you've entered an invalid browser name.

Browser: 'chrome' was not found on your system or is not supported by Cypress.

Solution 1

I always like to have a command I can run before and after a change to verify a change had the desired effect:

google-chrome --version

This should fail. Now install the latest version of Chrome in WSL:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

Followed by:

sudo apt -y install ./google-chrome-stable_current_amd64.deb

You can delete the .deb archive afterwards.

Now again try:

google-chrome --version

Problem 2

Could not start Cypress headlessly. Your CI provider must support XVFB.

Solution 2

Ensure you've enabled running the Cypress GUI in WSL

How to handle technical debt

2021-06-30T00:00:00Z

Problem

With the best intentions not to release anything that doesn't meet the highest standards of quality, the pressure of speedy delivery and the realities of imperfect humans and circumstances can mean we build up technical debt which over time causes the code to be more difficult to maintain, the application to seem slower to end users, the company to be vulnerable to security breaches, and so on.

Solution

Expose it

Create tasks in your task & issue management system. Some teams wait to be given permission by management or product owners to handle technical debt and when they eventually get their way, they have no tasks ready to go. Create the tasks. If a team isn't coming up with enough initiatives, find out what's holding them back. Is it lack of time? Is it lack of expertise? Is it lack of analytical tools? Find out & address it.

Promote it

There is or should be a healthy push and pull in agile development teams between developers and the product owner when it comes to what goes into a sprint. The team picks the tickets, but the product owner should protest if what's being pulled in is out of touch with the product roadmap. The product owner can protest, but the team must stand up for quality. If either side is too shy, you'll end up delivering insufficiently valuable work or insufficient valuable work.

Protect it

Even during a sprint, external pressures can mean work on technical debt tasks is halted or delayed in favour of more urgent feature work. In theory, a team should be assertive enough to protect against the inevitable eroding of technical debt work. In practice it means persuasive "just this time" requests. To prevent this, get buy-in from the highest levels of the organisation, demonstrate to them the importance of working on technical debt in their language (ROI) and explicitly agree with them a % margin in each sprint that teams can dedicate to it, as otherwise the long term benefit of moving fast safely can't compete with the more immediate benefits of feature delivery until it's too late.

Spread it

Some organisations get so fed up with technical debt, they decide to dedicate 1 or 2 sprints solely to paying off technical debt. This doesn't work. First, you lose momentum on feature work and there will be pressure to make up for it. Second, the technical debt might be heavy on a particular specialty (such as backend). Third, you will just have to do it again after a few months. You'll want to keep momentum on both feature work and technical debt. Spread it out and keep up on your repayments.

The Good, the Bad, and the Pedantic

2021-06-23T00:00:00Z

If there's one thing the test community loves, it's to argue terminology. If you doubt this, throw the question "What's wrong with the term Quality Assurance?" in any public forum or social media channel with a software testing and quality focus.

I'm going to highlight three of these semantic arguments, find the good and bad in them, and with brazen hubris declare how they should be now and forever resolved.

QAs, do you mind QA'ing this release?

Years ago I read an interview with a well-respected test consultant in which he was asked the one thing he wanted non-testers to know about testing and his answer was that they shouldn't use the term QA, that the term is looked down upon by the test community.

QA stands for Quality Assurance and it's true that taken literally we cannot assure quality, if we interpret this as test specialists guaranteeing that a product release has perfect quality. It's important to know that testing cannot guarantee the absence of bugs and that even the absence of bugs does not guarantee quality.

However, that's not on any non-tester's mind. Their idea of what QA is did not come from reading the dictionary definitions of the individual words that make up Quality Assurance, it came from how the term was used in the companies where they first encountered it. QA, for non-testers, is almost synonymous with testing. Almost. They often come to understand there's an ineffable extra 'thing that QA does' (asking tricky questions, critiquing things that weren't in the requirements) that doesn't fit their idea of testing (the latter of which would traditionally be called Quality Control), so they ask someone to QA instead of to test.

I've made the mistake myself of trying to move people from saying Quality Assurance to saying Quality Assistance (and there are many other alternative interpretations of the QA abbreviation, such as Quality Advocacy or Quality Acceleration). The battle to stop people from saying QA or Quality Assurance has raged for a long time (I first came across it in the late 2000s) and it's not worth the effort.

Instead of writing blog articles (am I guilty of it myself now?) and social media posts complaining about the term, which doesn't give the greatest impression to newcomers or casual guests to the test community, explain at appropriate moments what testing and quality specialists can contribute and try not to cringe when someone says QA (which in the case of "QAs" or "QA'ing" is really hard to do!) We don't have to break up conversations about things that matter with complaints about things that matter less.

Manual testing

Manual testing is a terrible term. It sounds like grunt work, like a repetitive chore. Scripted testing and exploratory testing are both called manual, but are vastly different in how much they resemble grunt work or a repetitive chore. This is especially significant because manual testing is contrasted with test automation while scripted testing can indeed be automated and exploratory testing cannot (unless you want to be pedantic and say that once a test scenario has been uncovered through exploratory testing you may then be able to automate it).

This term just has to go.

I've seen some traction within the test community for the idea of getting rid of the manual vs automated dichotomy altogether and instead talk about "attended" and "non-attended" testing. I think this is another battle that's doomed to be lost, especially considering how thought leaders in the test community have typically engaged with the wider development community (either very little or in an adversarial way). The persuasive power needed is entirely absent.

A more realistic - and, I think, elegant - solution is to keep the manual vs automated dichotomy, but simply replace "manual" with "human". Contrasting test automation with "human testing" feels apt, easy to understand, and replaces the association of grunt work with an association of conscious presence and higher reasoning. It's difficult to imagine someone getting confused because humans are involved in test automation as well, since no-one was confused about test automation involving manual work despite being called automated.

Non-functional requirements

Non-functional requirements (NFRs) have been an industry term for many years and used as meaning the additional requirements beyond requirements describing the functionality to be implemented. However, some have realised that with a bit of ill will it's possible to interpret the term as being requirements for an application not to function. And surely we want applications to function?

I don't think anyone has ever been genuinely tripped up by the term and thought that it meant an application should be implemented in a way so that it doesn't function, but rather that this is used as a rhetorical device to advocate for more precise language, but either way it has led to alternatives like extra-functional, para-functional, or cross-functional requirements. To a person who doesn't know anything about the background, it's not very intuitive to hear someone talk about para-functional or especially about extra-functional requirements. Are they talking about extra functional requirements (note the absence of a hyphen)?

Admittedly the term non-functional requirements is a bit imprecise, but instead let's use cross-functional requirements. It's by far the most commonly used alternative already. Personally I also find it more intuitive, as it clarifies that they apply across all functionality - for example in the form of SLAs - instead of applying only to a specific feature - for example as defined in a Jira ticket's acceptance criteria.

In short

Don't get too excited about the imperfection of industry-standard terms. You want people to understand what you're talking about. If you get a chance to use better terms (such as human testing instead of manual testing), promote their use in a human-friendly way (such as adding "also called traditional term" or a very succinct explanation).

Handling incorrect iPad data in Google Analytics

2021-06-17T00:00:00Z

This is important to know about when you use mobile/tablet data to make decisions.

Problem

In Google Analytics, if you filter data by Device Category regex (mobile|tablet) or by Mobile Device Model -> iPad, it won't include data for iPads running iPadOS 13 (released September 2019) and above.

Cause

Apple decided to force desktop behaviour for iPads by changing the User Agent to resemble a desktop device.

Solution

You have to identify the data from iPadOS 13 and above by yourself. You're looking for Operating System Macintosh instead of iOS, with Screen Resolution width (note, this is the logical resolution, so may be lower than you're expecting) of 1024 or lower (which currently covers iPad models with iPadOS 13 and above and excludes macOS models).

An example of when I needed the percentage of iPad users versus all users:

Note the count of all sessions for a specific date range (Audience -> Overview)
Find the sessions for macOS (Technology -> Browser & OS -> Dimension: Operating System -> Value: Macintosh)
Dig down to screen resolution (Secondary Dimension: Screen Resolution, Show rows: 5000)
Export the data (Export -> Google Sheets)
Agree to importing the data to your Google account
In Google Sheets, select the Screen Resolution column (in my case: B)
From the Data menu, select Split text to columns
As separator, select Custom, then enter x
Now, in any empty cell, enter formula =SUMIF(B:B,"<=1024",C:C) (replacing B with the column containing the screen resolution width - the leftmost value - and C with the column containing the sessions)
Finally, divide this number by the number for all sessions and make it a percentage

What do bi-directional contract testing flows look like?

2021-04-20T00:00:00Z

It's pretty easy to find an explanation of bi-directional contract testing, but much harder if not impossible to find concrete explanations of how it impacts team workflows for realistic scenarios. Here's an attempt.

Background

Team Contractors' API endpoint /contractor contains parameters such as available and stock.

When assigning a contractor to a job, Team Odd Jobs consumes the endpoint, using available to retrieve available contractors.

When calculating total stock, Team Materials consumes the endpoint, using stock to retrieve active stock.

Now Team Contractors would like to make a change to their /contractor endpoint, changing the stock data type from a string to an integer.

Without bi-directional contract testing

Let's say there is some testing in place.

There is a test that checks the GET request on /contractor returns a 200 response. This should PASS.

There is a test that checks the method that works with stock is supplied with a string. This should FAIL.

Or maybe there is a test that checks the full API response structure against a JSON schema. This should FAIL.

Team Contractors attempt to merge their change to production.

The pipeline succeeds.

For Team Odd Jobs this is fine, but places where Team Materials retrieve the value for stock from the /contractor endpoint fail, not with a 400 Bad Request on the request itself (they are doing a simple GET on /contractor and taking the value from something like response.data.stock), but at the point the value is used in the code with some method expecting to work with a string.

Why did the tests not stop the breaking change?

Because those tests run in Team Materials' pipeline, not in that of Team Contractors.

Each team is able to independently deploy their service, with build and test stages covering only their own service and mocking or virtualising any interaction with other services. There are companies who, despite having a microservices architecture in every other respect, solve this by having a stage in the pipeline of each service that builds all services and runs integration and end-to-end tests in order to safeguard interactions between services. This is antithetical to the idea of independently deploying. There is an alternative that's (almost) as safe and allows us to move a lot faster.

With bi-directional contract testing

Now let's say we have bi-directional contract testing (BDCT) in place.

Team Contractors attempt to merge their change to production.

Their pipeline fails. It fails at the point where the provider-side contract containing the changed data type is statically compared (by a contract broker) against the latest consumer-side contracts, which in the case of Team Materials still defines an expectation for the old data type. Team Odd Jobs uses the same endpoint, but doesn't do anything with the stock parameter, so their contract doesn't even include stock and the contract comparison passes. This consumer is not affected.

Team Contractors have discovered their breaking change and can instead handle the change properly, such as by applying the expand and contract pattern or by communicating with Team Materials about being more liberal in what they accept or by some other method the team finds appropriate.

Flow details

This is what happens in BDCT when a provider (in this case Team Contractors) makes a change:

Team Contractors makes a change to their API specification.
Team Contractors makes the same change to their API endpoint.
The pipeline runs automated schema validation checks comparing the API specification against the API endpoint.
The pipeline runs any additional human-coded functional API checks against the API endpoint.
The pipeline generates a provider-side contract from the API specification, now that we've proven it matches the actual endpoint.
The pipeline returns a response from the contract broker, which statically compares the provider-side contract against the latest consumer-side contracts that use the endpoint.

This is what happens in BDCT when a consumer (in this case Team Materials) makes a change:

Team Materials changes their mocked test which defines what they need from the provider endpoint.
Team Materials makes the same change to the actual request.
The pipeline generates a consumer-side contract from the request-response pair the team's mocked test defines.
The pipeline returns a response from the contract broker, which statically compares the consumer-side contract against the latest provider-side contract for the endpoint.

In this case of a deliberate change on the consumer side, a failure would mean they should communicate with the provider about the new needs they have, so the provider can update their capabilities. Of course they should be doing this before actually making the change and seeing the pipeline fail, BDCT is just there to say "what do you think you're doing?" in case this does not happen or when a valid expectation is inadvertently changed to an invalid one.

Who are the providers and consumers?

The flows apply regardless of whether the providers and consumers are:

Microservices interacting with each other (and each could be a provider, a consumer, or both)
The backend (provider) and frontend (consumer) within a service
External services, if we can agree to upload contracts to a broker one of us owns

Error while loading shared library libnss3.so running Cypress in Ubuntu

2021-03-15T00:00:00Z

When attempting to run Cypress in Ubuntu:

Problem

/home/wilco/.cache/Cypress/4.12.1/Cypress/Cypress: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

Solution

sudo apt install libgconf-2-4 libatk1.0-0 libatk-bridge2.0-0 libgdk-pixbuf2.0-0 libgtk-3-0 libgbm-dev libnss3-dev libxss-dev

What you're installing

libgconf-2-4: GNOME configuration database system (shared libraries).
libatk1.0-0: ATK accessibility toolkit.
libatk-bridge2.0-0: AT-SPI 2 toolkit bridge (shared library).
libgdk-pixbuf2.0-0: GDK Pixbuf library.
libgtk-3-0: GTK+ graphical user interface library.
libgbm-dev: Generic buffer management API (development files).
libnss3-dev: Development files for the Network Security Service libraries.
libxss-dev: X11 Screen Saver extension library (development headers)

Setting up the Cypress GUI in WSL2 Ubuntu for Windows 10

2021-03-12T00:00:00Z

ℹ️ Update April 2025: With the inclusion of WSLg in WSL2, you should no longer need an X server. To test this, run xeyes in a WSL2 terminal and check if the simple eyes UI appears.

First, check that you're actually running WSL2. In a Windows 10 command prompt (not in WSL bash):

wsl -l -v

For this guide, you should be seeing NAME: Ubuntu, STATE: Running, VERSION: 2.

You'll also need Node. To check if you've already got it, again in a command prompt:

node --version

If you don't have a project yet with Cypress in it, let's go into WSL bash and create one. In a command prompt:

bash
cd ~
mkdir some-project
cd some-project
npm init
npm i cypress --save-dev

Then (or if you already have a project with Cypress), let's add a convenience command to package.json:

  "scripts": {
    "test": "./node_modules/.bin/cypress"
  },

This allows us to run Cypress tests by simply using npm t.

If you run it right now, it won't bring up the Cypress GUI. WSL2 doesn't have the ability to display graphic interfaces, but we can ask Windows 10 to display them for us. We do that using an X Server. There are a number of them, but I'll use VcXsrv here. Go ahead and install VcXsrv.

Now we have to make sure graphical interfaces launched from WSL2 know what local ip WSL2 is at:

cd ~/home/YOURUSERNAME/.bashrc
Add the following to the top(!) of the file:

# set DISPLAY variable to the IP automatically assigned to WSL2
export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2; exit;}'):0.0

Next, we'll have to enable D-Bus so these applications can exchange messages.

Add the following underneath the place where we set the DISPLAY environment variable:

# Automatically start D-Bus to allow communication with Cypress GUI app
sudo /etc/init.d/dbus start &> /dev/null

Now launch the X Server (if you're using VcXsrv, look for XLaunch.)

You can accept the defaults for Display settings and Client startup, but under Extra settings make sure to check the Disable access control checkbox.

In the Windows security alert that comes up first time XLaunch is launched, allow public & private networks.

Now launch your project's web server.

Finally, launch Cypress with npm t.

What is the value of UI-level end-to-end tests?

2021-03-11T00:00:00Z

By UI-level end-to-end tests I mean tests which cover a user flow (the path to complete a task) both horizontally (from start to end of the flow) and vertically (the test involves the entire stack). Specifically I'm going to talk about automated UI-level end-to-end tests, which drive a browser to run the tests.

Levels

Level	Cost	Speed to Run	Confidence Gained	Reliability
Unit (solitary, i.e. classes or functions)	Low	High	Low	High
Unit (sociable, i.e. includes some dependencies)	Low to Medium	High to Medium	Low to Medium	High to Medium
Integration (component, service, or contract)	Medium	Medium	Medium	Medium
End to end (UI, acceptance)	High	Low	High	Low

The test pyramid

There you see the rough reason why these levels of testing are traditionally represented as a pyramid. At the lowest level — of a unit test against a single class or function — the test takes a trivial amount of time to run, the test is so close to the code it's relatively quick to debug, the test is relatively quick to write, and the test is unlikely to pass or fail inconsistently. The downside is that a solitary unit test like this doesn't tell you very much.

On the other extreme you have a UI-level end-to-end test scenario. If even a single part of the flow succeeds (for instance, after submitting a valid username and password a My Account link appears) you know a number of things are working together correctly. The downsides are that the test is relatively slow to run, the test could pass or fail inconsistently for various reasons such as fluctuating page load speed, the test is costly to maintain as it depends on a relatively large part of the codebase, and a failing test is more difficult to debug as it lives several layers of abstraction away from the code.

Justification

With all those downsides, is it even justified to have automated UI-level end-to-end tests? I would say it is if you limit the scenarios to the highest priority paths through the application. For the low cost of just a few scenarios, you can increase a team's confidence in their output by showing that the most important flows in the application can be completed successfully in a realistic simulation. See it as automated acceptance tests on top of all your other tests.

Example

For an eCommerce company I covered priority flows which involved the searching for a product, navigating to a product page, adding the product to the cart, going to checkout, and filling the checkout information. To identify these priority flows, you (or a Data Analytics or User Experience specialist near you) can analyse the user behaviour in your analytics suite and come to an agreement with your stakeholders or product owner. Be careful not to overreach. For example, logging in (maybe even by API) to be able to start a flow is fine, but testing various happy and unhappy paths for logging in, account creation, forgotten password, etc. might not be best placed in your end-to-end tests. These might be better done at lower levels.

How much testing is enough in software testing?

2021-03-10T00:00:00Z

Answers to this question typically fall into two categories: narrative and numerical.

Narrative

Testing is never done, but you should stop when all relevant risks have been covered to the point you can afford (budget) and accept (agreed with stakeholders) / Testing is done when you've comprehensively explored all high and medium priority functionality / Testing is done when a skilled test specialist judges they have gathered enough information about the product for informed decisions to be made about its delivery / Testing is done when you've exhausted all the test ideas you can come up with

Numerical

Testing is done when 100% of planned test cases have been executed / Testing is done when at least 80% of the test scenarios are passing / Testing is done when 0 issues are reported from automated or manual test suites / Testing is done when test coverage is above an agreed percentage, e.g. 95%, and all those tests have been executed with 0 fails

My perspective

The exit criteria in the numerical category upset me and I will explain, with a number, why that is. The chance that your documented tests accurately represent all the tests that matter is precisely 0 (zero). There simply are too many ways in which software can disappoint.

That being said, feeling very strongly about an opinion can be a sign of missing important nuance. I will seek that nuance. If you're working in a highly regulated environment, I can imagine you have to demonstrate your formal exit criteria and how you measure them in concrete KPIs such as executed test cases and passed test cases. I would still hope your testing wouldn't stop at executing already documented tests, but perhaps prescribe a certain coverage for session-based exploratory testing, and include a formal signoff – if you really work in that type of environment – based on the informed advice of the test specialist(s).

Another nuance I can think of is marketing agencies producing high-turnover near-identical email, banner, or microsite campaigns. You could choose to have standardised test suites of which you can say that if 100% pass, it's automatically a go for that campaign. Those test suites could also include less prescriptive high level test scenarios for exploring functional and non-functional requirements, thus covering off more risks whilst still figuring into a number.

I lean more towards the narrative category. Testing is done when a skilled test specialist judges that - within a particular context of risk tolerance and regulatory requirements - additional time spent on testing is not worth the additional information that would be revealed. Narratives, however, are not the most practical thing. If I would have to give practical exit criteria for testing in the environment of Agile web development (in which I have done most of my work), I would say:

Requirements have been interrogated and testing needs anticipated (both can be done in 3 Amigos and Refinement sessions)
Local testing and meaningful code reviews have been completed
Static, unit, integration, and UI-level test scenarios are passing
Acceptance criteria in the User Story have been verified
Other relevant functional and non-functional requirements have been verified (exploratory as well as standards from your test strategy)
The impact of code changes (performance, live issues, etc.) has been monitored after deployment to production

When all these have been done, you can stop testing that piece of work. Or can you? You could argue that once a code change is in production, it's in your care forever, but anything resulting from that care would be a new piece of work.

Common mistakes when handling outages or data breaches

2021-02-06T00:00:00Z

Your company's site is down or you've become aware of an active data breach or some other incident occurs and:

it needs to be fixed straight away
it can't ever happen again
you need to deal with the fallout

Here are some pitfalls to avoid:

Wasteful lines of communication

You receive a notification from a monitoring tool about slower response times on a website you control.

You investigate.

Five minutes later, assessing the cause and scope of the slowdown, the site goes down.

Two minutes after that, your manager wants to know what's happening, how it's possible this is happening to us, and whose fault it is.

Whilst talking to your manager, the CEO joins in to explain how bad it would be if the site were to stay down.

During an active incident, people will want to know what went wrong, whose fault it is, and what processes need to change to make sure it doesn't happen again. In the meantime, the incident will not be investigated or resolved.

The obvious solution would be to politely ask those people to wait until you have better answers and to reassure them that they will receive updates every X minutes. This doesn't work very well in real life. It wastes time and not everyone can resist management pressure even when they know it should be done. What you need is a lightning rod. Designating someone as Incident Manager can free up the right people to focus on what needs to be done whilst the rest of the business know who to chase for information.

Another example of wasteful lines of communication is when multiple people and teams have identified something's happening and are independently investigating and discussing it and coming to conclusions without being aware of each other's efforts. This is another reason why it's so useful for people to know about the lightning rod that is the Incident Manager. They contact the Incident Manager, the Incident Manager decides who is best placed to take action, and they will communicate with that person and the rest of the business.

What if your business can't afford to designate someone as Incident Manager? In that case it could be agreed that there will be a messaging group in which any active threats to the business are raised and actions delegated by the group among themselves, with business stakeholders added to this group so they can view progress. However, it may be more expensive to distract all the group members in this way than to designate one person as managing the response to incidents.

Inappropriate follow-through

Say a data breach has occurred and was handled, you understand exactly what happened, and recommendations have been put together for concrete improvements of which everyone agrees that any one of them would have prevented the incident from occurring if they had been in place.

A year later the same incident reoccurs, because the improvements have not been done. How do you explain that?

To avoid having to explain it, prevent similar incidents from occurring again. Now, it's easy enough to say that you should chase recommendations for improvements until they've been completed, but there's a reason this is such a common omission. There may be strong pressures to return to business as usual, as the incident already caused business disruption and the incident is over. This pressure may be added onto by senior management, the very people with the most to lose if a similar incident did reoccur. You can use that fact - that similar incidents occurring again would damage the business more than the time and budget spent on preventive activities - to gain support.

So, use the leverage of a harmful event to make real changes. Then, once improvements have been implemented, consider how you might simulate the incident that has occurred and similar realistic versions of it, and test whether your preventive measures are adequate. Perhaps someone's login details were compromised via a spearphishing attack and used to log in to a CMS through which malware was added to a website. If you've implemented defense-in-depth protections, you should then test whether users recognise phishing attempts, whether users have permissions limited to only what they actually need, the ability to add custom HTML/CSS/JS to the CMS disabled or reduced to only what is genuinely needed, and any additional relevant protections such as ip whitelisting.

Telling people to do their job better

Another pitfall in the recommendations for improvement is to conclude that procedures were not followed and if people would just follow the procedures the incident would not have happened. Whilst that conclusion might be entirely true, telling people to just follow procedures or be less negligent is a fool's errand. There's a reason (or there might be a complex web of reasons) why they didn't following procedures and telling them to do better doesn't remove those reasons.

Is their workload so high they simply can't take the time to go by the book if they want to deliver results? Do they face a great deal of context-switching and distractions? Are the procedures viewed as out of date and out of touch? Are the procedures difficult to find or make sense of? Do the required tools suffer from issues in functionality or usability which invite skipping steps or creative workarounds? There could be any number of reasons.

You can find out by interviewing the relevant person. They can explain all the reasons something seemed like a good idea at the time. In hindsight it will likely be just as clear to them as to you that something could have been done better, but that doesn't change that everything they knew and experienced at the time made them do something different. If you don't change what they or someone else in their position knows and experiences the next time something similar happens, there's no reason they won't act the same way next time.

Implicit expectations about out-of-hours support

When immediate action is required outside of regular business hours, you don't want to have to call around and find that the needed people are not available or don't have access to the tools they need. You also don't want to get into disagreements with vendors about their conditions of out-of-hours support.

Say you work with an offshore development team and you ask if they provide 24/7 on-call support in case of incidents. Perhaps they respond you can email the project manager at any time and the project manager will then call someone from the team to assist. Don't stop there. Should we understand the project manager has sound on for their email notifications at all times? Does the rest of their team know (and are prepared for) that any of them can be called at any time of the day or night to respond to an incident? Are we agreed on what counts as an incident? How do the lines of communication proceed once everyone who's needed is on the case?

Or say you have internal development teams that have agreed on-call schedules. Have you also agreed whether their time will be compensated? Have you agreed a method of communication, such as a messaging group for which notifications can be set individually, that each team will use? Have you ensured there are fallbacks for people who turn out to be unreachable? Have you ensured incidents that have been known to occur in the past and which can be easily resolved (such as through a service restart) have been documented, so you can prevent waking up people for such issues?

Incorrect conclusions due to cognitive biases

The direction of investigation into an incident can be powerfully derailed by cognitive biases, as can the root cause analysis and forming effective recommendations for improvement. It would be tiresome to list all the possible cognitive biases, but let's view a few examples which will alert you to the danger and then let's see how we can avoid this pitfall.

Perhaps a manager says a website you're working on has become dramatically slower without there having been a deployment of any changes. The manager asks a developer about this and the developer loads the website's homepage with the browser console open and notices a JavaScript error. The developer states "there you go, there's your problem" and the manager gets another developer to go fix the JavaScript error. The website is no longer slow, then the fix is done, and everyone is happy.

As you can tell from the order of events, the JS error wasn't the cause of the reduced performance of the site, plus the performance of the site did not actually change at any time (it was the office's WiFi connection). How then did the developer and manager get it so wrong? The developer didn't typically look at the browser console, so they assumed that since there was a problem and since the console showed an error, they must be related. The developer could have seen from the error message that it had no relation to performance, they could also have looked further to see if there could be any other explanation. The developer could have also challenged the premise of slow performance by viewing the monitoring tools to see whether users of the site are experiencing any slowdown, the manager could also have asked whether the site is any slower instead of stating that it is.

In another example, perhaps a website goes down and someone discovers that there were hundreds of logins during the downtime. When the site returns after a server reboot, an additional basic authentication dialog is added to the login page to prevent what is seen as a brute force hacking attempt. Leaving aside whether this would be the best way to protect on-page and API logins from brute force attack, there was no relation between the logins and the downtime. Cursory investigation would have shown that the hundreds of logins during the downtime came down to five logins per minute, which did not noticeably affect the server or application performance, but it was the same average logins per minute as in the days and weeks before the downtime.

How to avoid such mistakes? Again there is an obvious answer that doesn't work in real life. The obvious answer is that people should be aware of cognitive biases in order to avoid them. This is much easier said than done, plus training in recognising cognitive biases does not appear to transfer between different contexts even when closely related. It's much easier, however, for someone trained in cognitive biases to recognise other people's cognitive biases. As a result, a testing or quality specialist is well suited to being involved in incident investigation and remediation.

How to write effective Gherkin scenarios

2020-04-30T00:00:00Z

Why do we use Gherkin?

Easier and less error-prone translation of business needs into functional behaviour, easier and less error-prone understanding of functional behaviour by the development team, and easier and quicker conversion of functional behaviour into (manual and automated) acceptance tests.

When is a scenario good enough?

If you're worried you're spending too much time crafting the perfect user story, stop when each scenario in your user story:

Covers a single behaviour
Contains Precondition -> Action -> Result
Avoids implementation details
Uses Specification By Example in case of many permutations
Contains succinct and accurate language

Quick help

Here's one of the most common pitfalls for teams who just started using Gherkin:

Mistake: merely reformatting plain text acceptance criteria into Given/When/Then format.

Solution: writing acceptance criteria from scratch as things a user would see and do.

Merely rewriting acceptance criteria in Gherkin format leads to scenarios which are ambiguous and hide requirements.

Example 1

Scenario 1

GIVEN the item list
WHEN wanting to add a new item
THEN I find a button below the item list to add new items

What works less well?

This scenario describes the presence of an element, not behaviour.
The scenario implies that if I do not want to add a item then the button should not be there.
We want to allow adding of items, let the team decide if it should be a button below the list.
What kind of items are these? Would every stakeholder interpret correctly what items these are?
Is the precondition correct? Do I just need to view a list or do I need to be logged in as a certain user?

How could it be improved?

If the scenario really is about validating the presence of elements, it could be a design document instead
The scenario could be rewritten to cover the intended behaviour:

Scenario 1

GIVEN I am logged in as a delivery driver
WHEN I view the list of deliveries
AND I choose to add a delivery
THEN I see a list of available parcels

There is no definite correct scenario. Multiple people could come up with different scenarios that are useful.
This would be one of several scenarios defining a flow for the imaginary process of adding a delivery.

Example 2

Scenario 1

GIVEN our application
WHEN attempting to click on the In Progress button after the previous screen
THEN it is enabled only when delivery is planned for today

What works less well

The supposed precondition tells us nothing.
The actual precondition is at the end. Once I get to the end, I have to go back to re-interpret the scenario.
Don't refer to other scenarios ("previous screen") within the scenario, set up a precondition if needed.

Example 3

Scenario 1

GIVEN the Open button on the delivery details
WHEN I change the dropdown to In Progress
THEN I get a pop-up window with 3 options (Skip, Cancel, OK)

Scenario 2

GIVEN the displayed pop-up after I have clicked In Progress
WHEN selecting Skip
THEN the pop-up is dismissed
AND I get navigated to a screen where the first button is Arrived

Scenario 3

GIVEN the displayed pop-up after I have clicked In Progress
WHEN selecting Cancel
THEN the pop-up is dismissed
AND I go back to the original screen

Scenario 4

GIVEN the displayed pop-up after I have clicked In Progress
WHEN selecting OK
THEN the pop-up is dismissed
AND the delivery is marked as In Progress

What works less well?

Difficult to identify gaps or issues in the user flow without a full view of interaction options.
Inconsistent wording for the same result ("delivery is marked as" and "screen where the first button is").
In Scenario 2, "the first button" is ambiguous, as the previous one (not the first in the UI) is meant.
In Scenario 3, "I go back to the original screen" is extraneous and may be misinterpreted as a navigation.
Similar scenarios with a few variables are more clearly expressed using Scenario Outlines.
Specific text labels such as "In Progress" are clearer when in quotation marks.

How could it be improved?

Background: I am logged in as a delivery driver

Scenario Outline 1

GIVEN I view the details of a delivery
AND the delivery status is <current status>
WHEN I change the status to <desired status>
AND I select <dialog option>
THEN the new status is <new status>

Examples:

current status	desired status	dialog option	new status
open	in progress	Skip	arrived
open	in progress	Cancel	open
open	in progress	OK	in progress
in progress	arrived	Cancel	in progress
in progress	arrived	OK	arrived

Example 3

Scenario 1

GIVEN an additional notes section
WHEN I want to edit an existing note
OR add new information
THEN I can remove or add new text of up to 250 characters or photos (no other types of attachment)
AND save my changes

What works less well?

The use of an OR (or of multiple ANDs) is a sign a scenario may be overloaded.
There are a number of implicit behaviours which could use their own scenario(s).
There are a number of implicit constraints which should be made explicit.
There are missing preconditions.

How could it be improved?

Background: I view a work log as a logged in administrator

Scenario 1

WHEN I edit text in the "Additional Notes" section
THEN I can use a rich text WYSIWYG editor

Note: It's best not to force a Given when you don't need more preconditions than the Background provides.

Scenario 2

WHEN I add text to the "Additional Notes" section
AND I save the section
THEN I am given feedback that the changes were saved

Note: We should abstract away the actual text we add. We could also use property-based testing behind the scenes.

Scenario 3

WHEN I type more text than 250 characters
THEN I cannot add any more characters

Scenario 4

WHEN I paste text of more than 250 characters
THEN only the first 250 characters are pasted

Scenario 5 Outline

WHEN I upload <photo or video> to the "Additional Notes" section
THEN I am given feedback that <something happened>

Examples

photo or video	something happened
a PNG with dimensions 1920x1080 and size 1MB	Photo was saved. Preview is shown.
a PNG with dimensions 10000x10000 and size 10MB	Message “Unable to upload file. You can upload valid photo, video, and PDF/XLSX files below 25MB.”
a PDF	PDF was saved. Preview is shown.
an XLS(X)	XLS(X) was saved. Preview is shown.
a DOC(X)	Message “Unable to upload file. You can upload valid photo, video, and PDF/XLSX files below 25MB.”
an MP4 with size 10MB	Video was saved. Preview is shown.
an MP4 with size 50MB	Message “Unable to upload file. You can upload valid photo, video, and PDF/XLSX files below 25MB.”

Note: actual permitted extensions, dimensions, and sizes may be different than this example.
Note: in reality you'd likely want to use examples that account for boundary values and permutations.

Working with WebdriverIO | Issues you might encounter

2020-04-28T00:00:00Z

Examples of issues you might encounter:

This version of ChromeDriver only supports Chrome version X

Cause: the version of the chromedriver Node package has to match the Chrome version on your machine.

Solution:

Check the Chrome version on your machine
Open your package.json
Under devDependencies, change the chromedriver version specified to your Chrome version (Example: if my PC has Chrome 83.0.4103.97, my package.json needs “chromedriver”: “^83.0.0”)
Execute npm i chromedriver
Now run your tests again

Cannot use import statement outside a module

Cause: With ES6 you can create modules and import them from elsewhere in your application, but you do have to configure Babel correctly.

Solution:

Ensure you have the Babel Node packages installed (@babel/cli, @babel/core, @babel/preset-env, @babel/register).
Ensure you’re requiring Babel in your wdio.conf.js. In your cucumberOpts (or the options for whatever framework you’re using): requireModule: [ ‘@babel/register’ ],
Ensure you have a Babel config: babel.config.js

Example contents for Node 12:

module.exports = {
  presets: [
    ['@babel/preset-env', {
      targets: {
        node: 12
      }
    }]
  ]
}

Missing ) after argument list

Cause: not every test has been given a closing parenthesis and closing bracket.

Solution: go through the test script and ensure every opening parenthesis and bracket has a partner.

Text not found despite being present in the DOM

Cause: you’re checking the page title without visiting the page first.

Solution: even if you add a baseUrl, it won’t go there unless you do browser.url('/').

Working with WebdriverIO | Switching to Cucumber

2020-04-27T00:00:00Z

Cucumber is a BDD framework which adds another layer of abstraction (bad), but makes it easier to translate tickets to tests (good) and makes acceptance tests readable by non-technical people before the functionality is even written (good if used that way).

To switch to Cucumber, first make sure your tests run successfully in your current setup, so that if something goes wrong during or after the steps below, you know the blame is likely with the steps taken rather than with issues in your pre-existing setup.

If the tests still run successfully, let’s go ahead:

Install the Cucumber Node package

Run:

npm i @wdio/cucumber-framework --save-dev

Afterwards, if you look at your package.json, you will see it lists cucumber-framework alongside the other wdio packages, matching their version.

Switch the framework configuration

To switch from the Mocha framework (which we had set up in Working with WebdriverIO | Writing the first test) to the Cucumber framework:

Open up your wdio.conf.js
Search for text fragment framework:
Change mocha to cucumber

Set the Cucumber options

Add a new section to your wdio.conf.js as follows.

//
// Options to be passed to Cucumber.
cucumberOpts: {
  require: ['./test/steps/*.js'],
  backtrace: false,
  dryRun: false,
  failFast: false, // Fail on first step, useful for debugging
  format: ['pretty'],
  snippets: true, // Show pending step suggestions
  ignoreUndefinedDefinitions: true // Treat undefined definitions as warnings
},

It doesn’t really matter where you put it, but to be organised put it after mochaOpts.

Many of these options are, indeed, optional. I recommend adding them so they are made explicit. That way, when you ask yourself why something is happening the way it is, you don’t have to remember what Cucumber’s default options are, you can see them right in wdio.conf.js. I’ve not added all of them in this example, only the few I feel could be useful to know about.

Explanations and the fill list of options can be found at WebdriverIO’s Testrunner Configuration page.

Change the specs directory

You now have the step location in two places in your wdio.conf.js:

specs: [
  './test/specs/**/*.js'
],

and

cucumberOpts: {
  require: ['./test/steps/*.js'],

You can choose whatever location you like and match the folder and file structure to it.

We will add features to a new features folder, so let’s change the specs setting to:

specs: [
'./test/features/*.feature'
],

Now when you use the test runner it would look for features. We don’t have any yet though.

Change the folder structure

You can use Cucumber with or without using the page object model (POM). Using page objects has the advantage of being able to abstract away a page’s element selectors and custom functions into an intuitive structure, making it easier to stay up to date with a changing application. The downsides are that it’s yet another layer of abstraction, making debugging slower, and that dividing an application into pages is increasingly unrepresentative of modern applications.

We’ll include POM. Let’s address our folder structure:

Go into the test folder. If you’ve followed along with the Basic setup article, it would only contain folder specs at this point.
Rename specs to steps.
Create new empty folders and name them:
1. features (this is where we will define our scenarios)
2. pages (this is where we will define the page objects)
3. support (this is where we could add helper functions generic to all pages)

Add a feature

Let’s add a scenario to drive the tests:

In your IDE, navigate to the features folder
Create a new feature file. For this guide, we can just call it tutorial.feature
Add the basic keywords and spacing of a feature file:

Feature:
  As
  I want to
  so

Background:
  Given

Scenario:
  When
  Then

Explanation

Feature: what area of functionality are we covering and why do we need this functionality?
Background: what conditions need to be in place before we can run our test?
Scenario: what steps do we need to follow to verify the functional behaviour?

Writing the feature

There isn’t one right way, but here’s a suggestion. I will use a home automation web shop as example.

Feature:
  As a new user
  I want to find smart products that fit my living situation and needs
  so I can start the journey of making my home smarter

Background:
  Given I visit the 50five homepage

Scenario:
  When I search for the "Nest protect"
  Then I see search results

You can see I’ve put “Nest protect” in quotation marks. This way we can feed that string as a variable to our tests. If you then change the string in the feature file at a later time, you don’t have to update the test.

Note also that we don’t use a generic “When I search for a product” and define a batch of products in a hidden helper function. Cucumber and WebdriverIO are not meant to exhaustively test functionality, but rather to verify very specific example flows through the application as a layer of acceptance tests on top of our cheaper more exhaustive unit and integration tests, as well as to serve as a tool for communicating and documenting clear requirements.

In your IDE, you can install extensions for Cucumber and JavaScript to get nice syntax highlighting.

Rewrite the tests

If you have existing tests using the describe/it DSL, that needs to change now.

We’ll first do it without POM. Then we’ll create a page object and refactor.

Use or create test/steps/tutorial.js and add this to the top of the file:

import { Given, When, Then } from 'cucumber';

Given(/^I visit the 50five homepage$/, () => {

})

When(/^I search for the (.*)$/, (keyword) => {

})

Then(/^I see search results$/, () => {

})

If you have existing code, comment it all out for now so it won’t be run but you can use it to put inside the Given/When/Then statements later. In JavaScript, multiline commenting uses:

/*
The thing you want to comment out.
*/

Given/When/Then syntax

/^ and $/: These indicate the start and end of a regular expression (regex). The contents need to match the feature file exactly, except you can use capturing groups to indicate variables.

(.*): This is a capturing group. In this case, I’m saying "capture anything that’s in this position of the sentence”.

() => {}: This is an arrow function. It just means "go and do the stuff in brackets". When I put keyword in the parentheses, that just means that the capturing group I used in the regex will be available as a variable.

Babel

At this point, the suite still won’t run properly. This is because I’m using JavaScript ES6 syntax. This includes a bunch of cool new features which we’ll want to take advantage of. However, the browser can’t interpret it, so we’ll need to use a compiler. That’s where babel comes in.

To start using babel:

In your package.json, add the following to the devDependencies:

"@babel/cli": "^7.6.4",
"@babel/core": "^7.6.4",
"@babel/preset-env": "^7.6.3",
"@babel/register": "^7.6.2",

Now run npm i.

Once the packages are installed, add this to your wdio.conf.js in the cucumberOpts:

requireModule: [
  '@babel/register'
],

The cucumberOpts should now read:

//
// Options to be passed to Cucumber.
cucumberOpts: {
  require: ['./test/steps/*.js'],
  requireModule: [
    '@babel/register'
  ],
  backtrace: false,
  dryRun: false,
  failFast: false, // Fail on first step, useful for debugging
  format: ['pretty'],
  snippets: true, // Show pending step suggestions
  ignoreUndefinedDefinitions: true // Treat undefined definitions as warnings
},

Now we just need a babel configuration file and we can get trucking. In the root of your test suite, create a new file babel.config.js. Check your node version via node -v and add this to babel.config.js (where 12 is replaced by whichever major Node version you’re using):

module.exports = {
  presets: [
    ['@babel/preset-env', {
      targets: {
        node: 12
      }
    }]
  ]
}

At this point, you can run the suite again!

npm t

This should return:

With the right formatter, this will have pretty colours.

Now let’s move our previous scenario into this new format. Example:

From

import { Given, When, Then } from 'cucumber';

Given(/^I visit the 50five homepage$/, () => {

})

When(/^I search for the (.*)$/, (keyword) => {

})

Then(/^I see search results$/, () => {

})

/*
describe('the 50five homepage', () => {
  it('should show search results when using site search', () => {
    browser.url('/')
    searchBar = $('input#search')
    searchBar.waitForDisplayed()
    searchBar.setValue('Nest protect')
    searchBar.keys("\uE007")
    expect($('.search.results')).toBeDisplayed()
  })
})
*/

To:

import { Given, When, Then } from 'cucumber';

Given(/^I visit the 50five homepage$/, () => {
  browser.url('/')
})

When(/^I search for the (.*)$/, (keyword) => {
  const searchBar = $('input#search')
  searchBar.waitForDisplayed()
  searchBar.setValue(keyword)
  searchBar.keys("\uE007")
})

Then(/^I see search results$/, () => {
  expect($('.search.results')).toBeDisplayed()
})

Now let’s run npm t again!

You should see:

Presto!

The page object model

Now a little bit of refactoring to start using the page object model.

In your test/pages folder, create a new file called home.js. This page object can be used to define a generic page with selectors you want to be able to use anywhere. You could do the same for functions, but we’ll leave that for the support folder helper functions and just have a generic page visiting method here.

Start with this:

'use strict'

class HomePage {

/**
* Selectors
*/

/**
* Methods
*/
}

export default new HomePage()

When we define a page object this way, it means we can call it from other page objects to use anything defined in it (in this case we’d do so using import HomePage from ‘../pages/home’).

When you expand the test suite and add specific page objects for discrete sections or pages (for example: pages\search.js, you’ll need to refactor this a tiny bit to where you instantiate a class from those specific page objects and let those page objects import from the generic page object, instead of what we do now which is instantiate a class from the generic page object directly.

Now, let’s think about our sample scenario. It has 2 selectors, one for the search bar and one for the search results. These are not specific to a page, so we can define them in the main page object. Let’s do that:

'use strict'

class HomePage {

  /**
  * Selectors
  */

  get searchBar() { return $('input#search') }
  get searchResults() { return $('.search.results') }

  /**
  * Methods
  */

  open(path) {
    browser.url(path)
  }
}

export default new HomePage()

Now we can use those selectors anywhere we import this page object. So let’s import the page object.

Go into tutorial.js and underneath the Given/When/Then import add:

import HomePage from '../pages/home';

We can now use the search selectors we defined:

import { Given, When, Then } from 'cucumber'
import HomePage from '../pages/home'

Given(/^I visit the 50five homepage$/, () => {
  HomePage.open('/')
})

When(/^I search for the (.*)$/, (keyword) => {
  HomePage.searchBar.waitForDisplayed()
  HomePage.searchBar.setValue(keyword)
  HomePage.searchBar.keys("\uE007")
})

Then(/^I see search results$/, () => {
  expect(HomePage.searchResults).toBeDisplayed()
})

Run npm t and you should end up with the same test output as before.

Now we are in a land of beauty and sunshine.

Working with WebdriverIO | Find the right selector to use

2020-04-24T00:00:00Z

You could of course do the following (not recommended):

Open the page you’re going to test in a browser
Right-click the element on the page -> Inspect
Right-click the element in the HTML -> Copy -> Copy selector

If you paste it, you get something like:

#maincontent > div.columns > div > div:nth-child(15) > div > div > div:nth-child(6) > figure > a > img. pagebuilder-mobile-hidden

Cool. Theoretically you could use this. It’s a unique reference to the element.

It’s also horribly long and unwieldy.

What you want is the shortest possible unique reference. The easiest way to get it is through using the element’s id tag. An id should always be unique. Example: $(#login). The # symbol is how you select by id. However, many elements will not have an id tag defined. In those cases you can:

Use a custom data attribute defined by the developers. Example: $('[data-test="search"]'). This would require it to be added to the source code first. You can ask a developer or submit an MR yourself.
Use the smallest selector chain that will give you the unique element.
1. Maybe your element is the first element on the page with the class product-item. You could use $('.product-item'). This would select the first element in the DOM with the class product-item. Classes are not unique, but if we know where it sits in the DOM we can still get it this way.
2. Maybe your element is the second element on the page with the class product-item. You could use $$('.product-item')[1]. Note the $$. We are selecting all elements with class product-item instead of the first one in the DOM. Then of those elements we’re selecting the second one. It’s a zero-indexed array, which means that 0 would select the first item and 1 would select the second.
3. Maybe this is all Chinese to you. In that case here’s a simple trick for you. You could take the absurdly long selector you got from the DOM (#maincontent > div.columns > div > div:nth-child(15) > div > div > div:nth-child(6) > figure > a > img.pagebuilder-mobile-hidden) and simply remove all elements except the last one and put them back in one by one until they lead to a unique element.

Working with WebdriverIO | Debugging tests

2020-04-23T00:00:00Z

You’ve written a test, you run it, you think it should work, but it doesn’t. What can you do?

Pause browser execution
Inspect variables and elements
Manually run the steps

Pause browser execution

An easy way to debug is to run the test with a pause in it, so you can view whether an element is really visible, you can inspect the element, you can see if an element is actually overlapped, etc.

You simply add browser.pause(30000) on a line above the command that’s not doing what you want it to do, and run the test. You can replace 30000 with however many milliseconds you think you’ll need.

Inspect variables and elements

You can use console.log() to show the contents of variables and other useful things whilst it’s running through the test steps. For example:

console.log(searchBar) shows what’s in variable searchBar if it has been defined. If the variable represents a selector, it will also output what the selector is defined as, and all the properties of the browser object including session id and capabilities.
console.log(typeof searchBar) shows what the datatype is of variable searchBar. If the variable represents a selector, it should output object.

This output will get lost in all the other output if you use the default log level (which is very verbose). To change that, go into your wdio.conf.js and change the logLevel setting from info (or whatever you have it set to) to error. Now on running tests you’ll see only the test output, the most important potential problems, and your console.log output.

Manually run the steps

You could open your browser to simulate the steps you’ve written, but it wouldn’t be the same as running it within a chromedriver instance via WebDriver and with the wdio.conf.js capabilities. Good news! We can do the latter manually as well.

Open your package.json. The scripts section probably looks something like:

"scripts": {
  "test": "wdio wdio.conf.js",
},

Change it to:

"scripts": {
  "test": "wdio wdio.conf.js",
  "debug": "wdio repl chrome"
},

This means that in the console we can run npm run debug and it will open a chromedriver instance which is configured and run exactly as it would run during our tests. The beautiful thing is you can now run wdio commands in your console shell. For example, try browser.url ('https://www.google.co.uk'). You can run all your test steps against the browser instance that opens.

Working with WebdriverIO | How to recognise a good test

2020-04-22T00:00:00Z

For now, this guide will be for Windows specifically. Later on it will list alternative Unix-style commands (e.g. for WSL, Ubuntu, and macOS).

Let’s take this (overly) simple example:

WebdriverIO 6 with expect-webdriver

it('should include 50five in the page title', () => {
  browser.url('/')
  expect(browser.getTitle()).to.include('50five')
})

WebdriverIO 5 with Chai

it('should include 50five in the page title', () => {
  browser.url('/')
  expect(browser).toHaveTitle('50five', { containing: true })
})

When I run it, the test passes.

Does that mean we’re done?

I consider the shortest possible criteria for a good test to be:

Is it meaningful?
Does it pass when it should pass?
Does it fail when it should fail?

The example test passes. Should it pass? When I manually check the page, the page title indeed contains ‘50five’, so it should pass.

Does it fail when it should fail? To verify this, I can edit the HTML in my local browser instance. I would have to stop test execution, edit the HTML in the browser, and then let the test run on the now incorrect page title.

First add a pause to the test, to be placed after the page to be tested is loaded:

it('should include 50five in the page title', () => {
  browser.url('/')
  browser.pause(30000)
  expect(browser.getTitle()).to.include('50five')
})

I’ve assumed here that 30 seconds will be enough to make the change, but you can give yourself more time.

Once the page is loaded, we should:

Open the browser console (F12)
Ensure we’re on the Elements (Chrome, Edge) or Inspector (Firefox) tab
Find the title tag (CTRL + F title)
Remove | 50five.nl
Select a different element, which will save the current one
Wait for the test to run

Expected: the test fails, and it gives useful feedback (e.g. “found X instead of Y”). Observed:

Sweet. This is what we want to see. You can now remove the pause from the test.

So, this means the criterion of failing when it should fail has passed.

The final criterion is: is it a meaningful test? How much confidence does it give us when we see it passes? When it fails, does that tell us about a real issue? Does the error message allow us to easily find the root cause? Taken one by one:

Confidence in the code: well, when the test passes it only really tells us that the homepage is still accessible. This is good to know, but this would be more effectively monitored through uptime monitoring tools.

A real issue is identified: not really. This is because the most likely cause of the page title not matching (aside from the site being down) is that a colleague deliberately improved it. At most it is possible that when trying to improve part of the page title, they accidentally removed the “50five” part, but in modern browsers the latter part of the page title is likely obscured in the browser tab anyway.

Makes debugging easy: it tells us what is wrong and where. The test is so simple and specific that it does not hide a different issue. This is good. Overall, the only thing this test has got going for it is that it’s easy to debug. We wouldn’t miss anything of value if we didn’t have this test. Since every test takes time to execute and every non-meaningful passing test could create false confidence in the application as a whole, this test should be removed.

What would be a more meaningful test?

Let’s check whether you can do a search using the header site search. This is commonly used by visitors of the site and takes them towards products and therefore hopefully towards adding them to their cart.

This is how we could do it:

describe('the 50five homepage', () => {
  it('should show search results when using site search', () => {
    browser.url('/')
    searchBar = $('input#search')
    searchBar.waitForDisplayed()
    searchBar.setValue('Nest protect')
    searchBar.keys("\uE007")
    expect($('.search.results')).toBeDisplayed
  })
})

Translation:

Browser object, go to the baseUrl at its root.
Find the input field for the site search, and add it to a variable searchBar so we can use it later
Make sure the input field is visible, so we can interact with it
Fill the input field with the search query Nest protect
Send the Enter key to the input field, which submits it
Check whether the search results grid is displayed, which should be the case on the search results page

With input fields you can also find the submit button and click that. In this example I chose to submit the Enter key, making the assumption that this is the more common way for a user to submit their search query and because we’re not worried about mobile here.

Run npm t. See it pass when it should pass.

Comment out searchBar.keys("\uE007") (by adding // in front of that line). See it fail when it should fail.

Is it meaningful? Well, if it passes it tells us that critical functionality is working. Great. If it fails it tells us that either the site search is not present, or it cannot be filled, or it cannot be submitted, or it does not lead to search results. This is great to know about! However, it’s not perfect, since we’ll have to find out which one it is, but on the other hand it does tell us this in the results. For example:

This shows you can submit a search, but the results are not displayed (or of course it could always be the case that someone or something renamed the element selector on the page).

This shows there’s no site search on the page. (To make it easier to simulate this, I just looked for the wrong selector. I could have also just renamed the element in the HTML whilst the test was running.)

In conclusion, we now have a better test, using the criteria provided and by testing our tests!

Working with WebdriverIO | Writing the first test

2020-04-21T00:00:00Z

For now, this guide will be for Windows specifically. Later on it will list alternative Unix-style commands (e.g. for WSL, Ubuntu, and macOS).

If within your Node project directory you do npm list @wdio/cli, what’s the first number after -- @wdio/cli@ in the output? If it’s 6, see the instructions for WebdriverIO 6 below. If it’s 5, see the instructions for WebdriverIO 5 below.

WebdriverIO 6

WebdriverIO 6 bundles its own native assertion library, expect-webdriverio.

Configure expect-webdriverio

The only thing you have to do to activate it is the following.

Open up your wdio.conf.js in your IDE
Uncomment the beforeSession section
Add require('expect-webdriverio')

So it becomes:

beforeSession: function (config, capabilities, specs) {
  require('expect-webdriverio')
},

That’s it. You don’t need to call it within test files, it will get called at the start of each session.

Write a test

Let’s start as simply as we possibly can. For my examples I will use https://www.50five.nl as my test page.

describe('the 50five homepage', () => {
  it('should have a pending test', () => {

  })

  it('should include 50five in the page title', () => {
    browser.url('/')
    expect(browser).toHaveTitle('50five', { containing: true })
  })
})

Translation:

Browser object, go to the baseUrl at its root.
Tell me whether the value in the <title> tag of the current browser tab contains the substring 50five.

Run npm t. The result should be something like:

WebdriverIO 5

By default, WebdriverIO 5 uses Mocha's assertion library. Mocha's default assertion module only supports the functions listed on the Node.js Assertion Testing page, which is very limited. We will definitely want to change this, so we can assert in more ways and more elegantly.

A nice way to do this is with Chai’s ‘expect’ assertions.

Install Chai

We’ll need to install the Mocha adapter for WebdriverIO, and then the Chai package. Make sure you’re in your project folder, then run:

npm i --save-dev @wdio/mocha-framework

And:

npm i --save-dev chai

Now let’s get it configured.

Configure Chai

In your IDE, open up wdio.conf.js.

Find this commented out section:

// before: function (capabilities, specs) {
// },

Uncomment it, so it becomes:

before: function (capabilities, specs) {
},

As it explains right above that section, this gets executed before test execution begins. There are also other commented out sections where you could define what should run before each test, after each test, or what should run before a session is initialised, etc. We won’t bother with those now.

Now, let’s call Chai in this before section:

before: function (capabilities, specs) {
  var chai = require('chai');
  global.expect = chai.expect;
  chai.Should();
},

Save the file.

Now open up your test file. If you’re following on from Working with WebdriverIO | Basic setup, this will be tutorial.js and it will contain:

const assert = require('assert');

describe('the 50five homepage', () => {
  it('should have a pending test', () => {

  })
})

Change this to:

var expect = require('chai').expect

describe('the 50five homepage', () => {
  it('should have a pending test', () => {

  })
})

This is still an empty test, but we want to know Chai is working right before trying anything else:

npm t

The output should still be 1 passing test.

Write a test

Okay, now we can get to business.

Let’s start as simply as we possibly can:

var expect = require('chai').expect

describe('the 50five homepage', () => {
  it('should have a pending test', () => {
})

it('should include 50five in the page title', () => {
  browser.url('/')
  expect(browser.getTitle()).to.include('50five')
  })
})

Translation:

Browser object, go to the baseUrl at its root.
Browser object, get the value in the <title> tag
Tell me whether the value in the <title> tag contains the substring 50five.

Run npm t. The result should be something like:

Next steps

That’s the first real test done!

Is it a useful test? Not really. We’ll work on that.

In other articles we’ll look at:

How do you test interesting things like login, search, interacting with overlays?
How do you switch from Chai to using Cucumber, if so desired?
How do you debug problematic tests?
How do you determine what to test and how to test, what are common pitfalls and solutions?
How do you tell a bad test from a good test?
How do you create a repo for the tests and work with that repo?

Working with WebdriverIO | Basic setup

2020-04-20T00:00:00Z

For now, this guide will be for Windows specifically. Later on it will list alternative Unix-style commands (e.g. for WSL, Ubuntu, and macOS).

Prerequisites

To get started with WebdriverIO, we first need an IDE (development environment), Node, and Java.

Choose and install an IDE (recommended: vscode / Visual Studio Code)
Install Node.js (check if you have it via node -v in a command prompt)
Install the Java Development Kit (JDK) (check if you have it via java -version in a command prompt)

Create a test project

Let’s create a folder to work in and initialise a new Node project:

Decide where to place your project (example: C:\Node) and go there in a command prompt: cd\node
Decide a project name (example: wdio-sandbox) and create it: mkdir wdio-sandbox && cd wdio-sandbox
Initialise the project: npm init
1. Package name: just press Enter
2. Version: just press Enter
3. Description: Sandbox for playing around with WebdriverIO
4. Enter point: just press Enter
5. Test command: wdio wdio.conf.js
6. Git repository: just press Enter
7. Keywords: just press Enter
8. Author: enter your first and last name
9. License: just press Enter
10. “Is this OK?”: just press Enter

If you make a mistake, you can start over entirely by using rm -r wdio-sandbox to delete the directory and all its contents. This is a command to be careful with.

Now open the folder you created in your IDE.

You should see it only includes a package.json file at this point.

Install WebdriverIO

Now we install WebdriverIO itself, specifically the command line tools (cli) including the test runner.

Switch back to your command prompt and execute:

npm i --save-dev @wdio/cli

Translation: Node package manager, install for me the @wdio/cli package and add it to the devDependencies section of package.json

Let's check if it was successful:

.\node_modules\.bin\wdio --version

If you get '.' is not recognized as an internal or external command, operable program or batch file., ensure you’re using forward slashes.

It should now return the version number of the wdio cli tools. This should also be shown in the package.json under devDependencies.

Configure WebdriverIO

Now we generate the wdio configuration file which we can then manually change.

.\node_modules.bin\wdio config -y

Open it up. Most interesting at this point are:

specs - Defines the location of your tests. You can put tests in this path once we start creating them, or change the path if you'd like to structure things differently.
capabilities - Here we define the browser(s) we'll use to run the tests (Chrome, Firefox, and/or Internet Explorer).
baseUrl - The domain to run the tests on, so you don't have to include it for every url you visit.
framework / mochaOpts - Here we could set the assertion framework and BDD framework options, which will become relevant very quickly.

For now, just change the baseUrl value from http://localhost to any public url of your choosing. You can change this later when you're writing tests for a different domain.

Create a test file

In your IDE, create a folder structure matching what you have in wdio.conf.js under specs. So if you kept the default, create a folder test and within it a folder specs. Within that, we can add test files. Create a file called tutorial.js within the specs folder.

Run a test file

We can already try running the test suite now. What would we expect? We'd expect the test suite to run successfully, with 1 passed test (because it will count our empty test file without any tests as not having any failed assertions and therefore passing).

In a command prompt, run:

npm t

This is short for npm test, which will run whatever you have defined in package.json under scripts -> test. Earlier we defined this as part of the npm init process. If we hadn't done this, we would have to run tests by typing .\node_modules.bin\wdio wdio.conf.js every time. Later on we'll add another useful command to scripts to help solidify this notion of custom npm commands. As output you see something like the following.

Now, let’s add an empty test to the test file. Open tutorial.js in your IDE and add:

const assert = require('assert');

describe('our test page', () => {
  it('should have a pending test', () => {

  })
})

Save it.

Switch back to a command prompt and run:

npm t

The output should now result in something like:

You can see the test runner still counts the test file as not having any failed assertions and therefore passing. That's one to look out for once you've got a test suite going, as you might think a test is passing whereas in reality it doesn't have any assertions.

We now have a development environment in place and an empty test.

Finding a Cloudflare event by Ray ID

2020-03-11T00:00:00Z

When one of your legitimate users gets blocked by Cloudflare, they will be quoted a Ray ID which you can use to look up the exact rule that blocked them. You can then choose to change the Block to a Challenge (CAPTCHA) or to Disable it altogether.

Search a Cloudflare event by Ray ID

Log in to Cloudflare
Navigate to the website you'd like to manage access for
Go to Firewall
Under Firewall Events, hit Add filter
From the New filter dropdowns, select Ray ID, equals, and the user's Ray ID
Hit Apply

You should now be seeing the Activity log filtered down to 1 event.

Investigate the Cloudflare event

Select the event
Copy the value for Rule ID, e.g. 100173
Go to Managed Rules
Under Cloudflare Managed Ruleset, hit Advanced
In the search bar, submit a search query for the Rule ID you copied

You should now be seeing the rules filtered down to 1 rule. You can view its description — to help you understand why the block occurred — and change the Mode to decide what should happen when a user (whether they be well-meaning or hostile) encounters it.

Note: if you can't find the rule in the ruleset Cloudflare Managed Ruleset, try the other rulesets in the Managed Rules section.

Manage merge rights in GitLab

2020-01-21T00:00:00Z

To prevent all users from being able to push a deployment to a production or acceptance environment, GitLab uses the concept of protected branches.

Protected branches

This example means:

Everyone with at least Developer permissions can create and/or approve a merge request to the release branch.
Only those with at least Maintainer permissions can create and/or approve a merge request to the master branch.

For more information on roles, see Manage access to GitLab projects.

Set permissions for an existing protected branch

In GitLab, go to the repo you want to set branch permissions for
Go to Settings -> Repository
Expand the Protected Branches section
Now you can:
1. Select a different role tier for who is allowed to merge to a certain branch
2. Select a different role tier for who is allowed to push to a certain branch
3. Select whether the branch should be protected at all
4. Select whether a code owner should approve any MR (note that if you were to use this, the code owners have to be configured separately)

Changing an existing branch to a protected branch

In GitLab, go to the repo you want to create a protected branch for
Go to Settings -> Repository
Expand the Protected Branches section
From the Branch field, select a branch
From the Allowed to merge field, select a Role or Group or User (note: a Group or User could have lower permissions than you’d like) who should be able to merge a merge request into a branch. You’d want this to be as restrictive or more restrictive compared to the Allowed to push field
From the Allowed to push field, select a Role or Group or User (note: a Group or User could have lower permissions than you’d like) who should be able to try to push a merge request to a branch
Disable the Require approval from code owners checkbox (unless you know what you’re doing)
Hit Protect

Manage access to GitLab projects

2020-01-20T00:00:00Z

In GitLab, the following is how you set permissions for a repo.

Groups

You can divide your team or organisation into groups and subgroups. For example, you could have a group called weyland-yutani, with subgroups weyland-yutani/marketing and weyland-yutani/external.

This works as follows:

If you invite a subgroup to a project, access to that project is extended to members in the subgroup and the parent group.
If you add a member to a subgroup, they gain access to projects assigned to the subgroup, but not to projects assigned only the parent group.
Why not create more groups instead of subgroups? Firstly because groups are GitLab-wide. You cannot use group names which another organisation uses. Secondly, you wouldn’t be able to inherit members from the parent group, you’d have to manage those members separately.

If you want to give access to one or more people:

Giving access to a repo

For each individual:

Go to Groups
Select the group or subgroup for editing (cog icon)
Go to Members
Ensure you’re on the Invite member tab and:
1. Enter their GitLab username, GitHub username, or email address
2. Choose the appropriate permission role (roles explained below)
3. Hit Invite

If you only added people to the main group, they’ve already been added to each project in the space, in which case the following steps are not needed.

For each project:

Go to Projects -> Your projects
Go to the project (hit the linked project name)
Go to Settings -> Members
Go to the Invite group tab and:
1. Select the group or subgroup
2. Select the maximum permission role they should have (roles explained below)
3. Hit Invite

Roles

Summary:

Role	Can	Cannot
Guest	View basic project details	Leave comments or create issues on private projects
Reporter	Create issues, manage labels, see merge requests	Create branches or push to branches.
Developer	Create branches, create merge requests, cancel and retry jobs	Push to a protected branch
Maintainer	Can push to a protected branch, can add members, can manage CI/CD	Transfer/remove projects
Owner	Transfer/remove projects	Fly, become invisible, stop time

More detail can be found in the Permissions section of the GitLab Docs.

Commonly used Magento 2 commands

2019-11-01T00:00:00Z

Display the cron jobs

crontab -l

Install the cron jobs

php bin/magento cron:install

Reindex

php bin/magento indexer:reindex

Disable the cache

php bin/magento cache:disable

Enable a module

php bin/magento module:enable Vendor_ModuleName

php bin/magento setup:upgrade

Check status of modules

php bin/magento module:status

Flush the cache

php bin/magento cache:flush

Clean the cache

php bin/magento cache:clean

Compile classes

php bin/magento setup:di:compile

Set Magento to developer mode

php bin/magento deploy:mode:set developer

Check the Magento mode

php bin/magento deploy:mode:show

Publish assets for an area

Example for Back-end (adminhtml):

php bin/magento setup:static-content:deploy en_US -a adminhtml -f

Example for Front-end (frontend):

php bin/magento setup:static-content:deploy en_GB -a frontend -t YourTheme/YourTheme -f

Example for both areas (adminhtml & frontend):

php bin/magento setup:static-content:deploy -f en_GB en_US

Generate factory classes

sudo chmod -R 777 generated/

php bin/magento setup:di:compile

Unlock an admin user

php bin/magento admin:user:unlock AdminUsernamE

Show query from collection

echo $collection->getSelect()->__toString();

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found

2019-08-06T00:00:00Z

"SQLSTATE[40001]: Serialization failure: 1213 Deadlock found"

This error can occur during reindexing.

Example

Category Products index is locked by another reindex process. 
Skipping.
Product Categories indexer process unknown error:
SQLSTATE[40001]: Serialization failure: 1213 Deadlock found
when trying to get lock; try restarting transaction, 
query was: DELETE FROM catalog_category_product
WHERE (category_id = ('1050'))

Cause

Either another reindex is running, or someone cancelled a previous reindex (in which case indexing for that table will remain locked).

Solution

php bin/magento indexer:reset

php bin/magento indexer:reindex

How Multi Source Inventory (MSI) works in Magento 2.3+

2019-03-01T00:00:00Z

Example view of Quantity per Source and Salable Quantity for a product:

Name	Quantity Per Source	Salable Quantity
Product X	Warehouse X: 100	UK: 185
	Dropship A: 50	DE: 180
	Dropship B: 20	AT: 180
	Dropship C: 10	CH: 180
	Dropship D: 0
	Brick store: 5

Explanation (Quantity per Source)

This particular store setup uses separate Websites and Store Views per locale
The Salable Quantity for each locale combines stock from Sources which have been set for that locale
In this example, UK ships from every Source, so 100 + 50 + 20 + 10 + 5 = 185
The other stores ship from everything except the physical store, so 100 + 50 + 20 + 10 = 180

After a customer orders 1 Product X from the AT store, but before it has been shipped:

Name	Quantity Per Source	Salable Quantity
Product X	Warehouse X: 100	UK: 184
	Dropship A: 50	DE: 179
	Dropship B: 20	AT: 179
	Dropship C: 10	CH: 179
	Dropship D: 0
	Brick store: 5

This reservation reduces Salable Quantity by 1 for all stores connected to the same Source
If Out of Stock Threshold has a positive value, it is subtracted from Salable Quantity

After the order has been shipped:

Name	Quantity Per Source	Salable Quantity
Product X	Warehouse X: 99	UK: 184
	Dropship A: 50	DE: 179
	Dropship B: 20	AT: 179
	Dropship C: 10	CH: 179
	Dropship D: 0
	Brick store: 5

Now that the order has been shipped, the reservation is removed and the stock is reduced
The stock is reduced at the Source which is the highest priority in the configuration and has stock
Reservations which do not lead to shipments (order cancelled, for example) are removed once daily

Out of Stock Threshold setting

Products -> Catalog (under Inventory) -> Product X -> Advanced Inventory (under Quantity):

Formula

Salable Quantity = Source A stock - Reservations + Source B stock - Reservations (and so on in case of more Sources) - Out of Stock Threshold setting.

eCommerce test prompts

2018-08-23T00:00:00Z

Testing a web shop and think you're done? Here are some test ideas you may or may not have covered yet (that will still not be exhaustive):

Regions

Tested across supported countries? What differences should there be? Currency? Date & time formats? Payment methods? Shipping methods? Regulations?

Languages

Tested across supported languages? Special character rendering, capitalisation & punctuation, impact on cross-browser font rendering.

Form factors

Tested with touch navigation? Tested across supported resolutions?

Payment types

Pay now versus pay later. Credit card versus bank transfer.

User states

Check out as logged in user versus guest user. With and without address set on account. With old and new account (keep old accounts around).

Product types

Simple products. Configurable products. Bundles.

User actions

Printing a page. Letting a page time out. Navigating in landscape mode.

Product states

Out of stock product. Product with limited stock. Reserved product.

Cookie acceptance

Test impact of rejected or accepted state on cookies loaded. Choice remembered when it should be?

Cart

Other products in cart from old session. Other products in cart without stock. Other products in cart from different warehouse(s).

Microsites

Changes on main site also made on other sites when consistency is expected?

Tracking

Impact of changes on whether user data is still tracked as it should be.

Compliance

What safe shopping or other regulations need to still be safeguarded?

Performance

Impact of changes on page load time, Lighthouse score, transaction times of affected functionality.

Security

Do changes have implications for data privacy or create security vulnerabilities?

Accessibility

Do changes violate your supported level of the Web Content Accessibility Guidelines?

SEO

Verify that the structured data has the expected properties and values.

Mocking API requests and responses through Charles Proxy

2018-06-13T00:00:00Z

As a prerequisite to the below, ensure the SSL proxying settings and certificates are set and Charles is logging requests from the browser you're targeting.

Map Local

You can define modified response bodies to be loaded instead of the actual responses.

First, ensure the following rewrite rule is in place for CORS:

Under Tools, select Rewrite...
Ensure the Enable Rewrite checkbox is checked
Add a new rule
Add the relevant URIs (you can use an asterisk as a wildcard)
Add the actual rule
Type: Modify Header
Where: Response
Name: X-Charles-Map-Local
Replace:
Name: Access-Control-Allow-Origin
Value: *
Ensure the rule is checked
Apply it
Reload the page with the request you're targeting

Then create the modified response to map to:

In a browser, open up the developer console's Network tab
In the browser, trigger the response you want to modify
Copy the response
Paste the response into an editor which parses it nicely
Modify the response to whatever you would like to see (whilst remaining valid)

Then map to it:

Under Tools, select Map Local...
Ensure the Enable Map Local checkbox is checked
Add a new mapping
Add the relevant URIs (you can copy the URI from the browser request and paste it in the Host field, Charles will break the URI into the appropriate fields once you move focus away from the Host field)
Choose the local response file to map to
Save the mapping
Ensure the mapping is checked (and that potentially conflicting mappings are not checked)
Reload the page with the request you're targeting

Rewrite

You can enable or disable rewrites of request/response bodies, parameters, or status codes to be in place whenever you use the targeted URI(s).

Under Tools, select Rewrite...
Ensure the Enable Rewrite checkbox is checked
Add a new rule
Add the relevant URIs (you can copy the URI from the browser request and paste it in the Host field, Charles will break the URI into the appropriate fields once you move focus away from the Host field)
Add the actual rule
Ensure the rule is checked (and that potentially conflicting rules are not checked)
Apply it
Reload the page with the request you're targeting

Example rules:

Type: Response Status
Where: Response (automatic)
Match - Value: 200
Replace - Value: 500

Type: Body
Where: Response
Match - Value: "Expires":"2018-07-31T14:44:44Z"
Replace - Value: "Expires":"2018-07-31T14:49:44Z"

Type: Body
Where: Request & Response
Match - Value: Details
Replace - Value:

Type: URL
Where: Request (automatic)
Match - Value: register
Replace - Value: asdf

Type: Modify Query Param
Where: Request (automatic)
Match - Name: LanguageCode
Match - Value: en
Replace - Name: LanguageCode
Match - Value: jp

Breakpoints

You can target requests 'live' where Charles will interrupt further requests and allow you to edit its parameters and continue with the changed parameters.

Under Proxy (or from the toolbar), select Enable Breakpoints
Under Proxy, select Breakpoint Settings...
Add a new breakpoint
Select whether you want to target a GET or POST request in the Scheme field
Add the relevant URIs (you can copy the URI from the browser request and paste it in the Host field, Charles will break the URI into the appropriate fields once you move focus away from the Host field)
Select whether you want to target the request and/or the response
Ensure the breakpoint is checked (and that potentially conflicting breakpoints are not checked)
Trigger the request in the browser
Edit the parameters as desired
Execute the amended request/response
Return to the browser

The over- and underestimation of test automation

2018-06-10T00:00:00Z

Let's be clever about test automation in web development at scale. If you think manual testing no longer has a place, this post is for you. Conversely if you think programmed checks have low value, this post is also for you.

Underestimation

"Test automation is just a series of dumb checks telling us whether what was true is still true"

Whilst test automation is not limited to automated regression checks, those are already hugely valuable. Imagine putting changes into production every day and having to wait for a suite of manual regression checks to be completed each time to learn whether the changes break anything.

There's an understandable predicament for testers who have no interest in an SDET (Software Development Engineer in Test) role and find the job market increasingly demands a tester to be just that. However, the response should not be to attempt to deny the value of test automation. The response should be to celebrate that test automation (whether it be done by you or by developers) will provide a test specialist with more time for the interesting aspects of testing, due to being able to outsource the repetitive yet necessary regression checks to one's friendly neighbourhood robot.

Overestimation

"When can we expect to reach 0% manual testing?"

Looking on the bright side, it's encouraging that test automation has become so valuable in web development that some people think it can deliver us from all our woes. Less encouraging is that after at least two decades of teaching and improving the effectiveness and efficiency of exploratory test techniques and teaching that quality is more than verifying a few known functional claims are true, the global test community has failed to reach outside of their own silo and influence a new generation of software engineers and engineering managers.

Fewer and fewer view testing as more than mere verification of acceptance criteria and/or scripted regression checks and/or arbitrary bug hunting. By those standards it's true — we could handle testing through test automation and, for the rudimentary bug hunting aspect, through developers clicking around their own work and reviewing screenshots from cross-browser previewing tools. Yet this leaves a number of risks unexamined and a number of other risks inadequately covered.

Where we are now

"For testing we always add 1 story point."

Testing has evolved. It has become increasingly fit to handle the typical modern day web development project with its continuous delivery and swiftly changing requirements, thanks in large part to the increased emphasis on test automation at all levels (unit, integration, end-to-end) and all stages (TDD/BDD early on, regression tests in CI, logging and monitoring in production) and the application of Agile (or agile) and DevOps practices of shifting left and using a whole team approach to testing.

However, with developers taking a more active and integrated part in testing — which is very welcome — and testers changing focus from exploratory test techniques, custom test tooling, and non-functional testing to a primary focus on developing browser-driven end-to-end regression suites, and with no test manager or test coach taking responsibility for tirelessly promoting quality and appropriate ways of achieving it, a number of gaps have opened up and are widening.

There are now development teams which haven't had experience with professional testers (aside from SDETs) or test managers and think there are three types of testing: 1. unit tests written by a developer, 2. UI tests written by an SDET, and 3. verification where a developer or tester manually checks whether a user story's acceptance criteria are met. Some teams are also aware of integration testing.

The gaps are filled only when someone happens to have that capability, depending on them having interest in a particular quality aspect and the time to look into it. Validation is done to some extent by product owners, with a business view rather than an integrated user/business/technical view — and this is not usually done until the acceptance stage rather than when requirements are worked out. Negative testing is done to some extent by ad hoc checks on one's own work and dogfooding, and canary releases are sometimes used to compensate for the lack of depth. Performance, security, accessibility, l10n/i18n, and other types of cross-functional testing are done by individual developers (depending on familiarity, motivation, and personality), or are outsourced, in both cases without anyone being able to explain why the chosen test approach is adequate to the needs of the project. Exploratory testing is not done at all. Test tools, harnesses, and frameworks are chosen and built or not built without considering cost/benefit, applicability, or limitations.

The theme in all this is that we're so preoccupied with delivering a feature and making sure it works for a minimum set of use cases, that important risks are not addressed and are waved away with the idea that addressing risks is not providing value to customers (seeing as it does not provide a new feature to the user) or that these risks do not matter because you can respond quickly when things do fail in production. This is absurd when you could mitigate those risks early on at a much lower cost, preventing or identifying issues which are much harder to resolve at later stages, and when you'd have to have monitoring in place that's more sophisticated and broad than quality controls you use before release.

Where we can go next

"So then we just need to have developers think about quality more, right?"

You do, if you leave out "just". Developers should consider risks, quality, and testing, and testers should promote that and assist with it and train developers in it, but don't expect every skilled developer to also become so skilled in testing as to equal the years of deliberate practice and experience of a dedicated test specialist. What's more, it would require switching mindsets effectively from "how do I make this work?" to "what are all the circumstances under which this wouldn't work or wouldn't meet the needs of our target audiences or the business?" throughout development and in my experience this is an unrealistic imposition on developers in terms of human cognition and available time and personal interest.

So you can have a team where everyone is a software engineer (aside from the PO/BA, UX, and the Scrum Master), but you may still need at least one person among them who is a highly skilled test specialist. You also need at least one person - ideally the same person(s) - with both the will and the mandate to stand up for quality. You can pontificate about quality being everyone's job, and it certainly is, but in practice it still takes a torchbearer to light the way.

You could take it a step further and have one person on the team focused on test automation, test tooling, test environments, and supporting developers in their testing, and another person focused on domain knowledge, big picture thinking, exploratory testing, cross-functional testing, and still a degree of technical testing. I've heard it said that software engineers on a Scrum team should be interchangeable. I think that's a harmful idea. Great teams are not homogeneous, they are a group of specialists working together. One of those specialists should know all about the return on investment of every level, phase, and type of testing.

Conclusion

We should exhaust the possibilities of test automation. We should apply it wherever it can save us time. We should apply it wherever it can provide swift, reliable, and meaningful feedback. We should apply it to functional testing as well as cross-functional testing. We should explore new avenues like 'AI-driven' test generation. We should also use exploratory testing wherever it can save us time. We should apply exploratory testing wherever it can provide swift, reliable, and meaningful feedback. We should use exploratory testing to uncover scenarios no-one had thought of. We should learn and ask questions and challenge assumptions at all times.

EPERM: operation not permitted, unlink chromedriver

2018-05-12T00:00:00Z

On running yarn with a selenium-standalone dependency, you may encounter this error:

yarn install v1.3.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
error An unexpected error occurred:
"EPERM: operation not permitted, unlink
'/node_modules/selenium-standalone/.selenium/chromedriver/2.37-x64-chromedriver'".

If this occurs after you've run WebDriver tests, there may be a very simple solution. If the tests used chromedriver and didn't properly close the instances, then there may still be an active chromedriver process prohibiting it from being unlinked. Check if this is the case and close the process(es).

If there are no open chromedriver processes and the error still occurs, check npm -v. If the version is 5.4 or 5.5, either downgrade to 5.3 (npm i -g npm@5.3) or upgrade to 5.6+ (npm i -g npm).

ERROR: Cannot define class using reflection

2018-05-11T00:00:00Z

Running tests driven by WebDriver, I experienced the following issue.

ERROR: Cannot define class using reflection
 at new RuntimeError
 (.\node_modules\webdriverio\build\lib\utils\ErrorHandler.js:144:12)

This occurred after, and proved to be a direct result of, a Java update.

To solve this, sadly you'll have to downgrade from JDK 10 to JDK 9 on your local machine (until Selenium WebDriver is updated to be compatible with Java 10 and until you're ready to update WebDriver on your project).

You can find legacy JDK versions in the Oracle Java Archive.

ESLint gives "Uncaught SyntaxError: Unexpected token"

2018-03-31T00:00:00Z

If ESLint reports Uncaught SyntaxError: Unexpected token for an empty line or for a character which does not violate any linting rule, there may be a conflict between multiple ESLint instances.

For example, I received the Unexpected token error for an RxJS Observable which had nothing wrong with it. The reason turned out to be that I had the eslint and babel-eslint libraries running on it, whereas my IDE (VSCode) already used the eslint extension as well.

I opted to disable the native extension in VSCode. The error vanished and linting ran as expected.

Switching between GitHub accounts on the same machine

2018-02-01T00:00:00Z

Update 18 June 2021: Replaced RSA with ED25519

If you want to be able to work on different repositories with different GitHub accounts, for example using a work account for one repository and a home account for another repository, then this is what you need to do. In summary:

Generate an SSH key on the local machine for each account you want to use
Edit the /.ssh config file to include Host information for each account
For individual repos, change the remote url to specify the Host to use

Generate an SSH key on the local machine for each account you want to use

When you do the following, fight the tendency to accept default options. We'll need to make sure not to accept the default name for a new SSH key and instead save it under a custom name.

For any GitHub account you don't have a local SSH key for yet, run ssh-keygen -t ed25519 -C "your.email@address.here" for the email address associated with that GitHub account.

When it generates the key, save it under a new name, so instead of the default id_ed25519 make it, for example, id_ed25519_HOME or id_ed25519_WORK.

Your current keys are located in the .ssh folder. You can view existing pub keys by running ls ~/.ssh/*.pub

Edit the /.ssh config file to include Host information for each account

Edit the SSH config in ~/.ssh/config and add any additional GitHub accounts on top of the default account. You need a unique Host for each, for example github.com-username, and add the SSH key you created for it.

Example config:

#HOME
Host github.com-homeuser
HostName github.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_ed25519
AddKeysToAgent yes

#WORK
Host github.com-workuser
HostName github.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_ed25519_WORK
AddKeysToAgent yes

For individual repos, change the remote url to specify the Host to use

If you have a default GitHub account you use, but have a specific repository you want to use a different GitHub account for, check the current Host by running git remote -v.

The remote will be something like:

origin git@github.com:username/repo.git (fetch)
origin git@github.com:username/repo.git (push)

All you need to do is change the host, so you want to end up with:

origin git@github.com-workuser:username/repo.git (fetch)
origin git@github.com-workuser:username/repo.git (push)

To do so, simply run (replacing the values with those from your git config) git remote set-url origin git@github.com-workuser:username/repo.git

Now this repository will use your work user account and its SSH key.

Since we updated the Host value for both the existing and the new key, we'll need to run the set-url command for every local repo on the same machine, adjusting the Host value depending on whether the repo is used professionally or personally.

Fixing the "invalid signing key or OpenId metadata document" error" in Bot Builder

2017-08-21T00:00:00Z

When running a local emulation of a Microsoft Bot Builder chatbot, you might find that without making any changes to your code or dependencies you receive a new error.

Example:

> node app.js
restify listening to http://[::]:3978
ERROR: ChatConnector: receive - invalid signing key
or OpenId metadata document.

Check the version of the botbuilder framework you're using, and the version of the emulator you're using. In my case, the Bot Framework Channel Emulator seemed to have updated itself to v3.5.31 (latest version at the time), with the botbuilder library at v3.5.4 (latest version being 3.9.1 at the time).

As of v3.5.31, the emulator uses a new authorisation endpoint. Microsoft's Bot Framework blog has more information on this change. To solve the problem, simply update the botbuilder library (Node.js: npm i botbuilder --save).

After the update, we're back in business:

> node app.js
restify listening to http://[::]:3978
ChatConnector: message received.

Configuring ConEmu as a multi-terminal workspace

2017-07-01T00:00:00Z

Installation

You can download ConEmu from https://www.fosshub.com/ConEmu.html.

You'll likely want the Installer download for the latest Stable release.

On running the installation, select either the x64 or x86 version, of course depending on whether you're running a 64-bit or 32-bit version of Windows respectively.

Splitting panes on launch

You can create a shortcut for ConEmu with the target path edited so it launches with multiple terminals.

Example:

"C:\Program Files\ConEmu\ConEmu64.exe" -cmdlist
cmd -cur_console |||
cmd -cur_console:s1TVn |||
cmd -cur_console:s1THn |||
cmd -cur_console:s2THn

This creates a grid of 4 terminals with equal width and height.

Alternatively, you can edit the predefined task for the default shell. In ConEmu, go to Settings and under Startup select Tasks. Edit the {Shells::cmd} task.

Example:

cmd.exe /k "%ConEmuBaseDir%\CmdInit.cmd"
cmd -new_console:s66H
cmd -new_console:s50H
cmd -new_console:s2T50V
cmd -new_console:s3T50V

Check the Default shell checkbox and save the changes.

This creates a 2x2 grid on launch, but with an extra full height terminal on the left hand side.

Setting default folders

To launch the terminals with specific work folders, add a :d flag.

Example:

"C:\Program Files\ConEmu\ConEmu64.exe" -cmdlist
cmd -cur_console:d:C:\Ruby23-x64\ |||
cmd -cur_console:s1TVn:d:C:\Ruby23-x64\project\app |||
cmd -cur_console:s1THn |||
cmd -cur_console:s2THn

Setting default task executions

As with setting default folders, you can edit the predefined task for the default shell. In ConEmu, go to Settings and under Startup select Tasks. Edit the {Shells::cmd} task.

Example:

>cmd -cur_console:d:C:\Python33\project
/k "Scripts\activate" & cd project

cmd -cur_console:s1TVn

cmd -cur_console:s1THn:d:C:\elasticsearch-1.4.4\bin &
/k "service start"

Check the Default shell checkbox and save the changes.

In the example, launching ConEmu would activate the virtual environment in a Python project, change the working directory to this project, and in a separate terminal it launches the ElasticSearch service.

Set background transparency

Open Settings and under Features select Transparency.

Now, ensure the Active window transparency checkbox is selected and drag the slider to change the transparency. Something like 75% opaque should be sufficient to comfortably read console text whilst being able to, for example, read a webpage behind the console.

Disable focus on hover

If you keep losing focus on your selected console, this is likely due to the setting which changes the active console when your mouse hovers over a different one from the one you have selected. This has become a default setting.

To disable this, in Settings under Keys & Macro select Mouse and uncheck the checkbox for Activate split/pane on mouse hover.

uninitialized constant Middleman::Sitemap::Store::Pathname

2017-04-24T00:00:00Z

Let's hope I'll never have to refer back to this page.

What I did

irb(main):001:0> require 'google-drive'

What happened

NameError: uninitialized constant
Middleman::Sitemap::Store::Pathname

What 'fixed' it

gem uninstall middleman-core

Which was at version 4.2.1, followed by:

gem install middleman-core -v 3.4.1

This fixed the immediate issue, but was followed by new problems.

What I should have done instead

irb(main):001:0> require 'google_drive'

There's a big difference between the gems google-drive and google_drive.

Enabling a Let's Encrypt SSL certificate on a Rails app hosted on Heroku with a Namecheap DNS

2017-04-21T00:00:00Z

This guide applies if you want to secure the Namecheap domain for your Rails app with a free Let's Encrypt SSL certificate and force HTTP access to HTTPS.

The following steps were last verified as valid on 21 April 2017.

Heroku

Log in to your Heroku account
Select the app you want to set an SSL certificate for
Go to Settings
Scroll to Domains and certificates
Click Configure SSL
Choose Automatically configure using Automated Certificate Management
Click Continue

Namecheap

Log in to your Namecheap account
Go to the Dashboard
For your app's domain, select Manage
Go to Advanced DNS
Change the CNAME Record for Host www to www.yourdomain.com.herokudns.com
Save your change

Heroku again

Click I've done this for the DNS settings change
Click Continue
Wait for the changes to take effect. You can use Refresh status to view progress or curl -vI https://www.yourdomain.com for more detail

Now the SSL certificate should be applied to your domain.

Rails

Edit config\environments\production.rb in your Rails app
Find the config.force_ssl toggle
Set config.force.ssl = true
Deploy the change

Now if you go to yourdomain.com or www.yourdomain.com you'll be brought to https://www.yourdomain.com and the browser's certificate verification will indicate a secure connection.

Why your EV3 might appear to shut down arbitrarily

2017-04-17T00:00:00Z

The EV3 programmable brick should be powered with 6 1.5 V AA batteries for a total voltage of 9 V.

When total voltage drops below 5 V, you will get strange issues such as the programmable brick spontaneously shutting down.

Connecting to Wi-Fi can quickly drop the voltage.

If you use ev3dev, you can make it easier to learn when the EV3 is dipping below 5 V. The following is an example using ev3dev with Python.

In a shell or script, use PowerSupply() to access the power_supply class:

ps = PowerSupply()

Then get the battery's voltage at the time of measuring through:

ps.measured_volts

Now you can have the EV3 give you a warning if the voltage drops below a threshold you specify. For example:

while True:
  if ps.measured_volts < 5:
    ALL_LEDS = [Leds.LEFT, Leds.RIGHT]
    for l in ALL_LEDS:
      Leds.set_color(l, Leds.RED)
    Sound.play("Feed me. Feed me. Feed me.")

Read more about the available methods in the ev3dev documentation.

Basic process for running Python scripts on the LEGO Mindstorms EV3

2017-04-15T00:00:00Z

The official ev3dev Getting Started guide provides very clear and thorough instructions on how to load ev3dev (an OS based on Debian Linux which comes with Python bindings included) on your EV3.

Example setup:

LEGO Mindstorms EV3 (typical UK price: £229.99)
SanDisk 16 GB MicroSDHC card (typical UK price: £7.99)
COTOP Card Reader Hub (typical UK price: £16.99)
TP-Link TL-WN821N Wi-Fi dongle (typical UK price: £9.87)

Once you're set up with the EV3 on, ev3dev booted, and a connection established (whether it be Ethernet, Wi-Fi, or Bluetooth), you have three options:

Run commands from a Python shell on ev3dev
Write a Python script on ev3dev
Write a Python script on your PC and transfer it to ev3dev

Let's go over the three options.

Run commands from a Python shell on ev3dev

Start an SSH client, such as PuTTY
Create a session with:

Host Name: ev3dev
Port: 22

Make the connection, and when prompted authenticate with:

Username (default): robot
Password (default): maker

Start Python with python3
In the shell, load the ev3dev library with from ev3dev.ev3 import *
Start using commands

Write a Python script on ev3dev

Start an SSH client, such as PuTTY
Create a session with:

Host Name: ev3dev
Port: 22

Make the connection, and when prompted authenticate with:

Username (default): robot
Password (default): maker

Start writing a script with nano example-script.py
Begin the script with #!/usr/bin/env python3
Import the ev3dev library with from ev3dev.ev3 import *
Write your Python script
To exit and save the script, use CTRL + X and select Y. Press Enter when satisfied with the filename.
Make the script executable through chmod +x example-script.py
Execute the script through python3 example-script.py

Write a Python script on your PC and transfer it to ev3dev

Start writing a script in your editor of choice, such as IDLE or Atom or Notepad++
Begin the script with #!/usr/bin/env python3
Import the ev3dev library with from ev3dev.ev3 import *
Save the script somewhere on your local drive.
Launch an FTP client, such as WinSCP
On the programmable brick, go to Wireless and Networks
Navigate to the network you're connected to
Select Status
Note the IP.
In your FTP client, create a new connection with:

File protocol: SFTP
Host name: the IP you noted down
Port number: 22
Username (default): robot
Username (default): maker

Once connected, transfer the file(s) to /home
Make the script executable through chmod +x example-script.py
Execute the script through python3 example-script.py

Example commands

Assigning the EV3 components to variables

ts = TouchSensor()
cs = ColorSensor()
us = UltrasonicSensor() # always to be connected to `in4`
ir = InfraredSensor()
gs = GyroSensor()

l1 = LargeMotor('outC') # if connected to `outC`
l2 = LargeMotor('outD') # if connected to `outD`
mm = MediumMotor('outA') # if connected to `outA`

Text-to-speech

ev3.Sound.speak('Hello. I am a robot.').wait()

Keep active until cancelled

while True:
  # code

Run a motor for a specific time at a specific speed

m.run_timed(time_sp = 5000, speed_sp = 500)

time_sp: time in milliseconds
speed_sp: tacho ticks per second = 1.4 rotations per second = 1 tick per degree = 500 degrees per second

Keep running motor

m.run_forever(speed_sp = 900)

Run multiple motors at once

l1 = LargeMotor('outC')
l2 = LargeMotor('outD')
motors = [l1, l2]

for m in motors:
  m.run_forever(speed_sp = 200)

Stop motors

m.stop(stop_action = "coast")

Save debugging messages to a log

import logging

logging.basicConfig(filename='runtime.log',
level=logging.DEBUG)

if do_something():
  logging.debug("Starting to do something.")

Shell commands

Get kernel version

uname -r

The uname command reports basic information about a computer's software and hardware. The -r option gives the kernel version number and release level.

Or, in Brickman, select About and scroll down.

Get volume

amixer get Playback,0

Change volume

amixer set Playback,0 100%

Improving your testing by improving your thinking

2017-04-09T00:00:00Z

Testing is an analytical activity. You form running hypotheses about how a product works, what kind of problems could occur, what the cause of an observed problem could be, what the impact of the problem would be to the user and the business, and you verify or falsify one hypothesis after another by gathering and evaluating information.

Training a tester's analytical skill is typically done through exercises where the tester is encouraged to generate testing ideas and ask questions. This helps testers be better investigators and is fun to do.

However, another aspect of being a better investigator is to examine your thinking. Our brains are vulnerable to assumptions, biases, and logical fallacies. The more we're willing to admit this and learn to recognise them, the less time and effort will be wasted following up on inaccurate notions.

Examples from the email marketing world to drive the point home

A marketing email is sent out every month. Last month it included a discount code. The analytics show more people unsubscribed from the email than at any other time. A policy is introduced that discount codes should never be used again.

Problems:

When we hint at a discount code in an email subject line the open rate is likely to be higher and accordingly the absolute number of people unsubscribing may be higher whereas the unsubscribe ratio may be the same as usual or in fact lower.
How do we know other differences in the email send were not responsible for the increased number of unsubscribers?

A project team commits to meeting the WCAG 2.0 accessibility guidelines. Colour Contrast Analyser is used to determine contrast ratio for meeting success criterion 1.4.3. The tool shows a subheading fails all 4 checks (AA small text, AA large text, AAA small text, AAA large text). The developer is asked to fix this. The developer increases contrast of the font colour versus the background colour. It now fails 3 out of 4 checks. It is reworked again and fails again. Eventually a decision is made that, now and in future campaigns, 1 out of 4 checks should pass.

Problems:

There is no basis for deciding 1 out of 4 checks should pass. What is the font size of the text? Are you committing to AA or AAA level conformance?
The reason for the guideline is forgotten. It's to ensure text is legible for people with reduced visual acuity. Tools should be used intelligently to inform whether or not you've succeeded in doing this.

Email open statistics show that 7% of users use a new email client with limited CSS support. Considering the CSS issues, the project team says "no-one uses it, we won't support it". When faced with the usage statistics, the project team says "we'll increase the support threshold to 10%". When faced with the consequence that this means no more support for desktop or web clients, the project team says "what is the usage share when we divide it by OS and OS version?". This goes on until the data supports the decision.

Problems:

The decision to support an email client should be based on the data rather than the other way around.
If you know you can't support something, justify it by the cost of trying instead of with massaged data. You can then offer it as a choice to the business: this % of additional users catered for versus this % of additional development and testing cost and effort.

The respective thinking errors in these examples

Illusory correlation. Our brains allow us to act quickly on a perceived correlation. We saw a connection, a plausible explanation, and acted. Yet, since it was not a life-or-death situation, we could have investigated the issue. Not sending discount codes might mean we weaken the conversion rate. It would have been worth looking into, even if it would end up leading to the same conclusion.
Special pleading. An exception is made without a reasonable justification for that exception. This, in turn, is because of an arbitrary goalpost. Which, in turn, is caused by an ill-understood requirement. If as a tester you report issues identified by a tool, you should learn why they are issues.
Motivated reasoning. You start from the conclusion and try to justify it instead of letting the evidence guide you. In the example, bad decisions are eagerly suggested and distract from the obvious solution: to present the forbiddingly high cost of supporting a difficult email client to the business stakeholder and let them either agree to exclude it or accept the additional budget and delays.

We should be aware of logical fallacies such as these to be able to monitor our own thinking and make better decisions. This applies to everyone, of course, not just to testers.

In addition all the examples could be said to have an element of knowledge insufficiency. A tester benefits from learning about the product domain, the technical underpinnings of the product, the business strategy, and linguistic knowledge (language comprehension and expression).

400 Bad Request for a POST request to a WCF service

2017-03-30T00:00:00Z

Problem

{"ErrorDetail":"Incoming messages must have a body format
of Raw or Xml. Is a ContentTypeMapper set on the
WebHttpBinding?","Status":"INVALID_ARGUMENTS"}

Background

A POST request to a WCF service through Postman (though a request through curl gives the same result).

https://client-webservice.provider.com/WebService/api/rest/
category/getInterestingDetails?ParamID=123456)

Username and password are provided through Basic Auth, which adds another param in addition to the ParamID example parameter.

The service can return JSON. Content-Type is set to application/json.

Solution

Solution in my case (in Postman) was to empty the params beyond the authentication details and place those parameters in the raw request body instead.

Access-Control-Allow-Origin header JS issue in Watir

2017-03-28T00:00:00Z

Problem

Using Watir to drive a Chrome browser, I found a button was not clickable which was clickable just fine when driving Firefox or when starting Chrome directly rather than through Watir.

The Chrome Developer Tools console displayed these JS errors:

Cause

It looks like Chrome does not trust the POST request the button triggers, since it did not originate from the same domain. In our case that's just because we're testing from localhost.

Solution

To do this POST request you'd need to add the Access-Control-Allow-Origin header to the request, but we don't want to muck around with custom requests. Instead, for local testing purposes, we can make Chrome stop enforcing the same-origin policy.

If you look at the ChromeDriver Capabilities under chromeOptions object, you will see you can feed args into Chrome when starting it.

The full list of possible arguments you can send along includes --disable-web-security. This sounds dodgy, but is fine for our local testing.

Using it in Watir

@browser = Watir::Browser.new :chrome, :switches =>
['--disable-websecurity']

As you can see, the switches have to be provided as an array of strings, even if it's a single value.

Real life use case

Before do
  @browser = Watir::Browser.new :chrome, :switches =>
	%w(--ignore-certificate-errors --disable-web-security
	--disable-translate)
end

(In Ruby, %w(foo bar) is equivalent to ["foo", "bar"].)

Regression testing SOAP web services using SoapUI

2017-03-22T00:00:00Z

Before you can get started you will need:

The WSDL document describing the web service, e.g. https://client-uat-webservice.com/WebService/api/service.svc?wsdl
Any required credentials to access the web services
Any additional parameter test values you may need to send along with your requests (e.g. If you're going to access a method for finding a user by their member id, you'll of course need member id test values).

Then:

In SoapUI, start a new project
In the SoapUI project, load in the WSDL to retrieve the API endpoints and automatically generate requests
For each SOAP request, in the Request Properties select a path for Dump File so as to store the response
Execute the SOAP requests
Verify the responses are as expected

The resultant response logs are your baseline.

After an update takes place which shouldn't affect the web service:

Ensure you've moved the previous results to a separate location, so that you won't override them
Re-run the same suite of SOAP requests
Verify the responses returned match your baseline responses

Also useful:

Creating a dynamic Dump File path

Select your TestSuite
Go to Custom Properties
Add a new Property
Add Property Name: OutputPath
Add Property Value: path to your project workspace, e.g. C:/SoapUI/ClientAPI/
For each Test Step where you want to save the response to file:
1. Select the Test Step
2. Go to TestRequest Properties
3. For the Dump File property, enter ${#TestSuite#OutputPath}${=request.name}.xml (reads the OutputPath property in the Test Suite scope, appends the name of the current request, and appends an extension)

Now when you execute a Test Cases, files will be generated in the selected location.

"RbReadline: HOME" environment variable issue on Windows

2017-03-20T00:00:00Z

Using Ruby on Windows, you may receive this error:

C:/Ruby23-x64/lib/ruby/site_ruby/2.3.0/
rbreadline.rb:1097:in `<module:RbReadline>':
HOME environment variable (or HOMEDRIVE and HOMEPATH)
must be set and point to a directory (RuntimeError)

To fix it, check your HOME, HOMEDRIVE, and HOMEPATH environment variables. To list them in a command prompt, type:

set home

An example result:

HOME=C:\windows\system32\config\systemprofile
HOMEDRIVE=C:
HOMEPATH=\Users\firstname.lastname

HOMEDRIVE + HOMEPATH (e.g. C:\Users\firstname.lastname) would have to be a valid path that you can access and which doesn't have special characters in firstname or lastname.

The HOME path also needs to be a valid path that you can access. In my case, I found that accessing the path required administrator rights. I had to go to the config folder, gain access by entering admin details, go to the systemprofile folder, gain access by entering admin details. Then I tried irb and was able to start it without issue.

Render comma-separated partials for an array of objects in Rails

2017-03-18T00:00:00Z

Say you have Posts which have Tags associated with them. In the Post index view you display all Posts, with the Tags for each.

Title: Cities of Africa
Tags: Cape TownFesNairobiKigaliKhartoum

Title: Cities of South America
Tags: Buenos AiresRio de JaneiroQuitoSantiagoBogotá

You'll notice a problem here. All the tags are listed without any kind of separation. The reason is because in the view we are rendering a partial for each tag whereby each partial simply lists and links the tag name, and these are then listed out.

tags\_tag.html.erb

<%= link_to tag.name, tag_path(tag) %>

posts\index.html.erb

<%= render post.tags %>

To solve this problem, We can't just join the tags through render post.tags.join(", "). That would make Rails look for a partial by the name of our joined tags. It would work without the partials through post.tags.join(", "), but to keep using our partial we need to join the tags separately per post.

posts\index.html.erb

<%= post.tags.map { |t| render t }.join(", ").html_safe %>

This gives us a nicely comma-separated list, whilst maintaining the tag links.

Title: Cities of Africa
Tags: Cape Town, Fes, Nairobi, Kigali, Khartoum

Title: Cities of South America
Tags: Buenos Aires, Rio de Janeiro, Quito, Santiago, Bogotá

Five fundamentals in devising a Support Matrix

2017-03-07T00:00:00Z

When designing your compatibility testing strategy, you're likely to formalise the environments that are going to be developed and tested against. You may call the result your Support Matrix, or Browser Compatibility Matrix, or simply your Supported Environments.

Here are five of the fundamentals in reaching an effective Support Matrix.

Use targeted tracking data to inform your decisions

An obvious one, yet you often see a Support Matrix which simply lists the newest devices and software versions out in the market at the time of writing. Yet new devices or software can take a long time to reach the same uptake as their predecessors (Android 7 for example), and if one of your (or your client's) primary markets is in, say, China, you're going to make your product harder or impossible to use for a significant portion of potential users if you don't take older versions of Internet Explorer or Android into account.

It's all about the size of the quality audience you can cater for. Meaning your product is going to be more successful the more users you're able to present with a great experience — where those users are the ones you care about (for example, those likely to convert) — in relation to the development and testing effort.

So if the tracking data for a client's existing digital presence in a market has a significant and stable percentage of users using Internet Explorer on an older version of Windows, there is value in putting development and testing time against it. It would make sense to ignore minor rendering issues which do not obscure content, and in some cases even to create a simplified version to direct older platforms to, but you cannot simply ignore an older platform because it is old.

For the same reasons, if a vendor stops support for their product, that in itself doesn't provide adequate justification for dropping it from your Support Matrix (unless you work for that vendor). If people keep using it - remember how corporations kept paying for Windows XP support after the official end of support - it could be valuable or essential to keep accommodating those users for a while longer if they are indeed your users.

When you have tracking data available, use it. Don't say "nobody uses device X" if you only know what you and your friend use. Evidence first.

Don't support everything

Not everyone involved in a project might be aware of the sheer number of combinations of devices, OS versions, browser versions, email clients, etc. there exist in the world. They might expect a product to get tested against a nebulous Everything. It may be wise to pre-empt this by communicating your support strategy, conveying how you use targeted data to maximise return on investment.

One tactic for reducing the number of supported environments is to establish a support threshold. You could employ an arbitrary but useful cut-off point of, for example, 5% of total usage share and support only those environments which are used by at least 5% of your target audience. Where you put this threshold depends on how much your organisation or client are willing to spend on accommodating their users and how much it would cost you to do so.

In some cases you'll want to advocate for comprehensive support more strongly than in others. A global eCommerce web build is a different beast than a static HTML5 banner.

Understand the technology

If the two most used devices in your tracking data are iPhone 6s and iPhone 7, it's a given that you'd support both, right? Not necessarily. iPhone 6, 6s, and 7 have the same screen size, screen resolution, and pixel ratio. If you're testing emails, that should tell you they are essentially identical if the iOS version is the same. In this case, supporting iPhone 7 would also cover iPhone 6s and 6. In the context of a demanding mobile app, however, the minor differences in hardware capabilities could be important.

Understanding the devices, browsers, and OSes in your tracking data will help you decide which to support and how to support them.

Another example is browser auto-updating. Chrome, Firefox, and Edge (through Windows Updates) update themselves. Whilst it is possible to be behind on an update or to disable auto-updating altogether, the vast majority of users will be moved to a new version of these browsers whenever it gets a new stable release. This should tell you your Support Matrix can be out of date as soon as you create it if you include specific versions of these browsers. Either support the latest version or - if you have the need and capacity - the latest and the version before it.

There are numerous more examples. Aside from learning about the technology on the environments people use, also learn about the technology you use to develop the product, and the technology you use to gather your tracking data.

Share with your project team

Having a Support Matrix to test against is going to be a painful experience if the product under test wasn't developed to support the same environments. We're not playing a game of "guess what rules I will fail your work against", we're working together to get something of value out the door. Before development even starts, it should be understood by everyone what environments the product will need to deliver a great experience in (whatever that means in your project) and to what extent.

Make sure you get involved in kick-offs for projects and ensure you have or get access to appropriate tracking data to base a Support Matrix on. Keep an eye on resource requests for your design department. What they are working on is likely to go into development soon.

If the work is for a client, also find out what environment(s) they are going to use when signing off on the deliverables. Then either add that to your supported environments or request that the client stakeholders use an environment that fits what their target audience use. Even if you do, experience shows that someone will pop up from somewhere with a list of issues in an unsupported environment. When that happens, you'll be happier if your account handler is already equipped to handle it, namely by understanding your explanation on why only certain environments are supported and what it's going to cost if they want to add additional support retroactively.

Actually be able to support them

What if 12% of the target audience use a low spec Android tablet and you don't have one? Do you sell it as "88% of people DON'T use it" and ignore it? You could, but not having it is a weak justification for not supporting it. Either you exclude it for a valid reason or you find a way of covering the environment with a representative device.

There are many options for building compatibility testing capacity.

You can create a physical test lab, with a range of test desktops, virtual machines, laptops, and mobile devices which you manage yourself in terms of software versions, configuration, purchasing, and availability.

You can use cloud-based platforms like BrowserStack or Sauce Labs (or a number of other options) for desktop and mobile browsers or Litmus or Email on Acid (or a few other options) for email clients.

You can use a 3rd party testing service provider who already have a mature physical or cloud-based test lab.

You can use a 3rd party testing service provider who have a distributed network of testers they can use to cover any range of environments.

Or you can use a combination of these. The world is your oyster, if you can afford it.

Examples of LUA usage in LoadImpact test scenarios

2017-03-07T00:00:00Z

GET request

http.request_batch({
	{"GET", "http://subdomain.domain.co.uk/",
  auto_decompress=true}
})

To take the effort out of gathering all relevant page requests, you can use LoadImpact's User Scenario Recorder and manually adjust the requests.

POST request

http.request_batch({
	{"POST",
  "https://subdomain.domain.co.uk/",
	headers={["X-Requested-With"]=
	"XMLHttpRequest, XMLHttpRequest",
	["Content-Type"]="application/
	x-www-form-urlencoded; charset=UTF-8"},
  data="X-Requested-With=
  XMLHttpRequest&nonJsSearch=false
  &searchTerm=", auto_decompress=true}
})

Log in

http.request_batch({
	{"POST",
  "https://subdomain.domain.co.uk/AjaxSignIn",
	headers={["X-Requested-With"]="XMLHttpRequest,
	XMLHttpRequest",["Content-Type"]=
	"application/x-www-form-urlencoded;
  charset=UTF-8"},
  data="Password=userpassword&RememberMe=false
  &SignInUrl=%2F&UserName=useremail
  &X-Requested-With=XMLHttpRequest",
  auto_decompress=true}
})

Sleep

client.sleep(3)

Of course the duration can be adjusted at will. It is measured in seconds.

Identify the boundaries of a page metric.

http.page_start("Page 1")

page requests go here

http.page_end("Page 1")

Looping

for loopCount = 10 do
	something ten times
end

for loopCount = 11, 21 do
	something ten times, enumerated as 11 to 21
end

String interpolation

for loopCount = 10 do
	http.page_start("Page .. loopCount .. ")
	page requests go here
	http.page_end("Page .. loopCount ..")
end

Redirect HTTP & HTTPS traffic to specific server IP

util.dns_remap("http://subdomain.domain.co.uk",
"95.95.95.95")
util.dns_remap("http://subdomain.domain.co.uk:443",
"95.95.95.95")

Preventing SessionNotCreatedError in WebDriver when using Appium

2017-02-21T00:00:00Z

Example error (for test using Appium with Selenium WebDriver and Rspec) when a previous test was aborted before a clean exit:

Failures:

  1) The APPNAME app displays SCREEN when user selects OPTION
  Failure/Error: driver =
  Appium::Driver.new(options).start_driver

     Selenium::WebDriver::Error::SessionNotCreatedError:
       Requested a new session but one was in progress
     # ./spec/spec_helper.rb:20:in `block (2 levels)
       in <top (required)>'

Finished in 0.06314 seconds
(files took 0.73578 seconds to load)
1 example, 1 failure

To prevent this error in subsequent test runs:

Stop the Appium server if it is running
In Appium, go to Preferences
Select General
Check the Override Existing Sessions checkbox
Start the Appium server

Finding the appPackage and appActivity values for an app on a connected device

2017-02-21T00:00:00Z

Say you're looking to run Appium tests against an app on an Android device connected to your desktop and you need to know the appPackage and appActivity values for the app.

Assuming you have the Android SDK installed (and you have the SDK's platform-tools directory in your PATH environment variable, if you're on Windows), you can run the following from the command line.

adb devices

This should return the hostname of the device you have connected.

adb shell

This will allow us to send commands to the device.

Make sure that the app under test is started and in focus on the device, then run the following.

dumpsys window windows | grep -E ‘mCurrentFocus|mFocusedApp’

Example output:

mCurrentFocus=Window{31ee112 u0 d0
com.Client.Appname/com.unity3d.player.
UnityPlayerNativeActivity}
mFocusedApp=AppWindowToken{b5717d6 token=Token{65b7ff1
ActivityRecord{2c5b98 u0
com.Client.Appname/com.unity3d.player.
UnityPlayerNativeActivity t39}}}`

For this example, the values for appPackage and appActivity are:

appPackage: com.Client.Appname

appActivity: com.unity3d.player.UnityPlayerNativeActivity

Google Analytics API requests using Ruby on Windows

2017-01-18T00:00:00Z

The hardest part of making Google API requests via Ruby on Windows is authentication. For this simple example, we'll use an API key rather than an OAuth 2.0 process.

Retrieve the API key

These steps assume you have a project folder including a Gemfile, and that you have a browser open and are logged in with the Google account for the Google Analytics you're interested in.

In your Gemfile, add gem 'google-api-client'
Run bundle install from a command prompt
From the API Manager, select Enable API
Select Analytics API
In the top bar, click Project (or Name of project)
From the dropdown, select Create project
Enter a name and click Create
From the IAM & Admin menu, select Service accounts
Click Create service account
Enter a name for the service account
For Role, select Project followed by Viewer (for the purposes of this example, since we're not interested in making changes)
Copy the service account ID to Notepad
Check the Furnish a new private key checkbox
Click Create
Add the downloaded private key to your project folder

Allow service account to access your Google Analytics

In Google Analytics, select Admin (has cog icon)
Select the account you want to be using
Click User Management
Take the service account ID from Notepad and add it to the Add permission for input field
Click Add

Get Mozilla's CA Bundle certificate

Navigate to C:\Ruby23-x64\lib\ruby\2.3.0\rubygems\ssl_certs (adjust to fit your Ruby install path)
Download Mozilla's CA Bundle Certificate as ca-bundle.crt into the folder
Assuming openssl is in your PATH, from the command line, in the same folder, run: openssl x509 -in ca-bundle.crt -out MozBundleCA.pem -outform PEM

Note: If openssl is not in your PATH, ensure openssl is installed and add the installation directory (e.g. C:\openssl\bin) to your PATH environment variable. Make sure you restart your command prompt afterwards, so that the environment variables are reloaded.

Create a simple example script

Create a new .rb file and add the following dependency:

require 'google/apis/analytics_v3'

In Google Analytics, find the view you want to retrieve data for by navigating through Accounts > Properties & Apps > Views. It will have an 8 digit number. Replace the hash signs below with this number and add it to your script.

account = 'ga:########'

Then add the following variables:

start_date = ((Date.today - 90).strftime('%Y-%m-%d')).to_s
end_date = (Date.today).to_s
metric_list = %w(ga:sessions)
dimension_list = %w(ga:screenResolution)
filter = 'ga:deviceCategory=~mobile'
sort = '-ga:sessions'
results = 10

This will give us the number of sessions per screen resolution in the last 90 days for mobile devices only, from most used to least used, limited to 10 results once we do the following:

resolutions = service.get_ga_data(account, start_date,
end_date, metric_list, dimensions: dimension_list,
results: 10, filter: filter, sort: sort)

Now you can do anything you want with the resolutions variable, and build something new using the documentation for the Google API Ruby Client and the Google Analytics API. Note that my example uses v3 and the API has since moved on to v4. However, v3 is still supported.

Receiving email via SendGrid and Namecheap

2017-01-09T00:00:00Z

If you have a domain with Namecheap and want to relay messages received on that domain to SendGrid so you can then process them as objects (for example via Griddler in a Rails app), read on.

SendGrid

We'll whitelabel your domain, then we'll return to set up an inbound parse.

Whitelabel

Log in to your SendGrid account (or sign up, if you don't have one)
Go to Settings
Go to Whitelabels
Click Add Domain
Choose a subdomain, e.g. email
Enter your domain, e.g. examplesite.com (no protocol)
Allow Automated Security
Click Save

This will need to be validated, after we've configured Namecheap and the DNS updates have propagated.

Namecheap

First, go to your advanced DNS settings. Steps below if needed.

Log in to your Namecheap account
Click Account
Click Dashboard
Select Manage for the domain you want to use
Click Advanced DNS

You may already have some host records, for example a CNAME record for www and a URL Redirect Record for @. Either way, we can add some additional ones without conflict. Add the following records, replacing the X and # characters with your own details from SendGrid.

Type	Host	Value	TTL
CNAME Record	email.	X#######.XX###.sendgrid.net.	Automatic
CNAME Record	s1._domainkey	s1.domainkey.X#######.XX###.sendgrid.net.	Automatic
CNAME Record	s2._domainkey	s2.domainkey.X#######.XX###.sendgrid.net.	Automatic

The full stop placement is essential.

Now we need to set an MX Record. Under Mail Settings select Custom MX and add the following record.

Type	Host	Mail Server	Priority	TTL
MX Record	@	mx.sendgrid.net.	10	Automatic

Note that here I'm relaying all email to SendGrid. This is not necessarily what you want to do.

Once you've added these records, it can take up to 48 hours for the changes to propagate. In my case it took about 2 minutes. For a quick check on whether you're there or not, you can use MxToolbox's MX Lookup.

Inbound Parse

Back in the SendGrid settings, make sure you validate the Whitelabel settings. If you don't get a nice green checkmark, either the settings in Namecheap and SendGrid are incorrect or you need to be patient for the DNS changes to propagate.

Once validated:

Go to Settings
Go to Inbound Parse
Click Add Host & URL*
Leave the subdomain empty
For domain, select your whitelabelled domain
As destination URL, choose the URL on your domain which is going to handle incoming emails, e.g. http://www.examplesite.com/email_processor

Now repeat these steps again, but with www for the subdomain.

Once done, you can test everything is working by sending an email to your domain. In SendGrid, this should then show up in the Activity Feed (Activity menu item).

Configuring the wkhtmltoimage path for IMGKit on Windows

2016-10-14T00:00:00Z

If you'd like to use IMGKit, you'll also have to install wkhtmltopdf, which includes wkhtmltoimage - a command line tool to generate images from an HTML page.

From IMGKit's documentation:

If you're on Windows or you installed wkhtmltoimage by hand to a location other than /usr/local/bin you will need to tell IMGKit where the binary is.

If you're using Rails, you can define this in config/initializers/imgkit.rb If you're not using Rails, you can use the same definition where you're requiring imgkit.

IMGKit.configure do |config|
  config.wkhtmltoimage = '/path/to/wkhtmltoimage'
end

In Windows, that path has to include the filename and extension. If you don't add it, you'll receive a NoExecutableError or a "Permission Denied" error.

Example path:

C:\\Program Files\\wkhtmltopdf\\bin\\wkhtmltoimage.exe

You can test the configuration in irb like this:

irb(main):003:0> html = "http://example.com/index.html"
=> "http://example.com/index.html"

irb(main):004:0> kit = IMGKit.new(html, :quality => 50)
=> #<IMGKit:0x00000002fa0120
@source=#<IMGKit::Source:0x00000002fa00d0
@source="http://example.com/index.html">, @stylesheets=[],
@javascripts=[], @options={:height=>0, :quality=>50}>

If it doesn't work, try adding the path to your PATH environment variable.

If it still doesn't work, you can amend the path in the gem's configuration file directly.

# C:\Ruby##-x64\lib\ruby\gems\#.#.#\
gems\imgkit-#.#.#\lib\imgkit\configuration.rb
def wkhtmltoimage
  @wkhtmltoimage ||= begin
    path = (using_bundler? ? `bundle exec which wkhtmltoimage`
    : `which wkhtmltoimage`).chomp
    path =
    'C:\\Program Files\\wkhtmltopdf\\bin\\wkhtmltoimage.exe'
    if path.strip.empty?
# Fallback
    path
  end
end

Using RSpec to verify a result matches one of multiple expectations

2016-09-29T00:00:00Z

Say you want to verify that a method returns either 'Yes' or 'Maybe'.

You could of course do:

valid = ["Yes", "Maybe"]
expect(valid).to include(response)

This turns the expectation around to check that the valid responses include the returned value.

However, this is likely inconsistent with your other RSpec expectations and is not as semantically lucid as the following option.

expect(response).to eq("Yes").or eq("Maybe")

You can find out more about compound expectations in the official RSpec documentation.

Cannot load such file -- 2.3/pg_ext (LoadError)

2016-09-03T00:00:00Z

Setting up PostgreSQL with Ruby 2.3, you may encounter this error:

LoadError: cannot load such file -- 2.3/pg_ext

This is due to the native extensions being built for Ruby 2.2 instead.

How I solved this:

Uninstall the pg gem

gem uninstall pg

Instead of pg gem v0.18 install a pre-release version of v0.19

gem install pg --pre

In your Gemfile, ensure it will pick up the pre-release version

gem 'pg', '>= 0.19.0.beta'

Why a .present? might fail on an exact match

2016-07-31T00:00:00Z

Say you have an element for which you are checking .present? returns true. It returns false, but you can see it's definitely in the DOM and it returns true for .exists?.

This is because the element is not visible.

One possible way this might happen is that the element is in a hidden section. The HTML might, for example, have a duplicate section that is conditionally hidden and has identical structure, classes, and text.

.exists? will pass for elements in this section. .present? will not.

In this case you'll want to determine which element should be the one that's visible and whether you're specifically selecting that element rather than its hidden duplicate.

Attributes making elements invisible
`visibility: hidden;`
`display: none;`
`width: 0;` and `height: 0;`
`opacity: 0;`
`<input type="hidden">`
`webkit-transform` properties on element or parent

Creating persistent device category filters in Google Analytics

2016-07-30T00:00:00Z

Here's how to save filters for metrics divided by device category (mobile, tablet, desktop) in Google Analytics. For a simpler, throw-away method have a look at Filtering for mobile and tablet data in Google Analytics.

OS\Browser - Desktop

In the main navigation, select Customization
Choose +New Custom Report
Give a title and name(s) to the report and tab(s) as you please, for - example: OS\Browser - Desktop
For Type, select Flat Table
For Dimensions, select +add dimension and add Browser Version
For Dimensions, select +add dimension and add Operating System Version
For Metrics, select +add metric and add User > Pages / Session
Under Filters select +add filter
Under Users, select Device Category
Choose Exact and enter desktop
Click Save

Device\Resolution\OS - Mobile\Tablet

Under the created filter, select Edit
Choose +add report tab
You could call this one Device\Resolution\OS, or whatever you wish
For Type, select Flat Table
For Dimensions, select +add dimension and add Mobile Device Model
For Dimensions, select +add dimension and add Screen Resolution
For Dimensions, select +add dimension and add Operating System Version
For Metrics, select +add metric and add User > Pages / Session
Under Filters, we see a copy of our existing Device Category filter
Under Filters, change dropdown selection Exact to Regex
Under Filters, change Device Category value from desktop to mobile|tablet
Click Save
Navigate to the newly created tab

Combining multiple path sections with os.path.join()

2016-07-24T00:00:00Z

Assume these variables are supplied:

base_path = "C:\\Python34\\myscript\\"
filename = "sassykitten.jpg"

This won't work:

path = os.path.join(base_path, "/images", filename)

The variable path will now contain:

"C:/images\\sassykitten.jpg"

This is due to the slash in /images, which will make the rest of the join start at root. You can join any number of strings, but they should not include slashes.

Do this instead:

path = os.path.join(base_path, "images", filename)

You will get the desired:

"C:\\Python34\\myscript\\images\\sassykitten.jpg"

Accessing an iframe loading an Angular app within a Web Form

2016-04-26T00:00:00Z

A bespoke CMS I worked on included a Web Form containing iframes of which one contained an Angular app. I wanted to access this Angular app using Watir-WebDriver and perform actions on its elements.

<div class="content">
  <script type="text/javascript">
    //<![CDATA[
    Sys.WebForms.PageRequestManager._initialize(
      "ctl00$ContentPlaceHolderInner$_scriptManager1",
      "aspnetForm",
      [],
      [],
      [],
      90,
      "ctl00"
    );
    //]]>
  </script>

  <iframe
    id="form-iframe"
    src="http://weneedtogodeeper/"
    style="border: none; height:1069px; width: 100%;"
  >
  </iframe>
</div>

Whilst the method for accessing iframes with Watir-WebDriver used to be frames, I couldn't access the desired iframe with @b.frame(:id => "form-iframe") and @b.frames was empty.

Instead, use @b.iframes. For example, @i = @b.iframes(:id => "form-iframe") allowed me to work with the desired iframe as a new instance variable @i.

Restoring a PostgreSQL database from production in Windows

2016-02-19T00:00:00Z

Create a backup of a production PostgreSQL database:

SSH into your server (e.g. by using PuTTY).
Use pg_dump with the correct user and database. Example:
pg_dump --username "myadminuser" --format custom --blobs --verbose --file "mybackupfile.sql" "mydatabase"
Submit the correct password when prompted.
The file is created in the directory you executed pg_dump from unless you specified a path.
You can pick this file up via an FTP client or by using pscp from a command prompt.

Restore the backup to a local development database:

Use psql with the correct user, file and database. Example:
psql --username "mylocaladmin" "mydatabase" << "mybackupfile.sql"

"Could not access directory" PostgreSQL error in Windows

2016-02-19T00:00:00Z

After upgrading from PostgreSQL 9.2 to 9.3.4, I was no longer able to access my database from psql or pgAdmin:

FATAL:  could not access directory
"pg_tblspc/16639/PG_9.3_201306121/16843": Permission denied

This directory exists.
All user groups have full permissions to the directory as well as all its parent directories.

PostgreSQL now requires a NETWORK SERVICE account, which it did not create. Since I was using PostgreSQL on the Home version of Windows 7 (64-bit), I cannot access the Local Group Policy Editor.

One possibility is to transfer your database to a different OS.

I solved it by restoring my database from production.

Test automation beyond end-to-end regression checks

2016-02-06T00:00:00Z

In discussions about test automation, I've found test automation to be typically understood as building and maintaining suites of end-to-end UI-level regression checks using Selenium WebDriver.

It can be that. It can also be many other things, without broadening the term to the use of test tools in general. In less time and with relatively little programming skill, you can still increase the speed, coverage, and reliability of some checks that otherwise would have to be done manually.

Examples I've used in a professional environment where basic scripting skills brought value outweighing the time invested:

Build validation through: checksum comparisons, filename comparisons, checking pattern of folders and files, checking for empty files.
API testing. Calling endpoints via GET and POST requests, checking the HTTP response code and performing pattern matching on the response body.
Build Verification Test, running a priority subset of functional regression checks triggered on every completed build.
Fetch email campaigns from test accounts, parse the HTML, and compare links, tracking, and copy against consistently formatted content matrices.
Check pages against brand guidelines, uniquely identify sections and check them for colours, font size, font style, character limits, etc.
Read a list of links from a spreadsheet, visit them all, parse all links on the pages, check the HTTP status on GET requests on them.
Compare spreadsheets to find items that should have been added or removed from a dropdown, then check dropdown values against it.
Fill in form fields with test data from a CSV file and submit the form, asserting the entered values are sent and returning the response.
For a decision tree, calculate what the product recommendations would be based on scoring conditions for each type of question answered.
Follow all distinct journeys through a decision tree and compare product recommendations at the end of it to the expected recommendations.
Verify links and tracking specifications against a list of mistakes commonly found in them.
Spoof the browser's geolocation repeatedly for a large number of lat/long coordinates and assert a location-based module responds correctly on being loaded from these locales.

You can also use scripting for collateral tasks:

Produce metrics from issue trackers or test reports.
Produce live dashboard of progress on and status of current projects.
Allow non-technical people to generate test seeds lists via a simple form.
Use tracking suite APIs (or web scraping) to retrieve tracking data.
Merge major browser versions in tracking data.
Identify iOS devices from their logical screen resolutions in tracking data.

Ruby Dir methods being performed on the current directory

2016-02-03T00:00:00Z

Dir operations are performed on the directory they're called from, regardless of what directory you feed in. So Dir.glob or Dir.foreach or Dir.entries on (base_dir + '*.something').count won't return expected results.

You'll need to change directory, then perform the operation, e.g.

Dir.chdir(base_dir)
Dir.glob('*.something').count

Find files through Ruby excluding those with a specific extension

2016-02-02T00:00:00Z

If Dir.pwd is the path you want to search including its subdirectories:

ignore_ext = [".xlsx"]

Find.find('.') do |file|
  next if ignored_ext.include?(File.extname(file))
  || File.directory?(file)
  puts file
end

How Nokogiri's type method corresponds to document types

2016-01-22T00:00:00Z

Update 2026: This is no longer the way Nokogiri works. As per the Nokogiri documentation, you can now get the document type by simply using .html? or .xml?

Say you have a Nokogiri document, for example through:

doc = Nokogiri::HTML(open(index.html))

When you now call doc.type, a digit will be returned, in this case 13.

What document type this translates to can be seen here:

(link that was here, but no longer valid: http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Node#DOCUMENT_TYPE_NODE-constant)

Notable examples:

9 - XML document
13 - HTML document

Making ElasticSearch work in a Django app on WebFaction

2015-05-06T00:00:00Z

These are some things I had to do to make ElasticSearch work in a Django app I deployed to WebFaction. After telnetting to the server:

Set ES_HEAP_SIZE environment variable to lower than 512 MB, e.g.

set ES_HEAP_SIZE=256mb
export ES_HEAP_SIZE=256mb

Check ES_HEAP_SIZE environment variable is set:

printenv ES_HEAP_SIZE

Switch to Java 1.8:

echo "PATH=\$HOME/jdk1.8.0_45/bin/:\$PATH" >> .bash_profile
source .bash_profile

Run ElasticSearch as daemon:

From elasticsearch-1.5.2 directory:

bin/elasticsearch -d

Inspect memory used:

ps aux --sort -rss

Top RSS value should be 255640 or thereabouts (i.e. less than 256 MB).

"Unable to obtain stable firefox connection error" in Selenium WebDriver

2015-04-02T00:00:00Z

The error:

unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055)
(Selenium::WebDriver::Error::WebDriverError)

When I've received this error, it has been due to Firefox updating itself to a new version and Selenium WebDriver not having been updated to support this new version.

The solution:

gem update selenium-webdriver

If you use Bundler, do:

bundle update selenium-webdriver

Still doesn't work? Check code.google.com with your firefox/selenium-webdriver version combination to see if there are any updates about it.

Meanwhile, you can get dev updates for the selenium-webdriver gem from:

https://rubygems.org/gems/selenium-webdriver/versions/

Last resort: downgrade Firefox.

Time zone / tzinfo-data issue running Rails on 64-bit Windows

2015-02-08T00:00:00Z

When running a Rails server on a 64-bit Windows installation, you may encounter:

rescue in create_default_data_source': No timezone data source could be found. To resolve this, either install TZInfo::Data (e.g. by runninggem install tzinfo-data) or specify a zoneinfo directory usingTZInfo::DataSource.set(:zoneinfo, zoneinfo_path)`. (TZInfo::DataSourceNotFound)

To solve this:

Ensure bundler is installed (gem install bundler).
In your Rails project's root folder, edit the Gemfile.
Find the tzinfo-data line in the Gemfile.
Change the line to: gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw].
From the command line from within the same folder, run bundle.

If Windows' security settings won't allow you to save the Gemfile change, edit permissions of the folder by right-clicking it and selecting: Properties > Security > Edit > enter administrator details > give Users account write permissions > Apply

Add "WAVE" accessibility checks to Web Developer Toolbar

2015-02-03T00:00:00Z

The Web Developer Toolbar add-on adds various convenient tools to Firefox for development and testing purposes.

It includes accessibility checks under Tools > Validate Section 508 and Tools > Validate WAI. These link to Cynthia Says pages which stopped working in 2013, but are still present in the toolbar and can be easily replaced with functioning accessibility checks.

In Firefox, with the Web Developer Toolbar enabled:

Go to Tools
Select Edit Tools...
Select Validate Section 508
Click Delete and confirm deletion
Select Validate WAI
Click Delete and confirm deletion
Click Add...
Replace the Description value with Validate with WAVE
Select the URL radio button
Replace the URL value with http://wave.webaim.org/report#/
Click OK to save the new entry
Click OK to close the Edit Tools dialog

Now you're all done. Visit a website and try Tools > Validate with WAVE.

Target pattern contains no `%'

2015-01-21T00:00:00Z

When updating gems in Windows for Ruby 1.9.3 (or earlier), you may get an error message like:

creating Makefile

make  clean
Makefile:165: *** target pattern contains no `%'.  Stop.

make
Makefile:165: *** target pattern contains no `%'.  Stop.

make failed, exit code 2

To fix this, downgrade your RubyGems version:

gem update --system 1.8.29

Fixing a "certificate verify failed" error in Ruby

2015-01-15T00:00:00Z

Updating installed gems
ERROR: While executing gem ...
(Gem::RemoteFetcher::FetchError) SSL_connect returned=1
errno=0 state=SSLv3 read server certificate B: certificate
verify failed (https://api.rubygems.org/specs.4.8.gz)

Or:

ERROR: Could not find a valid gem 'activesupport' (>= 0),
here is why: Unable to download data from
https://rubygems.org/ - SSL_connect returned=1 errno=0
state=SSLv3 read server certificate B: certificate
verify failed (https://api.rubygems.org/specs.4.8.gz)

To fix this (in Windows):

Save the cacert.pem file to a folder, e.g. C:\Tools
Go into your System Environment Variables, and create a new environment variable:
- Name: SSL_CERT_FILE
- Value: C:\Tools\cacert.pem
Restart your terminal/console

If it still doesn't work, are you trying to access a Google API? In that case, use the cacerts.pem from the Google API Client library instead. Note it's called cacerts.pem, so you either save/rename it as cacert.pem or update SSL_CERT_FILE.

To fix this (in OS X):

See this StackOverflow answer.

"The native gem requires installed build tools"

2014-12-29T00:00:00Z

When you're installing a gem with:

gem install GEM

And you get the message:

The 'GEM' native gem requires installed build tools

Make sure you have DevKit installed and that it is registered:

Open a terminal / command prompt
Navigate to the folder where you placed DevKit, e.g. DevKit-mingw64
Run ruby dk.rb init
Edit the config.yml file
Add the Ruby path you want to use DevKit with
Save the file
Run ruby dk.rb install

“Invalid configuration or no Rubies listed” error

2014-12-29T00:00:00Z

When you install DevKit or edit the config.yml for DevKit, you may get the following error.

Invalid configuration or no Rubies listed.

To fix this, edit config.yml and ensure:

The paths are correct
Each path has a hyphen in front of it
The paths you want to use have no hash symbol in front of them

Filtering for mobile and tablet data in Google Analytics

2014-12-29T00:00:00Z

In Google Analytics you can separate data by device category (desktop, tablet, mobile).

Depending on what you're measuring, you may want to combine data for tablet and mobile.

In the advanced filtering options for Google Analytics, filters are joined by AND and this cannot be changed to OR.

However, you can use regular expressions to accomplish the same thing.

In the 'Reporting' section, view a report for a dimension (for example, on the left-hand side, select Audiences followed by Technology followed by Browser & OS).

There should be an 'advanced' option (at the time of writing it is located next to the search field). Click it and set the value for Containing to /(mobile|tablet). This will exclude data for desktop browsers.

This is a quick and throw-away method. If you want the filter to persist and configure it further, see Creating persistent device category filters in Google Analytics.

Doing an Active Record migration in Rails

2014-12-29T00:00:00Z

To change your Rails models, you should create a migration containing the change and execute the migration.

First, generate the migration. From within your app directory:

rails g migration DESCRIPTION

Edit the file, which has been created in db/migrate

Define your desired changes. You can place them in a function change (with reversible if you want to be able to undo the change), or place them in functions up and down, where up defines what should be changed and down defines how to return to the previous state if you decide to roll back this migration.

If you want to make the change as a SQL query, you can use:

execute <<-SQL
 YOUR SQL QUERY
SQL

Then on the command line run:

(OS X Terminal) rake db:migrate

(Windows CMD) ruby bin/rake db:migrate (Heroku CLI) heroku run rake db:migrate

More information can be found on the Active Record Migrations section of the RailsGuides site.

Comment syntax across languages

2014-12-29T00:00:00Z

CSS

/* Comment (in-line) */

/*
Comment (multi-line)
*/

JavaScript, Java, Swift, C

// Comment (in-line)

/*
Comment (multi-line)
*/

Python

# Comment (in-line)

"""
Comment (multi-line)
"""

Ruby

# Comment (in-line)

=begin
Comment (multi-line)
=end

Lua

-- Comment (in-line)

--[[
Comment (multi-line)
]]--

Compiler version per Visual C++ version

2014-03-16T00:00:00Z

When you need to compile a Python extension, you'll need to know which version of Visual C++ to use.

First, find out the compiler version for the version of Python you're looking to use the extension with.

On the command line, simply execute python

Sample output:

Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 b D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

If it does not list the Python version you're going to add the extension to, switch to the version you need.

Then take note of the MSC value. Find it in the table below and there you have the Visual C++ version you need.

Compiler version	Visual C++ version
MSC v.1000	Visual C++ 4.x
MSC v.1100	Visual C++ 5
MSC v.1200	Visual C++ 6
MSC v.1300	Visual C++ .NET
MSC v.1310	Visual C++ .NET 2003
MSC v.1400	Visual C++ 2005
MSC v.1500	Visual C++ 2008
MSC v.1600	Visual C++ 2010
MSC v.1700	Visual C++ 2012
MSC v.1800	Visual C++ 2013