Today’s conference began with some rather funny commentary shared by Yvette Nameth’s mother from yesterday’s talks. I was mentioned as the ‘flaky’ guy:
My main takeaway from the entire conference is that it seems we get way too caught up on complex solutions for our testing. We need to keep asking ourselves: “what’s the simplest thing that could possibly work?” If we have complex systems why do we need complex tests? We need to take each large complex problem we work on and break it down till we get something small and manageable and solve that problem. Rinse and repeat.
Here’s some of my favourite talks from today (in no particular order):
Nest Automation Infrastructure by Usman Abdullah and Giulia Guidi and Sam Gordon
A good insight into how Nest intelligently test their devices using sensor simulator boards, cameras pointing at tvs showing known footage, and simulated data. Performance/efficiency is a big deal as some of their smart connected devices like the Nest Protect needs to last multiple years using 6x AA batteries.
Coverage is Not Strongly Correlated with Test Suite Effectiveness by Laura Laura Inozemtseva
As the title suggests, Laura shared the results of her research paper comparing test suite coverage with test suite effectiveness for large open source projects. The findings were that ‘high coverage does not guarantee high quality”. The strength of the oracles are the most important factor. Link to paper.
Enabling Streaming Experiments at Netflix by Minal Mishra
Minal shared some impressive stats about the scale of nexflix and the benefits they are seeing from pushing experiments to client side code (Netflix player) using canary builds multiple times per week. Cool stuff.
Chrome OS Test Automation Lab by Simran Basi and Chris Sosa
Chris and Simran shared how Chrome OS run automated tests. They work off trunk using about ~100 git repositories – ensuring trunk is always stable and do releases by cherry picking to release beaches. They run builds on both emulators (fast) and over 2400 ChromeBooks and ChromeBoxes in their device lab. Each device has a debug board connected for connectivity and debugging as each new build can potentially brick devices. They also shared their ModLab solution which is a test lab that can run on a ChromeBox which can be used by vendors to quickly and easily test their hardware. It’s managed by the Chrome OS, auto-updates and is open-source.
Mock the Internet by Yabin Kang
Yabin shared a tool they’ve developed at LinkedIn (soon to be open source) called FlashBack which acts as a proxy to mock service calls. The best thing about this approach is it can be used for any service (internal/external) and can be used with no changes to your apps and on real devices. Neat.