Development Central

Many of the GOTO Chicago 2013 convention slides are available at
http://gotocon.com/chicago-2013/schedule/tuesday.jsp
and
http://gotocon.com/chicago-2013/schedule/wednesday.jsp

My notes may make more sense when combined with these. :)

Benefits and Challenges of Implementing Riak in Erlang
Steve Vinoski - http://basho.com/

Erlang
  Requirements
    Large number of concurrent activities
    Distributed systems
    Continuous operation for years
    Live updates and maintenance
    Hardware/software fault-tolerant
Concurrency - common theme
  Erlang processes are much more lightweight than OS threads
Reliability - what it gives you
  Isolation - processes communicate only by message passing
  Distribution - works across nodes
  Linking/supervision/monitoring - one process takes action when another fails
Small language; few elements; functional - relatively easy to learn
  Variables are immutable; no globals
  Flow control via pattern matching; recursion
Concurrency primitives
  Processes, not mutexes, etc.
  Selective receive lets you receive specific messages from anywhere in the message queue (even if other types are ahead of them)
OTP framework - everybody uses this

Riak
  CAP theorem
  A distributed highly available eventually consistent highly scalable open source key-value database written primarily in Erlang.
  Modeled after Amazon Dynamo. See Andy Gross’s 5 years later talk.
  Also provides MapReduce, secondary indexes, full-text search
  Built for operational ease - it just runs
  Multiple clients - .NET, Java, Node.js, etc.
  Consistent hashing - no sharding
  Stores replicas - N/R/W values - adjustable
  (The ring is very similar to the Security Now discussion of Tor.)
  Vector clock - how it determines stale value - number of operations actors performed and a timestamp
  All the nodes in a cluster are peers - no masters or slaves
  Nodes exchange their understanding of ring state via gossip protocol
    Riak uses the Erlang mesh for this
  Can simulate multi-node installment on a single machine (nice for development)
  At about 150 nodes, the cluster doesn’t scale well.

Control vs. Data
  Distributed Erlang is good for control plane, not so good for data plane
  Sending large data can block
  Use TCP, UDP, etc. directly for data plane traffic
  Don’t mix control plane and data plane traffic
    Riak still does this in a few places, unfortunately (they’re going to fix this)

Hinted Handoff
  Fallback vnode holds data for unavailable primary vnode
  Hands it off once primary becomes available

Read Repair
  Vnode with stale data is repaired via asynchronous update
  Eventual consistency
  Active anti-entropy (AAE) - can actively repair stale value before it’s read

Monitoring is great for cleaning up after aborted operations.

Pattern-matching is an elegant way to parse binary data.

gen_fsm - one of the OTP library behaviors (finite state machine)

In Erlang, everybody uses these behaviors; makes for more readable code.

Let It Crash - Joe Armstrong’s doctoral thesis (he created Erlang)

Business logic goes in Workers; Supervisors are very simple and just start and watch Workers. Little can go wrong with Supervisors.

Erlang/OTP System Facilities
  Get status of OTP process
  Trace function calls, messages
    Put a trace on a process - VERY powerful
    Seldom need a debugger
    Be careful - you can take a system down by improper tracing

Linking with C/C++
  NIF - Native Implemented Function
  Can ref count binaries
  Portable interface to OS
  Can crash the whole VM with a bad NIF
  NIF calls execute within a VM scheduler thread - can block the thread
  NIFs should only block for a millisecond or less
  Put long-running activities in their own threads.

Eunit - unit testing
QuickCheck - create model of SUT, and it runs randomly-generated tests against it (like PEX?) - shrinks the test case for easier debugging - awesome

Watch your memory.

Hot code loading really works.

Understand the Erlang VM.

A DSL for distributed systems.

A Little Riak Book, Riak Handbook

Elixir - Ruby-like language on Erlang VM - also some Lisp-style languages

O’Reilly Erlang book is also very good. (Francesco Cesarini, Simon Thompson)

(And he gave me a free book for asking a question! - Bill S.)

Evolving Java
Brian Goetz - Java Language Architect, Oracle

Compatibility - avoiding breaking existing programs - takes up a lot of his time.
Serialization - big regret.
What Would James Do?

Java SE 8 - modernizing Java - Summer 2013

• Lambda Expressions
• Interface Evolution
• Bulk Collection Operations

Alonzo Church - 1930’s Lambda Calculus

Language features are only enablers.
Better libraries are what gets things done.

In 1995, few mainstream languages supported closures.
Today, Java is just about the last holdout. (Even C++ added them recently.)

In 1995, everything was sequential. (Single processor.) For loops, iterators, etc.
In today’s world, that’s an impediment to parallelizing code.

Mutability was the default.
Today, that’s the wrong default.

External Iteration
There’s a lot of accidental complexity in a for loop that’s mutating items in a collection. We want to separate the “what” from the “how".

Internal Iteration
Don’t ask me, just do it.
The library can use parallelism, out-of-order, laziness.
(This is an argument for ForEach() as a method. - Bill S.)

Lambda Expressions

• Anonymous method
• Compiler can often infer parameter types from context
• Method references are supported
• Treat code as data

What is the type of a lambda?

• C# has delegates, etc.
• Doing it in the VM would have been disruptive.
• Historically used single-method interfaces to model functions.

Functional Interfaces

• The type of a lambda expression is conditional on what we assign it to.
  Predicate<String> isEmpty = s -> s.isEmpty();
  (Don’t have to say that s is a String.)
  Runnable r = () -> { System.Out.printLn("Boo!"); };
• The obvious answer (just add function types) was wrong - would have divided the libraries into old & new libraries (like generics in C# - Bill S.)

Problem - Interface Evolution
If we had lambdas when we designed Collection libraries, our libraries would look nothing like they do.
New feature: default methods (similar to abstract)
  You can add a virtual interface method with a default implementation.
  If an existing implementation doesn’t have the new method, it still works (using the default implementation).
  Multiple inheritance of behavior, not state.
  This is not full-blown traits (like Scala).
  Not designed for monkey-patching.

It’s All About the Libraries
Best way to evolve the platform - cheaper, more scalable, etc.

Lambdas Enable Better APIs
Boundary between client and library becomes more permeable
The library owns the “how", the client owns the “what”
Key effect on APIs is: more composability
  You want to build behavior by composition.

Sorting today: The code reads nothing like the problem statement.
You want the code to look as much as possible like the requirements.

Bulk operations on Collections
Exposing aggregate operations on collections.
(Very LINQ-y.)
You have to ask for parallelism, but you don’t have to ask very loudly.

Guy Steele’s talk on How to Think about Parallel Programming: Not!
(Highly recommended by presenter.)

Streams
New abstraction
Stream of values - not a data structure
Pipelined
Lazy
(like IEnumerable in C# - Bill S.)

Parallelism
Let the experts write the libraries, and let the libraries handle the hard problems.
JDK7 added Fork/Join - lots of boilerplate code
parallelStream() lets you do map-reduce easily.

Aggregation
sum() is a special case of what functional languages tend to call fold
Reductions work really nicely in parallel.
Composable.

Gentle push towards a more functional style of programming.
People will have to unlearn some things. Want pure methods, immutability, etc.

Tuples coming!?

Frontend Architectures: From the Prehistoric to the Post-Modern
Pamela Fox (with Coursera - online training)

Bello walnut green tea - recommended.

Multiple architectures can make training new engineers difficult.

Picking an architecture is hard. Users and developers have different needs.
Users: Usability, searchability, etc.
Developers: Productivity, testability, etc.
Both: Performance

Prototypes become real products. Even then, make good decisions.

3 Architectures

1. Server-side HTML (classic, templates, form submission, PHP, etc.)
Usability generally poor
  Wait for page reload
  Easy to lose your place
Search- and share-ability generally good (but doesn’t matter if page is private)
Linkability good
Performance has pros & cons - predictable and optimizable, but can take a long time to load everything
Productivity pretty poor - developers can get away with mixing markup and code
  Not API-driven, not modular, encourages copy & paste
  Bringing in JavaScript can lead to spaghetti
  Tied to your server-side language (PHP, etc.)
Testability is so-so - predictable, but may have to test data & presentation layers together; often end up with lots of untested code

2. JavaScript-generated HTML (server outputs minimal HTML with script tag)
(Coursera uses Require.js, backbone.js, jade on frontend)
(They like jade for templates - auto-closes tags, enforces HTML prettiness, designers can use directly without knowing JavaScript)
(Backend doesn’t matter much - can be swapped out)
“I scoped it out to two weeks; it was two months…”
Usability is both good and bad.
  Can do quite a few actions (even simultaneously) in a short period of time without waiting for page reloads.
  Have to use loading indicators well to avoid confusing users.
  (May want to enforce indicators in Ajax.)
  Pages may be partially broken (parts work), which can confuse users.
Search- and share-ability is poor. You have to have an alternative view for bots.
  Coursera uses a just-in-time renderer uses Selenium to render page to the cloud and serve it to the bot - but only to bots. This can be cached.
Performance is generally good.
  Content can be loaded incrementally (even based on what’s onscreen).
  Fast pagination.
  Infinite scrolling (which does have usability features - Ctrl+F, Jump to Bottom)
  However, JavaScript can be slow to process.
Dev productivity is typically very high.
  Separates presentation from data.
  Reusable (even in mobile apps).
  Can expose an external API. (API-centric architecture.)
  Does require learning JavaScript and multiple frameworks. (There’s more than one way to do it.)
Testability is so-so.
  Can test presentation separately from API.
  (Coursera uses Mocha and jsdom, and Chai? for mocking)
  Many more cases to test; may have to wait for bug reports.

3. Single-page webapps (JavaScript is also controlling the routes)
(Coursera uses Backbone History API and hash trick)
Similar to #2.
Usability is similar; navigation is fast. REALLY need loading indicators.
Linkability - internal anchors are hard to use because of double hash.
Performance is similar; only need new resources.
  However may need to come up with multiple bundles if one is too big.
Dev productivity is similar; there’s no more window load/unload, though (bad).
  Have to come up with a way to keep users from losing their data.
  Does the current view have an unsaved model? (Mark as dirty.)
    (Coursera’s technique.)
  There aren’t as many people doing #3, so there’s less info on it.
Testability is similar, but it gets even harder. Now there’s state across your web views. You have to test state across routes. Combinatorial.

None of the approaches are perfect.
The presenter prefers #2 - but there’s no one right answer.
It Depends on what’s important to you.
#2 and #3 should get easier and easier.
Pick your battles.

Coursera serves many things off Cloudfront.

Escalation and Response to Outage Scenarios
John Allspaw (with Etsy)

Emergency scenarios under time pressure.

Managing the Unexpected - book

Skills, rules, & knowledge
1. Skill-based - Simple, routine; muscle memory
2. Rule-based - knowable, but unfamiliar
3. Knowledge-based - novel scenarios; trial & error

OODA Loop

Response to escalating scenarios
We assess how things are in the moment, but neglect rates (developments over time)
We have difficulty with exponential developments (accelerated change)
We tend to think in causal series, instead of causal nets. The world doesn’t work like a line-up series of dominos.

Pitfalls
Thematic Vagabonding - you’re trying to work out what’s going on, signals are coming in, but you’re flailing - context-switching
Goal Fixation - Intuition tells you this is something you’ve seen before; you’ll prove that to everyone
Heroism - non-communicating lone wolf-isms - “Hold on, I’ve got this” - not a team player - win or lose, the team loses

Emergencies are increasingly handled by teams, not individuals.

Successful teams
- know when to scatter, and when to swarm - but communicate
- stabilize the patient; we’ll figure it out later (incident resolution)
- reproducibility?
- fault tolerance - Erlang - does add complexity

Shotgun debugging - almost never works

Joint Activity
- Interpredictability - ability to predict the abilities of others involved in the activity (and sharing your own abilities/actions) - make it clear who’s doing what
- Common Ground
- Directability - deliberate attempts to modify the actions of others - also being open to direction

Improvisation
Typically have to improvise within constraints.
Needs to be practiced.

Communication
- Explicitness - be concrete, not abstract - who/what/when, Alpha/Bravo/Charlie
- Timing
- Assertiveness - passive-assertive-aggressive continuum - latter disregards another person’s perspective/signals - we want to be assertive (in the middle)
Yes-And (from Improv) - augment the other person’s comments

Airline pilots and co-pilots are co-located; they can see body language, etc.

Two-way communication is great, but takes longer (which is why developers hate meetings).

Feedback

Decision Making
Sources of Power (book) - Gary Klein - decision-making in the wild
1. What is the problem? (Sense-making) (focus on key aspects)
“Information is not scarce; attention is.” (missed the quote’s author)
2. What should I do?
Intuition
Rule-based (not good for experts)
Choice decisions (often takes more time)
Creative decisions (time-consuming, untested)

PRE-Mortems (Klein)
Before launch:
1. What could go wrong? (scenarios, contingencies)
2. It has all gone wrong. Now tell me why. (thought experiment)

Post-mortems are also extremely important.
Firing someone is almost certainly the worst thing you can do.
Human Error
- Attribution AFTER the fact
- A symptom, not a cause (you’ve stopped looking)
- “Root” cause? (myth - it will happen again)
- Useless labeling
- Using the term is the largest indicator that learning is not your goal

Mature Role of Automation
In the wake of an accident, we will reach for more automation.
- Moves humans from manual operator to supervisor (changes work; doesn’t remove it)
- Extends & augments human abilities; doesn’t replace them
- Doesn’t remove “human error”
- Are brittle (can’t adapt)
- Recognize that there is always discretionary space for humans
- Recognize the law of stretched systems

Law of Stretched Systems
Anti-lock brakes - people drive faster!

Procedures & checklists are written so that they make sense to the author at the time they were written.
When Canadian geese fill your engine, do you look for the procedure?