Development Central

Bill Sorensen's software development blog

Notes: Case Study: Algorithmic Trading in a Production Haskell Environment

Notes on a talk by John Lato of Tsuru Capital at Iowa Code Camp.

Coursera - financial modeling, etc. courses are available

Derivatives market is huge. (A quadrillion dollars!)

Create a model for the prices structure and derive a formula from the model.
Example: Black-Scholes Option Model - statistical volatility is the main variable, along with underlying price and time to maturity
Delta - underlying, Vega - volatility, Theta - time (The Greeks) (actually equations)
Wikipedia & Investipedia have articles on this.
Modeling is very math-y.

Algo Trading: Reality

  • Buggy 3rd party code
  • Out of spec implementations
  • Parsing

(Sound familiar?)
Can measure success: Are we making money?

1-2 GB/day for a small subset of the orderbook from the MarketFeed.

Knight Capital SEC article - how software deployment issues can cost your business money.

How Tsuru uses Haskell

  • Type system
  • Concurrency model
  • Foreign interface
  • Meta-programming


The Haskell Kool-Aid: If it compiles, it works! (Mostly true.)

Newtypes are "free," apply liberally. (Similar to units of measure in F#.)
  This way you don't apply the wrong function to a price, etc.
  Way to preserve invariants (can include constraints).

Coming to a strongly-typed functional language from a dynamic language is hard.
You have data, and you have functions. Model your data accurately.
If you have a Maybe Int, you can only use it with things that work with Maybe.
Rather than throwing exceptions, you create types that preserve invariants.

Parametric Polymorphism
Functions that work with, for instance, a list of any type
Can't alter your data, as they have no interface to interact with it
Similar to generics in C#/Java
Example: many :: Parser a -> Parser [a]
  (a is generic)
You can do a lot of this stuff in C# now, but it gets very verbose.
Type inference may infer a type more general than you thought, in which case you get code for free! (In functional languages.)
(In Haskell, it's considered good practice to write type signatures, at least for top-level functions.)

Concurrency
Immutable-by-default and lack of side effects are a big win
Some challenges remain (e.g. no mechanism for thread priority)
In F#, considered good practice to make a method pure unless good reason
Side effects are "Spooky action at a distance"
In Haskell, if something doesn't have IO (monad) in the signature, it doesn't do I/O.
Similar functionality to async/await. In functional languages, typically the syntax is very clean.

hackage.haskell.org - lots of code examples
ghc - compiler most people use - offers REPL, includes mem/perf profiler
Cabal - build system
learnyouahaskell.com
fpcomplete.com
ideone.com

github.com/johnLato

In Haskell, deep debugging can be hard. Traditional stack traces are tricky.

Testing
Have both unit tests and functional tests. (HUnit)
QuickCheck - Haskell (and more?) testing tool - generates random input
Checks results - good for catching corner cases
Also use a simulator in place of exchange, playing back feeds, etc.

Notes: Learn Every Programming Language

Notes on a talk by Patrick Delancy at Iowa Code Camp.

If you don't practice it, you lose it.

Why

The inability to learn programming languages could force you to change careers.
The idioms and concepts you learn will help you with your current language.

How

Start with the underlying principles. Understand the paradigms. The rest is just syntax.

Programming Paradigms (in order of complexity, low to high)

  1. Procedural
  2. Logical
  3. Functional
  4. Object Oriented

That's not the order in which we typically learn them.

Thinking Procedurally

  • Loops
  • Procedures/Functions/Subroutines
  • Global/Static Variables
  • Jump/Goto
  • Lexical Scoping
    • Text in your code defines variable scope
    • Not based on time (such as global after you define it)
    • Block scoping, function scoping, etc.

Thinking Objectively

  • Class/Entity
  • Abstraction
    • Making something simpler.
    • We don't have to understand how it works.
  • Encapsulation
    • An object, if it exists, should never be in an invalid state.
  • Inheritance
  • Polymorphism

Very domain-driven. Easy to translate the business needs.

Thinking Functionally

  • Functions as values
    • Lambdas, anonymous delegates
  • Pattern Matching
    • Regular expressions
    • Define shape of a value or object
    • Compiler can warn you of missing cases
  • Composition
    • Chaining function calls
  • Partial Application
  • Monad/Computation Expression
    • Binding functions together and defining how they behave
    • Helps to solve the problem of side effects (like I/O)
  • Closures
    • A function plus its referencing environment
    • Such as the memory address of a variable
    • Necessary evil (avoids globals)
  • Deconstruction
    • Pull out variables from object (often through pattern matching)
    • Lists are head + tail
  • Recursion
    • Tail recursion avoids running out of stack
  • Cons/Car/Cdr
    • Cons constructs a new value from two inputs
    • Linked lists
  • Etc.

What is the shape of the process?
Focus on behavior, not data.

Thinking Logically

  • Facts
  • Relationships
  • Goal-reduction

Classification
Languages don't fit into these buckets anymore.
They steal ideas from other paradigms.

Notes: Agile's Dirty Secret

Notes on a talk by Tim Gifford at Iowa Code Camp.

CelebrityAgilist.com - GitHub community

Agile Manifesto - no processes are listed.
See also the related 12 principles.

SAFe Agile process (new RUP?) - speaker has seen companies turn the PSI's from that into little waterfalls - easy to lose track of what the customer values

Incremental - you don't see the whole picture until everything is done
Iterative - see the picture, but it starts out fuzzy
(Like interleaved graphics downloading)

User Story Maps - read more on this!

Weinberg's Law of Raspberry Jam - the wider you spread it, the thinner it gets (poor understanding)

Testing Boundaries problem
3 code paths with 10 paths each - need 1,000 end-to-end tests
If we get a defect in one code path, 100 tests fail.
Spend all your time fixing broken tests.
10/10/10 unit tests + 200 integration tests + 1 end-to-end = 230 tests
Only 22 or so fail if there's a defect.
(Same combinatorial problem Mark Seemann talks about on Pluralsight.)

Large problems need leadership & courage.

Large Teams
More people => fewer questions => less learning
Interpersonal issues - why not split them up on clique boundaries?
Responsibility is spread out.

Small Feature Teams
Top-to-bottom, delivering customer value.
More opportunities for leadership roles.

He's using Feature Toggles.
He's very much in favor of it, but there's a downside.
You need to make sure you're truly releasing the software, not just delivering with features turned off.

It's important to still have a release plan.

Give pilot teams learning time - an extra point per story, etc.
You don't start out with TDD/BDD/ATDD at full velocity.

You cannot teach craftsmanship. Knowledge does not change behavior.
(There are marriage counselors who are divorced.)

"It's always a people problem" - Weinberg's 2nd Law of Consulting
(Secrets of Consulting is Weinberg's book)

Drowning in Defects
Agile practices can help with new defects - but what about legacy?
Defect age ranged from 2 hours to 2 years on one of his projects.
The variability resulted in angry customers.
What if we do defects in the order in which they were received? FIFO?
What about the old ones? Ignore them - if they're important, they'll be reported again.
Prioritization can be waste.
Capacity needs to align with demand - the business needs to step in if the bug requests are overwhelming the developers (more hires or stop reporting bugs until developers catch up).
Leadership doesn't come from authority. If you see a problem, fix it.
Why do we let defects accumulate?

In order to run fast, you need to run clean.

Don't start coding until you have a Given-When-Then scenario.

Notes: Becoming an Outlier

Notes on a talk by Cory House (@housecor) at Iowa Code Camp.

Career Reboot for the Developer Mind

What would you do if money were not a concern?
Why aren't you doing that?
If you choose money over happiness, you're working to make money so that you can live and work doing something you dislike.

Job, career, or calling?

Get up every day excited about what you're going to do.
Work "with" a company rather than "for" them.
Do it because you're having fun!

How to be an outlier;
Spend your time consistently doing things that people care about.

We're all weird. Some people will always look at you with pity.

Step 1: Be Modal
40% of your day is habitual.
Takes 10,000 hours for expertise (of deliberate practice)
You under-use your free time.

Multithread your life.
Hack the commute. (Podcasts, audiobooks, etc.)
Work from home.
Take public transit.
Read or watch tech videos while doing cardio.
Listen to podcasts while doing yard work.

Rethink giving back.
Contribute to open source, pro bono development, Give Camp, etc.
(Rather than building a house, donate your most valuable skill.)

The Maker's Schedule
Managers live with regular meetings, interruptions, etc.
Developers need uninterrupted thought to get into Flow.

Alter your schedule - go in early, etc.
Don't even open your email for an hour.
Larger, less frequent meetings

Improve your Signal to Noise
The cheaper you can live, the greater your options.

Automate relentlessly.
Direct deposits, auto bill pays, reminders, etc.
RescueTime - software tool

Target media consumption.
Recognize what you're giving up by watching reality TV.
InfoQ, TekPub, Pluralsight, TED talks, DNRTV
Manicure your stream. Slow media is for the focused few.
The less news you consume, the bigger the advantage you have.
Why waste your time on something you cannot control?
Focus on what you can influence.

The more you make per hour, the more time you can buy.
Use delegates. Hire someone to do the yardwork, etc.

Step 2: Manage Your Image
Pick Two - Neil Gaiman graduation speech on YouTube

Purpose built - how do you want people to look at you?
Self-image is your greatest constraint.
Can I see your body of work?
Do it in public - StackOverflow, GitHub, Twitter, blog, etc.
Your words are wasted if you're not blogging.
Measure: GoogleAlerts, TweetBeep, etc.
Shelved credentials - books attract alpha geeks and convey passion.

Step 3: Own Your Trajectory
(Of your career.)
What if every piece of your workday moved you toward independence?
(Example: Pluralsight)

Search for Scale

  1. Work
  2. Lead (talk about work)
  3. Own (products, frameworks, author)


Time = potential knowledge
Beware of becoming an assembly line coder.

Opportunity wedge
Life is characterized by closing horizons and lost opportunities.
You can extend this by enhancing your skills.

Don't work for money.  Work to learn.

Key to success: Learn to teach yourself more efficiently than any institution can.

Luck Surface Area - do what you're passionate about and communicate that to as many people as possible
Speaking, blogging, conferences, user groups, Tweet, never eat alone
Make the Hang - spend time with people you look up to
You are the average of the five people you spend the most time with. - Jim Rohn

Lean into Uncertainty
Find what scares you and run at it.
Speak at a conference.

You Don't Need Permission.

If it's work, we try to do less.
If it's art, we try to do more.

Notes: Pragmatic Architecture in .NET

Notes on a talk by Cory House (@housecor) at Iowa Code Camp.

Clean Code video on Pluralsight.

Religion is the problem in architecture. Always/never.
Look at the context.
It depends.
People who are experts answer questions with more questions.

Over-architecting wastes money.

Timelines

What if one month late is worthless?

Parkinson's Law: Work expands to fill available time.

Consider Complexity (speaker considers the following complex)

  • Automated testing
  • Coding to an interface
  • SOA
  • Rich Domain Model (DDD)
  • ORM vs. custom DAL
  • Repository Pattern
  • Layered architecture


Consider Simplicity

  • Do the simplest thing that could possibly work
  • Lean/Agile principles (minimize WIP, etc.)
  • YAGNI
  • 80/20 Rule


MVP
Book: Lean Startup - minimal viable product (MVP)
Scalability, maintenance costs, performance, etc. may not matter

We're paid for solutions, not code.

Flexing features (scope) is a lot better way to go than flexing quality.

Is it worth taking on technical debt today?
Hard vs. Soft deadlines

  • Trade show?
  • First to market?

Single loud customer, salesman misspoke, wild-ass guess, MS Project said so - terrible reasons for hard deadline

Technical
Layers - logical (separation of concerns)
Tiers - physical (often decided for you)

For a small enough application, methods are layers.

Everything's a trade-off.

Architectural Levels

  1. Simplest thing
  2. Somewhere in between
  3. Every tool in the shed


Active Record pattern mixes Domain and Data. Breaks SRP.
Easy to understand, though. Consider for CRUD apps or simple domain.
Rigid: Domain model = DB
Leads to God object
Hard to test
Converting to Repository Pattern = pain

Eric Evans - DDD book
Great for complicated, long-lived applications
Takes time

Level 2 (somewhere in between)
Focus on the pain

No Free Lunch

At what point does our application complexity grow to the point where the effort to enhance gets too painful?

MVP, junior team, simple domain, tight timeline, throwaway - consider Level 1

Flagship product, senior team, complex domain, long-term, security matters, flexible timeline - consider Level 3

POEAA (Fowler) and Dino Esposito's architecture book inspired this talk.
Microsoft .NET: Architecting Applications for the Enterprise is the latter.

Speaker recommends clean, readable code regardless.

Kickstarter for chair

I'm backing The FreedMan Chair Kickstarter. Despite my Herman Miller Mirra chair, I still have the occasional day of back pain. While my faith in osteopaths is right up there with my faith in shamans, the design is intriguing. It's not cheap (about $515 US), but retail is supposed to be about $1,300 US. And good chairs aren't cheap.

More info: http://www.gizmag.com/freedman-orthopedic-chair/29287/

That's not TDD

I'm currently reading Pro ASP.NET MVC 4 as part of learning web development. So far it's been a good tutorial, but I take issue with the author's coverage of Test-Driven Development (TDD).

I applaud the coverage of unit testing, dependency injection, mocking, separation of concerns, and other best practices. It's the details that bother me.

In chapter 6, the author says he will follow the TDD approach in a unit test example. He starts out by writing the system under test and having it throw a NotImplementedException. Bzzt. Write the test first. There is a reason for this: the tests guide your design. You write the API you want to consume.

Next, he writes the unit tests. All of them. In TDD, you write one test, see it fail, write code to get it to pass, refactor, and then repeat. While there's no law that unit tests must be written in that fashion, changing the process means you're not doing TDD. If you don't want to do TDD, fine, but don't call it that.

Several of the tests violate the guideline of one logical assert per test. The only justification I can see is brevity. This comes at a cost: when the first assert fails, subsequent asserts are not run. In addition, messages have to be added to the asserts for clarity; it's much better to include this information in the test method name.

// These should be three separate test methods
Assert.AreEqual(5, TenDollarDiscount, "$10 discount is wrong");
Assert.AreEqual(95, HundredDollarDiscount, "$100 discount is wrong");
Assert.AreEqual(45, FiftyDollarDiscount, "$50 discount is wrong");

The author encourages use of the ExpectedExceptionAttribute. This is far too coarse-grained, and is one reason for avoiding MSTest (which the book also promotes).

I encourage everyone to go to the experts to learn TDD and unit testing. Two good sources are Test Driven Development: By Example and The Art of Unit Testing.

The book's coverage of dependency injection is incomplete as well; the authoritative work is Dependency Injection in .NET.

I'm being harsh for a book on ASP.NET MVC. If you're going to cover these topics in detail, though, please get the details right.

Disfunctional

A coworker and I spent a total of at least 8 man-hours this week tracing down one threading bug. The original author was locking access to shared state, and got it almost right.

Writing thread-safe code that uses global mutable state is hard. It's much easier to avoid the problem entirely.

Functional programming emphasizes pure functions (the same inputs will always give the same output, with no side effects) and immutable types. If you write your code in that style, thread safety comes naturally.

Regardless of programming language, consider a more functional approach. Your future self will thank me.

For a good book on the subject, try Real-World Functional Programming: With Examples in F# and C#.

In the NOOO

I'm a signatory to the Manifesto for Not Only Object-Oriented Development.

# 11010100. :)

Speaking at CRineta

I'm giving a presentation on FSM.NET at the local users' group on Monday night. @dahlbyk is opening for me, so I've got a tough act to follow.

Meeting info