Thursday, October 29, 2009

OOPSLA 2009 Thursday, October 29th

Moving Fast at Scale, Lessons Learned at Facebook, Robert Johnson

Facebook has over 1 million active users per engineer.

Slowing down to get it right is not a good idea, unless you know exactly that your idea is right. If you try things fast, you can try out more things and you can get feedback fast.

How do you get fast?

  • Never block developers
  • Give developers control. At Facebook developers code, test and deploy. No QA is involved.
  • Tune processes for speed
  • No deadlines, but the site cannot go down.
  • Frequent small changes, no delays, easier to isolate bugs.
  • All development in trunk with weekly releases.
  • Deployment tools
  • Gatekeeper
  • Do less work
  • Build tools
  • phpsh -interactive development
  • diffcamp - code review
  • XHProf - profiler
  • Scribe - moves data from server to central repository.
  • Hive - data warehouse infrastructure on top of hadoop
  • Work with open source

How to deal with a lot of users?

  • Scale Horizontally
  • Web Server, the relations have moved to the web layer from the database.
  • Memcache
  • Database, only used as a persistence layer. No joins etc.
  • Thrift
  • C++, Java, Erlang, Python

Open Problems/Future Work Languages for real-time data-access parallel with lots of dependencies. Distributed indexing Automatic clustering of data Profiling of parallel data * Better ways of expressing client and server code + Javascript and PHP are not well matched.

Toward Cloud-Agnostic Middlewares, E. Michael Maximilien

The Cloud Computing Landscape

  • Service Providers
  • Platform Providers
  • Service Brokers
  • Application Providers (SaaS)
  • Users

Challenges in Cloud Computing

  • Data lock-in
  • Application programming lock-in
  • Management lock-in
  • API lock-in
  • IncreaseRisk
  • Security and Privacy
  • Catastrophic failures
  • Business models and impacts

Research Opportunities

  • Cloud middleware
  • Agnostic wit respect to providers, frameworks and interfaces.
  • Learn Best Practices
  • Fluid cloud application deployment
  • Optimize could usage prices
  • Give indication of cloud readiness

Use Cases

  • COBRA Java text analytics solution. (JavaEE)
  • CoScripter application (RoR)
  • JumpStart sMash application (WebSphere, Smash)

The Architecture

  • Core APIs
  • Cloud APIs
  • Cloud Adapters
  • Clouds and Brokers

IBM Altocumulus

Altocumulus is a configuration tool that helps you to deploy your application into different clouds. It supports multiple different service and platform providers. It helps you configure your deployment so that it follows the commonly known "good" practices. You can use it via the Web or via a RESTful API. It also supports Atom and RSS feeds. It is very interesting.

Lessons Learned

  • Cloud Providers need to include some support service s to be enterprise ready
  • Cloud Providers need to expose flexible image creation facilities
  • Cloud virtual machines are mostly hardware based and their memory management is suboptimal
  • Best Practice Pattens works but need to evolve.
  • Standardized hardware is important when implementing cloud infrastructures
  • Instance monitoring capabilities should coma as part of the cloud facilities and APIs
  • The cloud space is still maturing

When Users Become Collaborators: Towards Continuous and Context-Aware User Input, Walid Maalej et al.

User input is critical for the success of projects.

Various types of input:

  • Field observation, Lead Users
  • Perpetual Beta, Legacy Documents, Usage Data
  • Issue and Bug Report, Enhancement Request, Feature Request
  • Workshop, Interview, Survey, Classification Requests

Built in Feedback Mechanisms

Examples: Application Bug Reports, Application Usage Data, Discussion groups

Most applications don't have feedback mechanisms built into them.

Users are motivated by feedback, so if they submit a report and don't get any feedback, they will probably not do it again.

Wednesday, October 28, 2009

OOPSLA 2009 Wednesday, October 28th

Jeannette Wing, CMU, Frontiers in Research and Education in Computing

Jeanette said that there has been a paradigm shift.

Not just about computing's metal tools (transistors and wires), but also our mental tools (abstraction and methods)

Is this really a new paradigm shift? Maybe for the National Science Foundation (NSF), certainly not for anyone that has been working with software the last ten years.

The limits of Moore's Law forces the NSF to focus on programming languages and abstractions.

Three drivers of computing research, Society, Science, Technology.

Encouraged research areas:

  • Data Intensive Computing
  • Cloud Computing
  • Map-Reduce

  • Cyber-Physical Systems (computational core that interacts with the physical world)

  • Smart vehicles
  • Smart Flyers
  • Smart Devices

  • Network Science and Engineering

  • Understand the complexity of large scale networks
  • Trustworthy Computing

  • Socially Intelligent Computing

  • Humans are still much better at image recognition
  • Programs where the human is a port of the program.

  • IT and Sustainability (Energy, Environment, Climate)

  • Computer Science and Economics

  • AdSense
  • eBay

  • Computer Science and Biology

Agile development: Overcoming a short-term focus in implementing best practices, Karthik Dinakar

In the project they concluded on the following good practices.

  • Good version control
  • Coding guidelines
  • Build automation
  • Unit testing framework
  • Automatic sanity tests
  • Accurate task estimations
  • Effective pre-spring planning
  • Solid design discussions
  • Involve QA and Operations
  • Effective post-sprint-reviews

They also thought that it is difficult to implement best practices under way.

They had many problems:

  • Integration was not in the plan
  • Sprint backlog changing all the time.
  • Long sprint meetings

Management did not allow them to implement the changes that they needed and the reason was "It's Agile!"

All in all, their problems seemed to be the usual, they had no idea of what it meant to do Scrum in the first place. Don't people read books anymore?

Are systems Green? Panel with Steve Easterbrook et al

  • A Google query has a Carbon footprint.
  • We don't know what it is.
  • It may actually be less than the energy it uses since it may permit the person posing the query to save a lot of energy.
  • The term Green is just marketing bullshit.
  • It is not measurable at all.
  • Carbon Emissions are permanent.
  • It won't go away even if we stop burning any carbon today.
  • The goal of emissions are ZERO, anything else is not sustainable.

There is a book called Green IT for Dummies

  • What can we as a computing industry do.
  • Analyze the problem.
  • Make a list of what we need and can do.
  • Create a wiki.

If you are going to read an article for more than three minutes, YOU SHOULD PRINT IT OUT! Believe it or not, but check your facts.

Architecture in an Agile World, Panel with Steven Fraser et al

Randy Miller: We allow the customers to make changes, but we don't tell them what the cost will be. If we have no architecture the cost will be high. The people behind the agile manifesto were all good at architecture and that is why the importance of architecture was not emphasized in the manifest.

Bill Opdyke: Some people seem to see a difference between architects and agilists. But most people that are good don't have the this problem. Architects can learn from agilists that change is not that hard. Agilists can learn from architects that architecture matters. There is a middle ground.

Ethan Hadar: We need a stable vision of what the architecture should be while delivering solutions iteratively. Show me the architecture road map of your product, because it will be integrated with another program in nine months. Accountability and responsibility for the architecture is important.

Dennis Mancl: You need to plan architecture early, or you will have to do it later on and then it will be harder.

Audience: How does an architect work in an agile team?

Randy Miller: From an agile team, you're all part of the team. There are no distinctions. The architect is in the team to make sure that

The architecture in the system is the point in time in which you have to step back and think about how everything interacts.

Ethan Hadar: The architect is the person, who needs to interact with the testers and operations to explain how and why the system is the way it is.

Dennis Mancl: The architect is responsible to the stake holders and he is responsible to know the problem domain.

Irit Hadar: The architect's role is to take a step back and say when it is time to review the architecture.

Audience: The architecture is what you get, regardless of what you do.

An interesting discussion, that could not find a consensus to if there is a need for a single architect or not.

Agile Anthropology and Alexander's Architecture, Jenny Quillien, Dave West, Pam Rostal

Do we need to pay any interest of to Christopher Alexanders' new book The Nature of Order?

The software community has always taken interest in Alexanders' books, but we take interest in the wrong things. We have only understood some rules, we have not grasped the deeper part of it, because we don't understand the culture of architecture.

In a pattern language, we looked at patterns, but we dismissed the QWAN. We missed the holistic point-of-view. Everything is part of the system, people, organizations. He is writing about things that are multi-dimensional and multi-faceted, not things that are simple and exact.

The Nature of Order contains the same multi-faceted ideas and if we look at it with the same eyes, we will miss the point again.

Alexander looked at centers and centers affect other centers. And there is no right or wrong, there are only degrees.

In Alexanders' world there is only one system, the Universe. Everything is connected!

Writing Code for Other People, Tom Mullen

Chunking and Memory

  • The mind groups memory into chunks. Most chunks are stored in long-term memory. Out conscious is in long-term memory.

Short-time memory can only hold about seven relations. Short-time memory is also short:) This gives us a time-limit when traversing code.

Meyers' open-closed principle is an echo of the mind's way to learn things.

If the code is a reflection of our brains, then most brains contain spaghetti.

Analogies

Analogies is the mapping from one thing to another.

Conclusion

Our brains are not good a processing more than 4 chunks at the time. This implies that we should write methods with less than four lines. Classes with less than four methods, modules with less than four classes, and applications with less than four modules.

Tuesday, October 27, 2009

OOPSLA 2009 Tuesday, October 27th

Barbara Liskov, the Power of Abstraction

OOPSLA 2009 opened with Barbara Liskov as the keynote speaker. She is famous for, among others, the Liskov Substitution Principle. This principle states that:

A subtype should be substitutable for its super-type.

That is, it should be possible to use a subtype in the same way as if the type itself was used. Any difference in behavior should NOT be noticeable to the client.

A History of Abstract Data Types

Data abstraction was developed as a solution to the software crisis. A crises that is just as present today as it was then. This was in 1968, the time when Dijkstra wrote the paper Go To Statement Considered Harmful. The problem with gotos is that it is difficult to know the context in which a statement is used.

Other important papers at the time were. Nicholaus Wirth's paper Program Development by Stepwise Refinement on 1971, about top-down design and David Parnas' paper Information Distribution Aspect of Design Methodology in which he stated.

The connections between the modules are the assumptions which the modules make about each other.

These assumptions are often much more than the simple interfaces that we see today. It includes the whole context in which the module is used.

Barbara then wrote a paper called A Design Methodology for Reliable Systems. The ideas from this paper was later reused in the context of programming. When seen from the outside it is apparent that the same technique she used when creating the Venus Operating System, Partition State, could be used when building programs but it was not at the time. Where do ideas come from? Perhaps the time is just right.

Other influential papers that are still valid today are: Hierarchical Program Structures by Dahl and Hoare and Protection in Programming Languages by Morris, introduced the early ideas of encapsulation and Global Variable Considered Harmful by Shaw and Wulf.

In 1973, the paper on Abstract Data Types was published and Liskov the realized her ideas in CLU. Its worth noting that CLU was way ahead of its time. It included features, like data encapsulation, exceptions, iterators via yield but no inheritance. She doesn't think that inheritance is very important and that it complicates things.

The Liskov Substitution Principle didn't appear until 1983, when she held a speech here at OOSLA, when she had noticed that inheritance was used for two different things that was not very well understood. It is used for:

  • Implementation inheritance, which violates encapsulation.
  • Type Hierarchy, and this was not very well understood.

She ended up with noting that modularity based on abstraction is the way things are done now. It wasn't at the time.

She also pointed some challenges that still exist:

  • New abstraction mechanisms
  • Massive Parallel Computers
  • MapReduce?
  • Transactional Memory?
  • Internet Computer
  • Storage and computation
  • Semantics, reliability, availability, security

And she also made the point that

Readable programs are much more important than writable programs.

When the questions were opened, it was interesting to note that among the questioners were Phil Wadler (Haskell), Andrew Black (Traits), Guy Steele (Scheme, Fortress), Dave Ungar (Self) and Ralph Johnsson (GoF).

Flapjax, a Programming Language for Ajax Applications

After Liskov's keynote I watched a presentation of a research paper about Flapjax, a language designed for web applications. It is based on event streams and the language itself is reactive. Flapjax is a Javascript-based language that can be used as a library.

The language introduces two new concepts Behaviors and Event Streams.

A behavior is a value that changes over time. It can be created like this.

// A variable that changes over time, every 100ms.
var nowB = timerB(100);

What is interesting is that the behavior is composable with normal Javascript functions. This is done by compiling or transforming the Javascript into Flapjax code.

If an expression is a behavior, all expressions whose values depend on it also become behaviors.

var nowB = timerB(1000); 
var startTm = nowB.valueNow(); 
var clickTmsB = $E("reset", "click").snapshotE(nowB).startsWith(startTm); 
var elapsedB = nowB - clickTmsB;
insertValueB(elapsedB, "curTime", "innerHTML");

Programming with event streams is a little different from programming with behaviors. A behavior masquerades as an ordinary JavaScript object whose content just happens to change automatically. In contrast, an event stream is a new kind of value, with new primitives for programming over it.

Event streams and behaviors offer complementary views of the world. It is easy, however, to overstate their differences. Given an initial value, every event stream can be converted into a behavior: the behavior always has the value of the last event to have arrived on the stream, starting with the specified initial value until the first event arrives. Likewise, every behavior can be converted into an event stream: when the behavior's value changes, send the new value as an event.

Thomas W. Malone, Keynote Onward, the Future of Collective Intelligence

MIT Center for Collective Intelligence

Collective intelligence - Groups of individuals doing things collectively that seem intelligent.

Collective Stupidity is also very much existent.

New examples of collectively intelligence are: Google, the Web, Wikipedia, Linux, Digg, YouTube, etc.

How can people and computers be connected so that collectively they act more intelligently than any person, or computer?

Thomas showed a video of a crowd of people flying an airplane by turning a reflective shield green or red.

What are the genomes of collective intelligence?

Every activity has to answer four questions, Who? What? How? Why?

  • There are two Who?-genes, crowd and hierarchical.
  • There are three Why?-genes, money, glory and love.
  • There are two What?-genes, create and decide.
  • There are four How?-genes, collection (contest), collaboration, group decision (voting, consensus, averaging, prediction markets) and individual decision (market, social network).

Failure to get motivational factors (thw why?) right is probably the single greatest cause of failure in collective intelligence experiments.

Interesting examples are: Climate Collaboratorium, TopCoder, Kasparov vs. the World, Amazon Mechanical Turk and TurKit

What's coming?

Human Brain is very much like Global Network.

  • We have global moods?

Quotes from We are the Web, Wired 2005

There is only one time in the history of each planet when its inhabitants first wire up its innumerable parts to make one large Machine.

Three thousand years from now, when keen minds review the past, this will be recognized as the largest, most complex, and most surprising event on the planet.

The Machine provided a new way of thinking (perfect search, total recall) and a new mind for an old species. It was the Beginning.

Brion Vibber, Wikipedia, Making your people run as smoothly as your site

As the number of people involved in a project grows, key decision-makers often become bottlenecks, and community structure needs to change or a project can become stalled despite the best intentions of all participants.

Instead of having a single admin looking at a page to decide if it is garbage, a group can vote if they think it is garbage. This allows the admin to delete pages without checking them first if everyone votes for deletion.

  • People have limited time and patience.
  • Waiting on other people is slow.
  • People want to do what interests them, not deal with process!

Get out of peoples' way and let them do stuff!

Onward!

The Commenting Practice of Open Source, Oliver Arafat

An analysis of 80GB of Open Source code. The average code density is one comment per five lines of code or 19%.

Average comment density is independent of code size.

Strong variation by programming languages.

  • Java code has an average of 26%.
  • Perl code has an average of 11%.

Successful open source projects follow consistent comment practices.

Comment Density by Commit Size * Smaller commits have higher comment density.

Polymorphic System Architecture, Jeffery E Bryson

Run-Time polymorphism (RTP) has been used in the software community for two decades to satisfy dynamic reconfiguration, plug-n-play, extensibility, and system redundancy requirements. RTP is also used to construct software systems of systems. System engineers now have the same requirements applied to large-scale system architecture.

A Polymorphic System Architecture (PSA) uses the same technology, by applying it to the system architecture. By defining specific polymorphic relationship within the system architecture the system architect can reduce the system complexity and satisfy functional requirement.

Polymorphism reduces the code size, but it also reduces understandability.

Value Added
  • Extendable/Reusable System Designs
  • Dynamic reconfiguration
  • An architecture that matures over time instead of becoming absolute.
  • OO and Refactoring.

Conclusion

All in all, this was a good day with the keynotes being the highlights. The Onward sessions were interesting, but most of them, were of very little use to me.

Some thoughts from "A Theory of Fun"

I just finished reading the book A Theory of Fun by Raph Koster. It is funny how everything comes together once you start focusing and noticing certain patterns. It is a good book and worth reading even if you're not into game design.

I already learned from personal experience and from other books, that our conscious mind is terrible at multitasking. It is however very good at internalizing things, learning things so that the brain can perform them unconsciously, without conscious supervision. Raph calls this chunking.

The act of learning is about turning many steps into chunks that we don't need to think about as separate entities.

When we don't see something, we don't perceive it, but once we become aware of a certain pattern, we see it everywhere. Koster calls this noise.

Noise is any pattern that I don't understand.

A good game keeps us on the edge of our abilities constantly and once we learn something the game will become harder. This is a variant of flow. But since a game cannot continue for ever it is doomed to become boring once we have mastered it.

The destiny of games is to become boring, fun is the process and routine is its destination.

Koster also mentions some of his grandfathers carpentry practices.

  • Work Hard on Craft
  • Measure twice, cut once.
  • Feel the grain, work with it not aginst it.
  • Create something unexpected, but faithful to the source from which it sprang.

That is not bad advice for anything.

Sunday, October 25, 2009

Javascript, the Esperanto of the Web.

I just gave the tutorial called Javascript, the programming language of the web at OOPSLA, this coming Sunday. It used to be called "the Esperanto of the Web", but no one seemed to know what that was, so I had to rename it.

If you want to learn good Javascript, there are three books that you need to read. Javascript, the Good Parts, by Douglas Crockford is a really good, really thin book that teaches you all that is worth knowing about the language. The other two books, the Little Schemer, and the Seasoned Schemer, by Matthias Felleisen and Daniel P. Friedman will teach you functional programming using Scheme. The reasons the Schemer books are so good for learning Javascript is that Javascript is more like a dialect of Scheme than it is like a dialect of C.

After reading these books you have a whole new appreciation for Javascript.

Here is one of the most beautiful functions in computer science, the Y-combinator in Javascript.

// The Y Combinator
var Y=function (gen) {
 return function(f) {return f(f)}(
  function(f) {
   return gen(function() {return f(f).apply(null, arguments)})})}

The fact that the Y-combinator can be written i Javascript shows the power and elegance of the language.

Friday, October 09, 2009

Lists in Scala

As with most functional languages, lists play a big roll in Scala. Lists contains, among others, the following operations.

// List in Scala or homogenous, they are declared as List[T]
val names: List[String] = List("Arnold", "George", "Obama")

// Lists are constructed from two building blocks :: and Nil
assert(names == "Arnold" :: "George" :: "Obama" :: Nil)

// Gets the first element of a list
assert(names.head == "Arnold")

// Gets the rest of the list
assert(names.tail == "George" :: "Obama" :: Nil)

// Checks if the list is empty
assert(List().isEmpty)

Instead of using head and tail, pattern matching is commonly used.

def length(xs: List[T]): Int = xs match {
 case Nil => 0
 case x :: xs1 => 1 + length(xs1)
}

From these simple functions a flora of functions is built.

// List length
assert(names.length == 3)

// ::: appends two lists
val namesTwice = names ::: names
assert(namesTwice == List("Arnold", "George", "Obama", "Arnold", "George", "Obama"))

// last gets the last element
assert(names.last == "Obama")

// init gets all but the last
assert(names.init == "Arnold" :: "George" :: Nil)

// reverse reverses the list
assert(names.reverse == "Obama" :: "George" :: "Arnold" :: Nil)

// drop drops the first n items
assert(names.drop(2) == "Obama" :: Nil)

// take keeps the first n items
assert(names.take(1) == "Arnold" :: Nil)

// splitAt, does both take and drop at the same time, returning a tuple
assert(names.splitAt(1) == (List("Arnold"), List("George", "Obama")))

// indeces, gives my the indeces of the list
assert(names.indices == List(0, 1, 2))

// zip, zips two lists together
assert(names.zip(names.indices) == List(("Arnold", 0), ("George", 1), ("Obama", 2)))

// toString returns a list as String
assert(names.toString == "List(Arnold, George, Obama)")

// mkString, lets you join the string with a separator
assert(names.mkString("-") == "Arnold-George-Obama")

There are also a few functions for converting to and from lists.

val array = Array("Arnold", "George", "Obama")

// Convert the list to an Array
assert(names.toArray == array) // Equality does not work for arrays
java.lang.AssertionError: assertion failed
 at scala.Predef$.assert(Predef.scala:87)
...

// If we convert it back it works
assert(names.toArray.toList == names)

// We can also mutate the array with copyToArray
List("Hilary").copyToArray(array, 1)
assert(array.toList == List("Arnold", "Hilary", "Obama")) 

// elements will give me an iterator
val it = names.elements
assert (it.next == "Arnold")

Pretty slick, but now it is time for the good stuff, Higher Order Functions!

// map, converts from one list to another, notice the placeholder syntax (_)
assert(names.map(_.length) == List(6 ,6, 5))

// Get the first char of the words
assert(names.map(_.charAt(0)) == List('A', 'G', 'O'))

// Get the names as lists
assert(names.map(_.toList) == List(List('A', 'r', 'n', 'o', 'l', 'd'),
 List('G', 'e', 'o', 'r', 'g', 'e'), List('O', 'b', 'a', 'm', 'a')))

// When you have a list of lists, you can use flatMap
assert(names.flatMap(_.toList) == List('A', 'r', 'n', 'o', 'l', 'd',
 'G', 'e', 'o', 'r', 'g', 'e', 'O', 'b', 'a', 'm', 'a'))

// Filter is used to filter out specific elements that satisfy the predicate.

// Filter out all names of length 6
assert(names.filter(_.length == 6) == List("Arnold", "George"))

val chars = names.flatMap(_.toList)

// Filter out all chars larger than 'a' (capitals are smaller in ascii)
assert(chars.filter(_ > 'a') == List('r', 'n', 'o', 'l', 'd', 'e', 'o', 
'r', 'g', 'e', 'b', 'm'))

// And combine them
// Give me the first letter of all words with length 6
assert(names.filter(_.length == 6).map(_.charAt(0)) == List('A', 'G'))

There is a bunch of other useful functions based on filter.

// partition returns a pair of list (satisfied, not satisfied)
assert(names.partition(_.length == 6) == (List("Arnold", "George"), List("Obama")))

// find returns the first element that satisfy the predicate
// Since this function may not be satisfied, an optional value is used
assert(names.find(_.length == 6) == Some("Arnold"))

// An optional value returns Some(value) or None
assert(names.find(_.length == 7) == None)

// takeWhile and dropWhile take resp. drop while the predicate is fulfilled
assert(chars.takeWhile(_ != 'o') == List('A', 'r', 'n'))
assert(chars.dropWhile(_ != 'm') == List('m', 'a'))

// Span does both at the same time
assert(chars.span(_ != 'o') == (List('A', 'r', 'n'), 
List('o', 'l', 'd', 'G', 'e', 'o', 'r', 'g', 'e', 'O', 'b', 'a', 'm', 'a')))

// forall checks that a predicate is true for all elements of the list
assert(!chars.forall(_ == 'a'))
assert(chars.forall(_ >= 'A'))

// exists checks that a predicate is true for some element of the list
assert(names.exists(_.length == 5))
assert(!names.exists(_.length == 7))

// sort, sorts a list according to an ordinal function
assert(List(3, 7, 5).sort(_ > _) == List(7, 5, 3))

The fold functions, fold left (/:) and fold right (:\) inserts operators between all the elements of a list. The difference between them is whether they start or end with the base element.

fold left: (0 /: List(1, 2, 3)) (op) = op(op(op(0, 1), 2), 3)

fold right: (List(1, 2, 3) :\ 0) (op) = op(1, op(2, op(3, 0)))


// Define the sum function for lists with fold left
def sum(xs:List[Int]): Int = (0 /: xs)(_ + _)
assert(sum(List(2, 3, 4)) == 9) 

// Define the product function for lists with fold right
def prod(xs:List[Int]): Int = (xs :\ 1)(_ * _)
assert(prod(List(2, 3, 4)) == 24) 

// Define reverse in terms of fold
def reverse[T](xs: List[T]) = (List[T]() /: xs) ((ys, y) => y :: ys)
assert(reverse(List(1, 2, 3)) == List(3, 2, 1))

Thats it for the methods of the List class. In the Companion List object we also find some useful functions. We have been using one of them, all the time.

List.apply or List() creates a list from its arguments.

Apart from this one, there are some other worth mentioning.

// List.range creates a list of numbers
assert(List.range(1, 4) == List(1, 2, 3))
assert(List.range(1, 9, 3) == List(1, 4, 7))
assert(List.range(9, 1, -3) == List(9, 6, 3))

// List.make, creates lists containing the same element
assert(List.make(3, 1) == List(1, 1, 1))
assert(List.make(3, 'a') == List('a', 'a', 'a'))

// List.unzip zips up a list of tuples
assert(List.unzip(List(('a', 1), ('b', 2))) == (List('a', 'b'), List(1, 2)))

// List.flatten flattens a list of lists
assert(List.flatten(List(List(1, 2), List(3, 4), List(5, 6))) == List(1, 2, 3, 4, 5, 6))

// List.concat concatenates a bunch of lists
assert(List.concat(List(1, 2), List(3, 4), List(5, 6)) == List(1, 2, 3, 4, 5, 6))

// List.map2 maps two lists 
assert(List.map2(List.range(1, 999999), List('a', 'b'))((_, _)) == List((1, 'a'), (2, 'b')))

// List.forall2
assert(List.forall2(List("abc", "de"), List(3, 2)) (_.length == _))

// List.exists2
assert(List.exists2(List("abc", "de"), List(3, 4)) (_.length != _))

And as if all this was not enough, Scala also support For Expressions. In other languages they are commonly known as List Comprehensions.

A basic for expression looks like this

for ( seq ) yield expr where seq is a semicolon separated sequence of generators, definitions and filters.

// Do nothing
assert(names == (for (name <- names) yield name))

// Map
assert(List('A', 'G', 'O') == (for (name <- names) yield name.charAt(0)))

// Filter
assert(List("Obama") == (for (name <- names if name.length == 5) yield name))

val cartesian = for (x <- List(1, 2); y <- List("one", "two")) yield (x, y)
assert(cartesian == List((1, "one"), (1, "two"), (2, "one"),  (2, "two")))

// And now the grand finale, the cartesian product of a list of list
def cart[T](listOfLists: List[List[T]]): List[List[T]] = listOfLists match {
 case Nil => List(List())
 case xs :: xss => for (y <- xs; ys <- cart(xss)) yield y :: ys
}
val cp = cart(List(List(1,2), List(3,4), List(5,6))) 
assert(cp == 
  List(List(1, 3, 5), List(1, 3, 6), List(1, 4, 5), 
  List(1, 4, 6), List(2, 3, 5), List(2, 3, 6), 
  List(2, 4, 5), List(2, 4, 6)))

Ain't it beautiful so say!

Saturday, October 03, 2009

Scream, Project Management for the Real World

Many companies today find themselves in a situation where they have a working product, but adding new features takes forever. Even worse, when new features are added, old features stop working.

To solve this problem many companies have adapted Scrum. Scrum has a nice lightweight appeal. All you need is:

  • A product owner who cares for a backlog with prioritized stories.
  • A team that cares about their craft and take responsibility to deliver a subset of new features every month.
  • A scrum master that makes sure the product manager and the team are playing by the rules of Scrum.

Thats it, the recipe for success...

But, what if you are not living in la-la-land where everyone on the project cares about the product?

What if your product manager doesn't keep a prioritized log of testable stories because she doesn't care. She just works here!

What if your team cares more about going surfing, than delivering well-tested, high-quality code.

Enter Scream!

Scream

Scream is project management for the real world! Scream is the way of managing unmotivated development organizations. The method itself is not new, it has been used for centuries to manage everything from husbands to entire countries.

In Scream, all you need is:

  • A product manager who handle the requirements.
  • A team whom will develop the requirements.
  • A Scream master who will make sure that the team is playing by the rules of Scream.

At first glance, it looks deceptively like Scrum, but the rules are different: The Scream master is responsible for the product being delivered at high quality and may use any means he sees fit to make it work.

This makes all the difference in the world.

Now all you need is a Scream master with enough gust to deliver.

The Scream Master

The ideal Scream master is Begbie in Trainspotting. He has all the characteristics of a good Scream master:

Francis Begbie is an aggressive pit bull terrier, a monstrous, brawling hard man ready to explode at any moment, at anyone, for any reason. Begbie isn't afraid to test his fighting prowess against the largest of opponents. "Begbie didn't do drugs, he did people," says Renton. His sole ambition seems to be to jack someone in.

The Process

After you have selected the Scream master, you have to let him know that he will be judged on the performance of the entire development team, including, the product owner.

You also need to set up some acceptance criteria for what done is:

  • All stories in the backlog, must be SMART, Specific, Measurable, Attainable, Realistic and Timely.
  • The code should be DRY.
  • 100% unit-test coverage of all non-trivial methods.
  • Acceptance test for all stories.

The you set the project in motion.

Some Typical Scenarios

The backlog items are not SMART.

The product owner says: "I didn't have the time." Begbie: Slaps her face, "You daft c**t, these items better by SMART, right f***in' now, or I will glass you."

Bugs appear in production, due to missing unit tests.

A developer says: "It worked on my machine." Begbie: Punches him in the nose, "You f***in' buftie, if one more bug enters the system on your account you've f***in' had it."

Typical stand-up meeting:

You ken me, I'm not the type of c**t that goes looking for f***in' bother, like, but at the end of the day I'm the c**t with a pool cue and you can get the fat end in you face any time you f***ing want, like.

A scream is not only against his team and product owner, he is against everyone. This makes him excellent for dealing with impediments. Project management is all about communication and motivation and no one can get the message through like Begbie.

Begbie: "I need access to the Active Directory." SA: "I don't have the time." Begbie: "I need access to the Active Directory." SA: "You need to fill out this form." Begbie: "I NEED ACCESS TO THE ACTIVE DIRECTORY." SA: "OK, here you go."

Notes on Greg Young's on DDD

I watched Greg Young talk about DDD on InfoQ. Here are my notes on the talk.

  • Only use domain driven design on appropriate projects. Most projects are not suitable.
  • Use state transition event streams to communicate between different bounded contexts.
  • Bounded contexts are one of the very keys of domain driven design. The same word may have different meanings in different contexts, and this is OK.
  • Use OO, avoid setters. Objects have behaviors, not shapes.
  • If you always have valid objects, you avoid the problem of having to check if an object is valid all the time. IsValid is not the solution.
  • Always use the domain experts and end-users language.
  • Separating commands from queries gives the benefit of eventually consistent, queries may read from a different place than the commands.
  • Coupling is not a problem, if it's in the same layer.
  • Model the view, such as screens, as reports, with no transactional behavior.
  • Explicit state transitions remove the need for auditing. They are the audit.