OOPSLA 2009 Thursday, October 29th

Moving Fast at Scale, Lessons Learned at Facebook, Robert Johnson

Facebook has over 1 million active users per engineer.

Slowing down to get it right is not a good idea, unless you know exactly that your idea is right. If you try things fast, you can try out more things and you can get feedback fast.

How do you get fast?

• Never block developers
• Give developers control. At Facebook developers code, test and deploy. No QA is involved.
• Tune processes for speed
• No deadlines, but the site cannot go down.
• Frequent small changes, no delays, easier to isolate bugs.
• All development in trunk with weekly releases.
• Deployment tools
• Gatekeeper
  
• Do less work
• Build tools
• phpsh -interactive development
• diffcamp - code review
• XHProf - profiler
• Scribe - moves data from server to central repository.
• Hive - data warehouse infrastructure on top of hadoop
• Work with open source

How to deal with a lot of users?

• Scale Horizontally
• Web Server, the relations have moved to the web layer from the database.
• Memcache
• Database, only used as a persistence layer. No joins etc.
• Thrift
• C++, Java, Erlang, Python

Open Problems/Future Work Languages for real-time data-access parallel with lots of dependencies. Distributed indexing Automatic clustering of data Profiling of parallel data * Better ways of expressing client and server code + Javascript and PHP are not well matched.

Toward Cloud-Agnostic Middlewares, E. Michael Maximilien

The Cloud Computing Landscape

• Service Providers
• Platform Providers
• Service Brokers
• Application Providers (SaaS)
• Users

Challenges in Cloud Computing

• Data lock-in
• Application programming lock-in
• Management lock-in
• API lock-in
• IncreaseRisk
• Security and Privacy
• Catastrophic failures
• Business models and impacts

Research Opportunities

• Cloud middleware
• Agnostic wit respect to providers, frameworks and interfaces.
• Learn Best Practices
• Fluid cloud application deployment
• Optimize could usage prices
• Give indication of cloud readiness

Use Cases

• COBRA Java text analytics solution. (JavaEE)
• CoScripter application (RoR)
• JumpStart sMash application (WebSphere, Smash)

The Architecture

• Core APIs
• Cloud APIs
• Cloud Adapters
• Clouds and Brokers

IBM Altocumulus

Altocumulus is a configuration tool that helps you to deploy your application into different clouds. It supports multiple different service and platform providers. It helps you configure your deployment so that it follows the commonly known "good" practices. You can use it via the Web or via a RESTful API. It also supports Atom and RSS feeds. It is very interesting.

Lessons Learned

• Cloud Providers need to include some support service s to be enterprise ready
• Cloud Providers need to expose flexible image creation facilities
• Cloud virtual machines are mostly hardware based and their memory management is suboptimal
• Best Practice Pattens works but need to evolve.
• Standardized hardware is important when implementing cloud infrastructures
• Instance monitoring capabilities should coma as part of the cloud facilities and APIs
• The cloud space is still maturing

When Users Become Collaborators: Towards Continuous and Context-Aware User Input, Walid Maalej et al.

User input is critical for the success of projects.

Various types of input:

• Field observation, Lead Users
• Perpetual Beta, Legacy Documents, Usage Data
• Issue and Bug Report, Enhancement Request, Feature Request
• Workshop, Interview, Survey, Classification Requests

Built in Feedback Mechanisms

Examples: Application Bug Reports, Application Usage Data, Discussion groups

Most applications don't have feedback mechanisms built into them.

Users are motivated by feedback, so if they submit a report and don't get any feedback, they will probably not do it again.