Erlang: Proposal for the next architecture meeting in Paris (long)

[Mickael Remond <mickael.remond@xxxxxxxxxxxxx>]

Hello,

As a follow-up of the Linux Solutions 2004 in Paris, and the thread
regarding benchmark, here is the promised introduction to Erlang
programming language and middleware.

Tsunami benchmark tool:
=======================

In a previous mail regarding the benchmarking framework I pointed to
IDX-Tsunami. Tsunami is in my opinion on of the most advanced Open
Source framework. It is distributed, fault-tolerante, highly scalable
and can be extended to support various protocols.

Here is a quick list of its main feature:
- High Performance and clustered (the load can be distributed on a
  cluster of client machines). IDX-Tsunami has been used to validate the
  scalability of a instant messenging platform (based on Jabber) for
  Tiscali/LibertySurf. 10,000 simultaneous clients were simulated on 3
  machines. IDX-Tsunami has been used to validate the scalability of a
  HTTP/S platform. 25,000 clients were simulated with peaks of 3,000
  simultaneous requests on 4 machines, for a particularly heavy site.
  Tsunami is also relevent for benchmarking small cluster, as it can
  simulate heavy load from a single machine. Other solutions most often
  requires to set-up a cluster.
- Multi-Protocols using a plugin system: HTTP, HTTPS and Jabber are currently
  supported. LDAP, SMTP and SOAP are on the TODO list.
- Sensor to monitor the benchmarked cluster: For the moment RAM, CPU,
  memory, network load, is being monitor. SNMP value monitoring is on
  the TODO list.
- Stochastic, non-determinist model: : in order to generate a realistic
  trafic, user thinktimes and the arrival rate can be randomize using a
  probability distribution (exponential currently).
- Scenarii definitions in XML (Or with a Web proxy): The load during the
  bench can be precisely defined by defining phases, with differents
  users arrival rates. Several different scenarii can be played during
  the same benchmark to simulate various kind of usage (For example you
  can define 80% of users playing a lightweight scenario, while 20% are
  doing a more demanding use of the application)
- Already very stable and used in production.
- Cookie and web authentification support,
- Proxy to record web scenarii,
- Framework proven on already two very different protocol: HTTP
  (disconnected) and Jabber (Connected protocol).
- Free Software (GPL)

The tools is only command-line for the moment (Can however generate
pretty graphes). A graphical console is on the TODO list.

The association Erlang-project is promoting this framework, as this is
already a proven approach. IDX-Tsunami has been used on several
industrial projects to simulate huge numbers of users and huge load. It
has succeded where ohter Open Source benchmark tools failed.
I propose to present it during the next architecture meeting, show its
potential and answer questions.




Erlang:
=======

The power of this injection tools rely on Erlang programming language
and middleware.

Erlang is a concurrency oriented programming language. It has been
created as a means to write robust applications, running on a set of
networked computers. Erlang is characterized by the following features:

- Concurrency
- Distribution
- Robustness,
- Soft-real time (The Virtual machine is beautifully designed and
  efficient. In particular, memory management is done to ensure small
  footprint)
- Hot code upgrade
- Open Transaction Platform (OTP) to simplify the realisation of
  application.

Originally developed at Ericsson (in 1985 !), the Erlang development
environment is now available under a Free Software licence (since 1998).
It's a functional language, to reduce side effect and ensure easy and
transparent distribution of an application on a set of Erlang nodes.
The language runs on a virtual machine, on a large number of platform
(very portable): Linux, Windows, *BSD, MacOSX, various flavour of Unix,
... The Erlang code can also be natively compiled on Linux x86, Solaris
and Windows.

It offer is more a complete middleware than a simple language. It is
provided with OTP (Open Transaction Platform) that simplify the
realisation of robust, distributed application. The framework for
examples allow upgrade of running applications without disconnected the
client currently using it (Hot code upgrade). Crash are handled through
a fundamental mechanism: Error code is usually separated from the
standard code and process are organised in a supervision with workers
and supervisors.

It is productive and expressive language: You usually write 4 to 10 time
less code than with other languages.

It is an industrial language provided with many crucial development
tools:
- Debugger (concurrent), application, monitor, process manager
- Code coverage, and Cross reference analyser,
- Distributed fault tolerance database (called Mnesia),
- Corba orb.
- Web server,
- Crypto SSL support,
- C integration library (To written bindings and virtual machine
  drivers).
- Java integration library,
- Sophisticated logging framework,
- Various protocol support (SNMP, ASN.1, ODBC)
- Open Transaction Platform (OTP) framework,
- Many standards features in the library,
- ...




I am strongly convinced that Erlang projects could efficiently complete
existing Objectweb tools. I have already several ideas on thoses
aspects, taking advantages of the unique aspects of Erlang:
- Benchmarking framework (Based on IDX-Tsunami)
- Highly scalable web server / proxy (Based on Yaws + an AJP connector):
  This should be put in front of Tomcat as a replacement of Apache to
  provide on more fault-tolerant and scalable proxy. Yaws has proven
  being able to handle 80 000 simultaneous user connections on a single
  node.
- Several others ideas that we can discuss later on.



Summary:
========

Erlang is a robust industrial middleware, with many outstanding
use already for strategic projects (Huge ATM routers with 7 nine
availability, Athletic events managements on statium, complete banking
system, Nortel appliances, ...). Erlang, as a functionnal language, is
already used to build robust and scalable Service Oriented Architecture
(SOA).

If you are agree, I propose to present you Tsunami, Erlang and how
Erlang projects could complete the Objectweb middleware offer with some
precises dedicated components, during the march architecture meeting in
Paris.

In the mean time, you can get further by reading:
- http://www.erlang.org/ and especially http://www.erlang.org/white_paper.html
- http://www.erlang-projects.org/
- http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf: Joe
  Armstrong Thesis on Erlang: Making reliable distributed systems in the
  presence of software errors (This is really a must read to understand
  the philosophy behind Erlang).
- French readers only for the moment: 
  
http://www.erlang-projects.org/Public/news/events/linux_solutions_2004/dossier_de_presentat/view
- http://www.idealx.org/prj/idx-tsunami/features.en.html
- http://www.idealx.org/prj/idx-tsunami/screenshots.en.html

Please do not hesitate to contact me if you need more details, or needs
helps evaluating the technology.

-- 
Mickaël Rémond
 http://www.erlang-projects.org/