Gridshore

3 Jan 2009

Recently, SpringSource released the first version of their DM server. The SpringSource DM Server provides the ability to build enterprise web applications. In the basis, S2DM is a fine mixture of Equinox and Tomcat.

Building OSGi-based web applications was already possible, but it is tedious and error prone work. The typical hello-world example was easy to get going, but as soon as Hibernate or any other framework that helps in larger applications show up, so do your good old class loading problems. For classes to be visible in OSGi, a bundle must declare an Import-Package entry in the Manifest file. Chances are small that Hibernate (even if it were packaged as an OSGi bundle) has these entries for your persistent classes. This is where S2DM server comes in. It allows the more-than-hello-world web applications to be deployed in an environment where you can benefit from the best of OSGi, without the class loading misery. To do this, they have included some extra Manifest entries that are translated to OSGi-compliant directives at load time.

Comparable to the WAR, or better, EAR file, S2DM server supports the PAR file. A PAR file is much like a Jar, with some special headers in the Manifest file, containing all your bundle jars. Some of these jars may contain web bundles, while other typically contain domain classes or the service layer implementation. Contrary to EAR files, a PAR should only contain your own code. It is best practice to deploy frameworks and third party libraries separately. I’ll explain why later on.

With enterprise applications come enterprise development processes, using continuous integration, build servers and maven. In this post, I’ll lay out what you need to get maven to build S2DM server compliant bundles, and better, PAR files.

OSGi manifest headers

But before we move straight to maven, let’s have a look at what a Spring DM bundle looks like. Basically, a Spring DM bundle is an ordinary OSGi bundle. It has the same requirements for the MANIFEST.MF file, of which the major headers are listed below:

• Bundle-SymbolicName is the only mandatory manifest header. It represents the technical name of your bundle. This value, in combination with the Bundle-Version, if present, must be unique.
• Bundle-Version is not mandatory, but defaults to 0.0.0, which isn’t a meaningful version. An OSGi compatible version number has the format: <major>.<minor>.<micro>-<classifier>.
• The Import-Package header allows you to specify a list of packages that your bundle requires to operate. It is also possible to specify version ranges and that some packages are optional.
• DynamicImport-Package can be used to dynamically import packages at the moment they are required. Use a wildcard (*) to set package patterns to limit the dynamic import to.
• The last entry I’ll mention here is Export-Package. It specifies the packages of the current bundle that form its API. Classes in these packages are available for other bundles to use.

For more details about OSGi manifest headers, see the OSGi-core specification, section 3.2.

The OSGi specification requires you to import a package (which has to be exported by another bundle) for a class loader to find it. This lead to problems when using frameworks such as Hibernate, which has to instantiate classes from your application. There is no way a Hibernate OSGi bundle can know beforehand which classes it will instantiate. Another problem is that the number of packages you will have to import is probably very large. Frameworks sometimes offer large amounts of classes that have been grouped into numerous packages. To mention each of those packages in your Import-Package manifest header is a lot of work. You could use the DynamicImport-Package to import all packages you require, but that will create as many problems as it solves. OSGi is designed to only resolve bundles when all requirements have been met. By dynamically importing packages, the OSGi framework has no way to know whether requirements are met, and will resolve your bundle. During runtime, you could get class loader exceptions.

Spring DM specific manifest headers

To counter both the large number of imports and the class loading problems, Spring DM introduced some additional manifest headers. Not that these are not (yet) official OSGi compatible headers and are only evaluated by Spring DM compatible OSGi environments, such as Spring DM Server, for which there is a community and an enterprise version.

The most important additional Spring DM manifest headers for bundles are:

• Import-Library can be used to import a library. A library is a specification of a set of bundles that should be imported as a whole. Examples of libraries are Hibernate and the Spring Framework.
• Import-Bundle is used to import all the packages from a specific bundle, such as Spring-Core. Typically, this header is used in library specifications, but it may also be used in bundles manifests.
• Module-Type specifies the type of this bundle. At the time of writing, only “Web” is supported. When adding this header to the manifest, you indicate that this bundle is to be treated as a Web Module.
• Web-context path specifies the URL in which a Web Module must be deployed. It default to the module’s file name minus the extension.
• Web-FilterMappings is used to specify the filter names (that reference spring bean names) and their URL mappings
• Web-DispatcherServletUrlPatterns indicates the URLs that have to be mapped to the DispatcherServlet. In Spring DM, a dispatcher servlet is automatically initialized with the context files in META-INF/spring/*.xml.

Creating an OSGi bundle with maven

Now that we know which headers have to be generated, lets move on to maven to find how we can generate a manifest file for them. Obviously, you could generate one yourself. This is, however, tedious and error prone. It is best to conform yourself to some design standards, such as using .internal or .impl packages to store classes that are not part of your public API. When you do so, the maven-bundle-plugin.

The maven-bundle-plugin is used to generate an OSGi bundle from your module. In order to make it work, you have to make a few modifications to an otherwise normal pom.xmlfile:

1. The <packaging> element value must be set to bundle: <packaging>bundle<packaging>
2. Configure the org.apache.felix:maven-bundle-plugin

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.felix</groupId>
            <artifactId>maven-bundle-plugin</artifactId>
            <extensions>true</extensions>
            <configuration>
                <instructions>
                    ...
                </instructions>
            </configuration>
        </plugin>
    </plugins>
</build>

This configuration will generate OSGi compliant manifest files. The Bundle-SymbolicName will be formed from your modules groupId and artifactId and the Bundle-Version will be set to you module’s version. By default, all packages that are mentioned in your source code’s import statements will be imported by your bundle, even if these packages are also included in your bundle (see this post on the OSGi Alliance website). Furthermore, all packages available in your bundle will be exported, even the .impl and .internal packages.

Manifest-Version: 1.0
Export-Package: springdm.domain;uses:="javax.persistence",springdm.per
 sistence.impl;uses:="springdm.domain,org.springframework.transaction.
 annotation,springdm.persistence,javax.persistence,org.springframework
 .beans.factory",springdm.persistence;uses:="springdm.domain"
Private-Package: springdm.*.impl.*
Built-By: allard
Tool: Bnd-0.0.255
Bundle-Name: Spring DM Persistence
Import-Library: org.hibernate.ejb;version="[3.3.2.GA,3.3.2.GA]”
Created-By: Apache Maven Bundle Plugin
Import-Bundle: org.springframework.orm;version=”[2.5.6.A,2.5.6.A]”
DynamicImport-Package: *
Build-Jdk: 1.6.0_06
Bundle-Version: 1.0.0.SNAPSHOT
Bnd-LastModified: 1230983705375
Bundle-ManifestVersion: 2
Bundle-SymbolicName: springdm.persistence
Import-Package: com.mysql.jdbc.jdbc2.optional,javax.persistence,javax.
 sql,org.hibernate.jdbc,org.springframework.beans.factory,org.springfr
 amework.transaction.annotation,springdm.domain,springdm.persistence,s
 pringdm.persistence.impl


I am not going into more detail about how to configure the bundle plugin to keep implementation packages private. See the felix documentation for more information about that.

Adding Spring DM manifest headers

Well, that actually quite easy. All you have to do is specify the name of the header as a sub element in the <instructions> element and the value of the header as the value of that element, for example:

<Import-Bundle>org.springframework.orm;version="[2.5.6.A,2.5.6.A]“</Import-Bundle>

The problem is to find out which bundles or libraries there are to import, and which versions they have. That’s why Spring started the Spring DM bundle repository. You can use it to find libraries and OSGi bundles for most commonly used frameworks. For each of thus libraries or bundles, it will tell you which maven dependency to use (for compilation) and which manifest entry has to be added to your bundle. Make sure that your maven dependencies and manifest entries are updated, otherwise you could run into problems at runtime.

For maven to find the artifacts in the bundle repository, you have to configure some repositories. You can find them here.

If your bundle is a Web Bundle, add the required <Module-Type>Web</Module-Type> element to the instructions, as well as any other required Web-* headers.

That’s actually it. That’s all it takes to generate Spring DM specific manifest entries.

Creating a PAR deployment unit

The PAR file is the deployment unit for Spring DM applications. It is a Jar file with special manifest headers that contains all application specific bundles. You should never include framework bundles in the par file. In conventional environments, class loaders would prevent you from having different versions of the same class on your class path. Therefore, you had to add frameworks to the WAR file, to prevent version conflicts with other applications. In OSGi, there is no problem with having the same class multiple times. You just specify a version restriction in your import headers and OSGi takes care of the resolution process. To keep PAR files small, you can deploy required libraries and bundles in the /repository/libraries/usr and /repository/bundles/usr folders respectively. Any application can reuse any existing framework bundles.

The PAR file has another advantage. Spring DM Server prevents bundles from outside the PAR file from accessing bundles inside the PAR file. This way, you can deploy applications without running the risk of classes and services intermingling.

Now, let’s get back to maven to create ourselves a par file. First, create a new module in your maven project. This module’s pom file should have <packaging> set to par. Next, add dependencies to each module you want to be included in the par file. The par plugin does not include transitive dependencies, since framework bundles should not be included. Finally, add the maven-par-plugin to your plugin configuration.

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-par-plugin</artifactId>
            <version>0.2.0</version>
            <extensions>true</extensions>
            <configuration>
                <fqn>true</fqn>
                <applicationSymbolicName>gridshore.samples.springdm</applicationSymbolicName>
                <applicationVersion>1.0.0</applicationVersion>
                <applicationDescription>Gridshore Spring DM sample</applicationDescription>
                <applicationName>Gridshore Spring DM sample</applicationName>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>par</goal>
                    </goals>
                    <phase>package</phase>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

You probably need to configure a repository to find this plugin: http://repo.steademy.com/beta/maven2. You can find documentation of the maven-par-plugin here.

Now run mvn install and see your PAR being created for you. Since your PAR does not contain framework bundles, you have to install them separately. To make sure that all dependencies are available, you can use the maven-assembly-plugin to create a zip file with the PAR file as well as all the framework bundles you require.

Have fun!

GroovyBlogs.org

2 Jan 2009

dzone_url = "http://thebull.macsimumweb.com/has-anyone-gotten-log4j-and-grails-to-play-nice";While basic configuration of log4j in Grails can be tricky, I’ve managed to get down the basic config...

Gridshore

30 Dec 2008

This blog item I have dedicated to zoom in on Bugs, a specific type of Waste (muda which is Toyta lexicon) aka DEFECTS as they have a huge effect on Productivity.

But lets start with the Analysts and their magic 8-bal that has shown them that Lean Software Development will become the hit of 2009.

Bloated applications, platforms and architectures slow application development and make QUALITY CONTROL and everyday usage time-consuming and nonproductive, said Forrester Research principal analyst John R. Rymer. Forrester’s newly-published report, Lean Software is Agile, Fit-to-Purpose, and Efficient, lays out how software got so fat, costly and inefficient; the evidence that IT organizations are moving to lean software; the challenges involved in lightweight software development; and strategies for joining the movement.

There is a lot to say about Lean Software Development as I already mentioned in my blog item on the REAL cost of Offshore .

The best you can do is to read the book of Mary and Tom Poppendieck and of course The Toyota Way. The XP, Scrum and even some RUP adepts under the readers they know that it is all about being Agile (whatever it means).

“Agile development is independent of any technical platform or development approach “It’s a method. What’s neat about this is that people are delivering application features sometimes every two weeks or each month. Rather than deliver the big-bang project after years of work, they’re delivering applications in increments. They’re working in an incremental fashion to deliver features over time, providing value quickly and continue to add value. That’s a way to modulate your costs, to spread your costs and investment over a period of time.”

And Yes, it`s not easy to break through old habits as I know by experience that companies are really good in adopting RUP and then transform this iterative methodology into a Waterfall approach.

What have DEFECTS to do with AGILE development and Offshore?

Partly taken from Yourdon Report on a presentation that Michael Mah(QSM) gave at the New York City SPIN group.

Assume, for the moment, that you’ve got a software development project where the end-users (or customers, stakeholders) and the developers (i.e., programmers, architects, database designers, business analysts, QA personnel, etc.) are all working in company X, local. And imagine that you have another software development project, of equivalent size and complexity, where the end-users are in company X, but the development team is located in an offshore outsourcing company on the other side of the world (e.g., China, India, Singapore, eastern Europe, etc.) Why is it that software metrics (from QSM) are showing that project #2 will have approximately 2.8 times as many software defects as project #1?

Precisely because the end-users and the developers are not “co-located” (in contrast to many of the “agile” or XP development projects, where a “customer proxy,” or end-user representative, is literally sitting in the same office cubicles, alongside the programmers), communication problems escalate. Despite the tools, the world, is not flat: you can’t just assume that software developers can be sprinkled randomly around the globe, where they’ll participate in a complex software development effort with no problems at all.

The second reason for the higher defect rate is because offshore outsourcing firms typically offer an essentially unlimited number of technical personnel at a lower labor rate, many firms think they can achieve faster delivery schedules by hiring larger teams of offshore developers than they would if they staffed the project locally - and yet still save money. Which I have explained in my previous article on “The REAL cost of Offshore” is wrong. Thus, if the firm had previously staffed a project with 10 “expensive” local programmers, they figure they can hire 20 “cheap” Indian programmers, and still save money - while (in theory) cutting the schedule in half, because they’re blissfully unaware of Brooks’ Law .

Aside from the schedule issue, it’s a known fact (but pity enough not to everyone) that software defects are strongly correlated to the number of “lines of communication” between the developers on the team; and that number increases combinatorially (i.e., N*(N-1)/2) as the size of the team increases (this is a variation on Conway’s Law , first articulated by computer programmer Melvin Conway in 1968, which says that the structure of a system is isomorphic to the structure of the organization that builds it). Hence, the number of defects in a software system increases in a manner proportional to the square of the number of developers; and as a result, he says, offshore projects are more likely to deliver buggy software.

There is a lot more to tell about core metrics needed to measure, compare and predict productivity but let me stick to the topic of Defects.

On an average, we introduce and detect 0.4 Defects (without Unit test defects) for every Staff day (including training, project management, requirements writing, testing, documentation, etc.) of effort I.o. in a medium project (10 people, 6 months) we can expect more than 500 defects!

You can find these research figures e.g. at the different SPIN presentations or use the QSM Slim tooling. But there is more, it also a known fact that the longer a Defect is in the system the more it cost (time, money, risk) to repair it. A defect inserted while writing Requirements but discovered during Acceptance test can cause a lot of rework as it can have also its influence on other parts of the system and sometimes even the architecture.

Cumulative Defect Density (defect per kloc) versus actual found Defects is an important metric to avoid the big bang on Defects in the last stage of the project. You all know by experience what management decides. First we have to make overtime, then the project team size increases, both resulting in even a higher defect density. Testers are not getting enough time to test, which leads to another important metric, defect turnaround time (time needed to change the state of a defect). More and more fixes are not being able to test and then the inevitable happens, either the project is released with bugs or it is cancelled.

A lot of defects can already being found through reviews lowering the Cost of Quality as Testing is more costly, but there is an optimum to find. But as we all know, reviewing requirements is not easy and I advocate here the Test Driven Approach. You should write Test Scenarios based on the requirements and whenever you encounter a discussion on Testing/Reviewing most likely the Project manager / Customer complaining about the extra cost then there is only one answer to give.

“If its not worth Testing, it`s not worth Developing“`which leads actually to the biggest saver in your project as the Standish Group once has estimated that 55% of the software features are hardly or not used by its users. Another point for Agile techniques, by focusing on the most important requirements first.

This conclude my view and experience on Software Development in general and more specific on what are the most important reasons why projects hardly live up to the expectations of the stakeholders.

Gridshore

27 Dec 2008

Introduction

Author’s note: this is an article that I co-authored
with a colleague, Robert van der Steen. It has also been
published in our company newsletter.

Many of the applications we write for our clients nowadays use the service paradigm: a dedicated and often reusable component within the application that is responsible for a particular task or process. Such components are often written and used in such a way that a component is instantiated once and reused often within the runtime of the application (such as a web service or a Spring managed bean).

Using components in this way is a very resource-friendly way of constructing an application; resource-intensive objects are created only once and reused instead of having to be recreated and reinitialized for each request. However, they do make it necessary to pay special attention to issues that arise in reuse and concurrent use — specifically state management. A common technique to avoid such state problems is to build stateless components.

Unfortunately, mistakes happen in software engineering. Sometimes team members introduce state to shared components without thinking about it. This article discusses using the Memento pattern as an easy way of transforming a stateful into a stateless service.

The problem

In order to illustrate the problem sketched above, let’s introduce an example. This example is an adaptation of a piece of code I stumbled upon in the codebase of a project:

                public class SpringManagedServiceClass {
                        private String oneField;
                        private String anotherField;

                        public boolean anOperation(final String input) {
                                oneField = someOperationOnInput(input);
                                anotherField = somethingElseInvolvingInput(input);
                                totalOperationSucceeded = operationOnTwoStrings(oneField, anotherField);
                                return totalOperationSucceeded;
                        }

                        public String getOneField() {
                                return oneField;
                        }

                        public String getAnotherField() {
                                return anotherField;
                        }
                }

The code above evolved from a class that was not managed by Spring and was instead instantiated each time the service was needed. In that form having state at the class level was no problem: each object was used once and then discarded. This meant it was also thread safe for a client of the service object to extract state using the getters. However, when moved over to Spring the class became a de facto singleton and therefore not thread safe. The person who converted the class to a Spring bean did not recognize this and so introduced a problem by allowing concurrent access to unguarded state.

A quick fix to this problem might be to move back from Spring to instantiation at use. Unfortunately, in this case the class had evolved to depend on a database connection managed by Spring, so moving away from Spring support would be a lot of work.

Introducing the Memento

The Memento pattern is a rather simple pattern that externalizes a classes (visible) state. The way it works is by following these steps:

1. Take a class C which has fields (i.e. objects with state).
2. Introduce a new class CMemento.
3. Move all the fields from C to CMemento. Make them private.
4. Introduce constructors, getters and setters into CMemento as needed.
5. In C, in your service methods, initialize a new instance of CMemento.
6. Refactor C to use the variables in CMemento instead of the variables C used to have itself.
7. Refactor the relevant methods of C to return a CMemento instead of their existing return types.
8. Refactor CMemento and C so that the previous return values of the methods of C are placed in the CMemento objects.
9. Finally, refactor the relevant methods of C again to place their previous return values in their CMemento objects before returning these objects.

The CMemento classes take over the state of the class C in this pattern. C can instantiate a single CMemento instance or as many instances of CMemento as it likes and so create a set of states for itself. Each of these CMemento instances can be the “current” state of any object of type C at any time, so instances of C can roll back to a previous state easily or even swap states.

The Memento pattern is actually not meant to address the problem discussed in this article; it is meant to enable rolling back to a previous state. Fortunately for us the idea can be reused easily to solve our problem as well. Instead of being swappable however, we will refactor our service object to create a Memento object (an object to hold the externally visible state). The public service method will then return this Memento object and the service class will “abandon” its “state” after that, making the service completely stateless.

The solution applied to the example

First, let’s introduce our Memento class:

                public class ServiceMemento {
                        private String oneField;
                        private String anotherField;
                        private boolean serviceResult;

                        public ServiceMemento(final String oneField, final String anotherField, final boolean serviceResult) {
                                 this.oneField = oneField;
                                 this.anotherField = anotherField;
                                 this.serviceResult = serviceResult;
                        }

                        public String getOneField() {
                                 return oneField;
                        }

                        public String getAnotherField() {
                                 return anotherField;
                        }

                        public boolean getServiceResult() {
                                 return serviceResult;
                        }
                 }

Now we can refactor our original service class and replace its state with a Memento object:

                public class SpringManagedServiceClass {
                        public ServiceMemento anOperation(final String input) {
                                String oneField = someOperationOnInput(input);
                                String anotherField = somethingElseInvolvingInput(input);
                                boolean totalOperationSucceeded = operationOnTwoStrings(oneField, anotherField);
                                ServiceMemento result = new ServiceMemento(oneField, anotherField, totalOperationSucceeded);
                                return result;
                        }
                }

After this refactoring all the client code will of course have to be refactored as well. However, all the state access has been replaced by getter calls to the new Memento object, so that refactoring is not very difficult.

The most important thing is that using the Memento has allowed us transform our service class into a thread safe, stateless service class rather easily. The price for this was introducing a new class. This is not as clean a solution as we would have had if the service class had been designed to be thread safe from the very start. But it is an easy fix to implement so it allows us to avoid a lot of (potentially difficult) rework. Also, this Memento class will do nothing but hold the result of the work done by the service class. So at least instantiating and having multiple instances of this class will not be as expensive as having to create multiple instances of the service class.

Summary

This article discussed using the Memento pattern as a quick fix for a thread safety problem in service classes. It covered the problem of accidentally introducing concurrent access problems in singleton classes by introducing state into those classes and then showed how that state can be externalized into a Memento class to make the service class stateless once more.

Graeme Rocher's Blog

23 Dec 2008

So Grails 1.1 Beta 2 is out. Rejoice! There are many new features that are detailed in the release notes. However, one of the main ones in this beta is the new support for Maven.

Regular readers of my blog will probably be aware of my long history as one who, ahem, is not particularily fond of Maven. Granted I am still not particularily fond of Maven, but it is the Christmas period and in the spirit of "why can't we all just get a long" I am proud to say that Grails integrates nicely with Maven now :-)

Peter Ledbrook, core Grails committers, has been tirelessly working away at making the Grails + Maven experience a seamless one. I personally even went as far as installing Maven 2 onto my machine, which was a huge jump for me I can tell you, and I can tell you Peter's done a great job!

As ever we encourage Maven users to give the Maven integration a go and report any issues that you may have with it in JIRA. Enjoy!

Gridshore

21 Dec 2008

The mac is a pretty complete system for developing applications. Once I was surprised that tools like subversion and maven come pre-installed. There is a big advantage for the non-technical users that these tools are pre installed. Although I doubt that these people will use it a lot. Would you teach a person to use command line subversion? I guess not. You would teach them to use a tool like Versions. As a developer you can run into problems when you want to upgrade one of these libraries. Some time a go I have already written a post about upgrading maven on the mac. This is a short post about upgrading subversion.

Like I said, mac osx comes with a subversion client out of the box. At least Leopard does. If you do not believe me, try the following command and watch the output:

svn –version

svn, version 1.4.4 (r25188)
   compiled May 31 2008, 03:45:57
Copyright (C) 2000-2006 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).
The following repository access (RA) modules are available:
* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
  - handles 'http' scheme
  - handles 'https' scheme
* ra_svn : Module for accessing a repository using the svn network protocol.
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme

This is not the most recent version of subversion. That comes with performance as well as use ability problems. I am not going into details here. The biggest problem to me is that I use Versions and IntelliJ with subversion 1.5. Now my command line client does not work anymore. This is a problem for some of the maven projects. Therefore I want to upgrade the commandline subversion client. I was surprised that there are almost no resources on the web to do this without installing other libraries. But I found a pretty easy way. Execute the following steps:

• Download the binaries from the website : http://subversion.tigris.org/getting.html#osx. I used the openCollabnet version. You do have to provided your details to download, just leave everything empty.
• Use the downloaded installer to install all the subversion binaries.
• Make sure that the new binaries are on the path before the original subversion libraries. Issue the following command in a terminal.
  export PATH=/usr/local/bin:$PATH

Those are the most important steps. If you do this in a terminal and execute the svn –version again you get the following output

svn, version 1.5.4 (r33841)
   compiled Oct 27 2008, 11:19:10
Copyright (C) 2000-2008 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).
The following repository access (RA) modules are available:
* ra_neon : Module for accessing a repository via WebDAV protocol using Neon.
  - handles 'http' scheme
  - handles 'https' scheme
* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - handles 'http' scheme
  - handles 'https' scheme

That worked, the version is now 1.5.4. If you do not want to execute this command every time you need it, you can add it to you .profile. If you need more info on how to do just that, you can find a good explenation here.

Hope this post helps you use the new and improved subversion.

Coding Horror

19 Dec 2008

Given the rapid advance of Moore's Law, when does it make sense to throw hardware at a programming problem? As a general rule, I'd say almost always.

Consider the average programmer salary here in the US:

You probably have several of these programmer guys or gals on staff. I can't speak to how much your servers may cost, or how many of them you may need. Or, maybe you don't need any -- perhaps all your code executes on your users' hardware, which is an entirely different scenario. Obviously, situations vary. But even the most rudimentary math will tell you that it'd take a massive hardware outlay to equal the yearly costs of even a modest five person programming team.

For example, I just bought two very powerful servers for Stack Overflow. Even after accounting for a third backup server and spare hard drives for the RAID arrays, my total outlay is around $5,000. These servers, compared to the ones we're on now, offer:

• roughly 50% more CPU speed
• 2 to 6 times the memory capacity
• almost twice the disk space (and it's a faster RAID 10 array)

I'd say that's a great deal. A no-brainer, even.

Incidentally, this is also why failing to outfit your (relatively) highly paid programmers with decent equipment as per the Programmer's Bill of Rights is such a colossal mistake. If a one-time investment of $4,000 on each programmer makes them merely 5% more productive, you'll break even after the first year. Every year after that you've made a profit. Also, having programmers who believe that their employers actually give a damn about them is probably a good business strategy for companies that actually want to be around five or ten years from now.

Clearly, hardware is cheap, and programmers are expensive. Whenever you're provided an opportunity to leverage that imbalance, it would be incredibly foolish not to.

Despite the enduring wonder of the yearly parade of newer, better hardware, we'd also do well to remember my all time favorite graph from Programming Pearls:

Everything is fast for small n. When n gets large, that's when things start to go sideways. The above graph of an ancient Trash-80 clobbering a semi-modern DEC Alpha is a sobering reminder that the fastest hardware in the world can't save you from bad code. More specifically, poorly chosen data structures or algorithms.

It won't hurt to run badly written code on the fastest possible boxes you can throw at it, of course. But if you want tangible performance improvements, you'll often have to buckle down and optimize the code, too. Patrick Smacchia's lessons learned from a real-world focus on performance is a great case study in optimization.

Patrick was able to improve nDepend analysis performance fourfold, and cut memory consumption in half. As predicted, most of this improvement was algorithmic in nature, but at least half of the overall improvement came from a variety of different optimization techniques. Patrick likens this to his early days writing demo scene code on the Commodore Amiga:

In the early 90s, I participated in the Amiga demo scene. It's a great illustration of the idea that there is always room for better performance. Every demo ran on the same hardware. It was the perfect incentive for demo developers to produce more and more optimized code. For several years, every month some record was beaten: the number of 3D polygons, the number of sprites, or the number of dots displayed simultaneously at the rate of 50 frames per second. Over a period of a few years, the performance factor obtained was around 50x! Imagine what it means to perform a computation in one second that originally took an entire minute. This massive gain was the result of both better algorithms (with many pre-computations and delegations to sub-chips) and micro-optimizations at assembly language level (better use of the chip registers, better use of the set of instructions).

Patrick achieved outstanding results, but let's be clear: optimizing your code is hard. And sometimes, dangerous. It is not something you undertake lightly, and you'd certainly want your most skilled programmers working on it. To put it in perspective, let's dredge up a few classic quotes.

Rules of Optimization:
Rule 1: Don't do it.
Rule 2 (for experts only): Don't do it yet.
-- M.A. Jackson

"More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity."
-- W.A. Wulf

Programmers have a tendency to get lost in the details of optimizing for the sake of optimization, as I've noted before in Why Aren't My Optimizations Optimizing? and Micro-Optimization and Meatballs. If you're not extremely careful, you could end up spending a lot of very expensive development time with very little to show for it. Or, worse, you'll find yourself facing a slew of new, even more subtle bugs in your codebase.

That's why I recommend the following approach:

1. Throw cheap, faster hardware at the performance problem.
2. If the application now meets your performance goals, stop.
3. Benchmark your code to identify specifically where the performance problems are.
4. Analyze and optimize the areas that you identified in the previous step.
5. If the application now meets your performance goals, stop.
6. Go to step 1.

Always try to spend your way out of a performance problem first by throwing faster hardware at it. It'll often be a quicker and cheaper way to resolve immediate performance issues than attempting to code your way out of it. Longer term, of course, you'll do both. You'll eventually be forced to revisit those deeper algorithmic concerns and design issues with your code that prevent the application from running faster. And the advantage of doing this on new hardware is that you'll look like an even bigger hero when you deliver the double whammy of optimized code running on speedier hardware.

But until the day that Moore's Law completely gives out on us, one thing's for sure: hardware is cheap -- and programmers are expensive.

SpringSource Team Blog

17 Dec 2008

Today we announced the public availability of the first milestone release of the newest member of the open source Spring project portfolio, Spring BlazeDS Integration. This project's purpose is to make it easier to build Spring-powered Rich Internet Applications using Adobe Flex as the front-end client. It aims to achieve this purpose by [...]

GroovyBlogs.org

16 Dec 2008

HTTPBuilder is basically a wrapper for Apache's HttpClient , with some Groovy syntactical sugar thrown on top. The request/response model is also inspired partially by Prototype.js Ajax.Request . In s...

GigaOM

16 Dec 2008

Until now, Google’s App Engine has been a great playground for coders: Everyone gets a daily quota of computing resources to play with. But without understanding how pricing will work when you go beyond those quotas, it’s been harder to understand business models built on it. Today, however, Google has shown us how the pricing model will work.

The approach is similar to AdWords: You set a daily budget, and when your application exceeds its free quota for that day, additional capacity comes out of the budget. The cost is split across processing, storage and bandwidth.

It’s easy for Google to offer a free daily quota because App Engine isn’t built around virtual machines the way competitors like Amazon’s EC2 are: You’re not paying by machine, because there aren’t any machines. Competition from Google’s free quota model may encourage other clouds such as Amazon to introduce free cloud computing quotas for small-traffic applications; meanwhile, Google is carefully launching an ecosystem for developers to build and sell their cloud-based software.

Concentric Hosted IT Solutions and Web Hosting
Click here to save cost on your IT demands