Environment needs of a standard web project

This blog entry covers my thoughts about the environment needs of the projects I have worked on in the last 5 years.

Summary:

The Development environment runs whatever the developer wants.
The Support environment provides version control and continues integration.
The Test environment runs the latest successful build deployed.
The Demo environment runs the latest stable build deployed.
The Acceptance environment runs either the same build as the production environment or the build that needs to be accepted.
The Production environment has the latest accepted build deployed.

The Development environment:

Development needs to be done. The one time i saw development being done directly to a server, i made sure it was the last time. Development needs to be done on the machine of the developer. In general I would suggest the development environment has no external dependencies what so ever, but in practice I have seen plenty projects where this is just to hard to be effective. At least try to minimize the external dependencies.

The Support environment:

Do not start a project without having a version control system nor continuous integration solution. I will not even explain why... if you disagree please go away. The version control software doesn't need to be on the same (virtual)machine as the CI software.

The Test environment:

After building a feature, committing it to version control, and check whether the build is successful, the feature should be tested. For feature testing we deploy the latest successful build to the Test environment. On this environment our testers perform their test.

The Demo environment:

From time to time our clients want to demonstrate what has been developed. For demonstrations I would like to introduce the Demo environment. We have tried to demo on the acceptance environment, but usually there is a need to demonstrate a newer build then the one that's currently in production. I found that the acceptance environment is crucial for good software development, and it's state should always be as described above.

The Production and Acceptance environments:

Nobody will disagree if i say we need a Production environment. Some of you will already think that you must have an Acceptance environment also, and i agree.

When we want to deploy a specific build to the production environment, we always deploy it to the acceptance environment first. Our acceptance environments are as similar to the production environment as we can make them. Deploying a build on this environment will serve as a deployment test, but more importantly, our client will have to do a complete acceptance test before we will deploy the build on the production environment.

If any bugs are found on the production environment, I want to be able to reproduce the problem on the acceptance environment, with for example higher logging settings. In our setups this usually means restarting tomcat, which i do not want to do on the production environment.

Did I get it wrong?

I think you will not disagree with what you just read. In case you do think i got it wrong, please educate me by posting a comment.

Retrospective as a process tool

Yesterday I was part of a retrospective which trobbelled me in many ways. I consider the retrospective the most important meeting for any project. The main goal of any retrospective should be to ensure continuous improvement. In most cases this is done by encouraging all participants to share their own interpretation of what went wrong, and suggest solutions.

This is really hard to accomplish. Can one really tell their own opinion as they see it? Should there be a (time-)limit to the time one presents their opinion? Should we allow a discussion about a person's personal opinion? What should we do with the ventilated opinions? How do we translate opinions into continuous improvement? How do we ensure continuous improvement?

Reading about retrospective thought me that any good retrospective should have a list of actions. Is it really that easy? Can we focus on achieving this list? Is the list the goal of the retrospective?

As i mentioned before, the goal is to ensure continuous improvement. What is the time base for improvement? We should be improving compared to when? The last sprint?

This particular retrospective was a very heavy retrospective mainly because of tree reasons;
* The planned release did not happen
* The sprint was extended 3 times, as was the release date
* The sprint before this sprint did not have a retrospective

The facilitators of this particular retrospective did a good job, the action list was quite impressive and i personally liked the outcome. What i didn't like was the pushing for a very short meeting. My feedback on this meeting can be summarized by the following sentence:

Individuals and interactions over processes and tools

This retrospective felt like a tool for the process, and imo didn't focus enough on individuals and interactions between them.

If you ever facilitate a retrospective, make sure you think about the questions in this blogpost and try to answer them in a way you focus on individuals and interactions between them.

Envers on Devoxx

History or Audit logs on Hibernate

You'll probably once wanted to create history for your information. If you are intrested in doing just this, you should look at Envers.

Why should i look at Envers?
1) Non invasive (no need to change your current code)
2) Annotation driven
3) will be included in the following Hibernate release

Yesterday Adam Warski gave a very nice presention about Envers on devoxx 2008. The presentation will be avialable on parleys.com in januari. I'll post the link when it's available.

For now go strait to Envers.

Bamboo Build Details Ant Task

Update
The code is now available as a google opensource project: http://code.google.com/p/bamboo-ant-tasks/

Any news about the ant task will be posted there, and/or on my personal blog: http://daniel-rijkhof.blogspot.com

Note: the compiled jar in the project repo, is compiled with java 1.6 as target.

What is it?
I build an ant task to get status information about bamboo builds. The task is build for and tested on bamboo 1.2.4. The current features include:
• Getting latest build number and latest build status (by build plan)
• Getting the status from any build (by build plan and build number)

Why is it useful
As a developer I want to have fully automated deployment on my environments. I can accomplish this with an Ant script. Our product is build on a bamboo build server. What stopped me?
• Our bamboo server cannot access our environments
• Bamboo does not provide direct links to the latest build artifacts (it does provide a redirect)
• Bamboo does not provide me with status information about the latest build

This ant task provides a way to fully automate deployment of the latest bamboo build, and add a sanity check to the automated deploy script (check if the build was successful).

Usage:

<property name="buildplan" value="your-buildplan-name-goes-here"/>
<property name="buildserver" value="http://bamboo.quintor.nl/"/>

<taskdef classpath="your-location-of/bamboo-ant-tasks.jar" classname="GetBambooBuildDetails" name="getBuildDetails"/>

<target name="shortExample">
<getBuildDetails server="${server}" buildplan="${buildplan}" buildnumberProperty="buildnumber"/>
<echo>buildnumber: ${buildnumber}</echo>
</target>

You'll have to change the red lines to make it work on your own bamboo server.

Don't forget to take a look at the example in the download package, where all parameters are used! In case there are any questions left, comment on this blog article.

Download:
You can download the task here: bamboo-ant-tasks.zip
The download contains the following:
• The source code of the ant task
• bamboo-ant-tasks.jar in the /target/dist dir, which includes only the compiled task
• Build script to build the jar
• 2 examples fullExample and shortExample, showing how to use this task (see build.xml)

The package is actually my eclipse project, so it will include .classpath and project files, and the ant.jar thats needed to compile.

Update
I fixed a bug found by Marcel; the ant task couldn't handle the status of an build that fixed previously failed build(s).

The download contains the newest code.




Update 2

Quintor upgraded there bamboo server to 2.1.3, and the ant task wasn't compatible. I'ld like to thank Arjan van der Veen to fix it, by implementing a rss feed reader.

We created a google code project for this ant task:
http://code.google.com/p/bamboo-ant-tasks/

Please leave a comment :Daniel

Announcing project Scrumbox: Opensource Scrum project management tool.

At our company we do all our projects using Scrum. Scrum is an iterative incremental process of software development commonly used with agile software development. A lot of projects use for example JIRA for their issue tracking. JIRA just isn't the best tool when you work with Scrum. Don't get me wrong, JIRA is a great tool, just not the tool for the job. There are some JIRA plugins to generate burndown charts or scrumboards, they make life al little easyer. You can do lots of tricks like using versions for sprints or backlogs. But what about Epics or Themes?
There are some other great software products out there build entirely around scrum, but they all are way to expensive (500$ per user/per year whoa!).

So we decided to build are own tool, totally open source. This tool is going to be build in java and flex. The focus is to make the GUI as fast and intuitive as possible. Making use of flex's capabilities like dragging and dropping issues from the project backlog to a sprint backlog, or prioritizing the backlog by drag and drop. Also changing the state of an issue should be as simple as looking at the scrumboard and dragging and dropping it from the 'in progress' column to the 'resolved' column.

So definitly keep and eye on http://scrumbox.googlecode.com and this blog for updates about the project. Anybody interesting in committing to this project should contact us..

Project architecture:
The backend will be build in java. Using Spring as IoC container and hibernate as ORM layer. As database we will use MySQL. Everything will be programmed TDD style using JUnit4 and EasyMock. We add a security layer with Spring security and build our project with ant.

The frontend will be build in flex. Using Prana as IoC container. Later there might also be an AIR version (for offline work). We also built this project with ant.

written by Marcel Panse

Agile Development 'nadelen'

written by Daniel Rijkhof

This blog entry is a reply on an article that’s in Dutch, this is why the blog entry is in Dutch to.

Vandaag las ik een artikel over Agile Development op de site van computable.

In dit artikel worden een aantal nadelen van AD genoemd, waar ik het niet mee eens ben. De nadelen die ik op de schop doe zijn:

• In AD fungeren ontwikkelaars tegelijkertijd als informatie-analist, software-architect en tester
• AD is alleen geschikt voor senior ontwikkelaars
• Ontwikkelaars hebben te weinig skills
• Het werk van de bureaucraten, moet worden gedaan door de specialisten.
• Stakeholders worden niet tevoren gedwongen om hun eisen en verwachtingen te formuleren.
• Planning en budget zijn moeilijk te bepalen
• Belangrijke beslissingen op ad-hoc basis
• Slecht doordachte architectuur en modellen, risico op spaghetti-code
• Veel code weggegooid

De eerste drie
De eerste drie nadelen komen voort uit de gedachte dat een team alleen ontwikkelaars mag bevatten van het zelfde niveau:

• In AD fungeren ontwikkelaars tegelijkertijd als informatie-analist, software-architect en tester
• AD is alleen geschikt voor senior ontwikkelaars
• Ontwikkelaars hebben te weinig skills

Is dit werkelijk een eigenschap van AD?

Het Agile Manifesto is gebaseerd op een aantal principes. De toepasselijke principes voor deze nadelen zijn:

• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

De principes achter het Agile Manifesto zeggen niets over het samenstellen van een team! Mag je dan zelf bepalen wie er in je team komen? Zou het zo kunnen zijn dat het team zelf weet waar de gaten vallen binnen het team? Zouden de bovenstaande 3 principes, een basis vormen om daar achter te komen, en het probleem op te lossen? Ik denk het wel.

Er is geen verplichting om een team te nemen waarin elk persoon dezelfde hoge kwaliteiten heeft. Er is geen regel in AD die zegt dat dit niet mag. Kan je een team samenstellen met requirements specialisten, architect(s), developer(s) en tester(s)? AD laat je hier vrij in. Als het werkt voor jou, doe het dan.

De gedachte dat je alleen ontwikkelaars in je team hebt zitten, is niet correct, en daardoor vervallen deze drie nadelen.

Meer ‘nadelen’:
• Het werk van de bureaucraten, moet worden gedaan door de specialisten.
• Stakeholders worden niet tevoren gedwongen om hun eisen en verwachtingen te formuleren.

Deze nadelen komen voort uit het idee, dat AD alles beschrijft dat in een bedrijf moet gebeuren om succesvol software te ontwikkelen.

Het volgende agile principe geeft aan dat het de bedoeling is, zo snel en vaak mogelijk waardevolle software op te leveren:
• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Stel je voor dat stakeholders niet vertellen wat hun eisen en verwachtingen zijn. Wat moet er nu gedaan worden? Het is een AD noodzaak om te weten wat de stakeholders willen. Vervolgens willen we ook nog weten welke waarde de stakeholders aan hun eigen eisen toekennen, zodat ze zo snel mogelijk werkelijke waarde opgeleverd krijgen. Het nadeel dat ze niet gedwongen worden om hun eisen te formuleren, verwerp ik.

Wie er achter de stakeholders aan gaat om te zorgen dat hun eisen bekent worden, wordt niet gespecificeerd door AD. Zou het een bureaucraat mogen heten? Niets in AD verwerpt dit. Zou het iemand in het team kunnen zijn? Ook dit wordt niet verworpen door AD.

Zou deze ‘bureaucraat’ een specialist zijn op exact dit gebied? Zou dat toch eens mooi zijn!

Planning en budget zijn moeilijk te bepalen

Is dit nadelen toe te kennen aan AD?

Mijn ervaring met non-AD projecten, is dat de planning nooit klopte, en het budget niet te bepalen was. De bedoeling van AD is, om zo snel en vaak mogelijk waardevolle software op te leveren. We kunnen dus kortere planningen maken, en budget voor vrij maken. Het is dan ook makelijker om een correcte planning te maken in vergelijking met non-AD projecten. Het los krijgen van budget is vast makelijker als jou planningen kloppen. Dit kan geen nadeel van AD zijn.

Belangrijke beslissingen op ad-hoc basis

Belangrijke beslissingen, zullen altijd gebeuren, en vrijwel nooit aangekondigd. Ik heb nog nooit gehoord dat een bedrijf een geweldige kans op een vergroting van de winst, heeft uitgesteld omdat er nog een planning vast stond.
Flexibiliteit van het ontwikkel proces is dan ook gewenst, zodat er niet te veel waarde verloren gaat.

Het volgende agile principe is de basis voor flexibiliteit:
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

In non-AD projecten, komt het voor dat er nog geen waarde opgeleverd is, en ook niet zal gebeuren in een korte tijd. In AD projecten word er elke iteratie waarde opgeleverd. Het verlies van waarde wordt geminimaliseerd door een korte iteratie tijd. Er is ook nog de optie, om de iteratie af te maken, waardoor er geen waarde verloren gaat, natuurlijk ten koste van een beetje tijd.

Het volgende agile principe maakt mijn argument af:
• Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Omgaan met ad-hoc beslissingen, gaat beter met AD projecten vergeleken met non-AD. Dit nadeel is wordt door AD zelfs niet gezien als nadeel, maar als een eigenschap van de business.

Slecht doordachte architectuur en modellen, risico op spaghetti-code

Het volgende agile principe zegt voldoende:
• Continuous attention to technical excellence and good design enhances agility.

In het geval dat dit nadeel toch voorkomt, dient het team zichzelf aan te passen. Zie weer het volgende agile principe:
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Veel code weggegooid

Het zou kunnen voorkomen dat er code is geschreven dat vervangen moet worden, of zelfs niet meer in gebruik is, en weg gehaald dient te worden. Als deze code is geschreven in een vorige AD iteratie, dan was deze code waardevol. Zie weer het volgende agile principe:
• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

In dit geval was de code ooit waardevol, en is het nodig om deze code aan te passen om de software nog waardevoller te maken. Waar is het nadeel? Ik zie het niet.


Is AD perfect?

Niets is perfect. Wanneer ik AD vergelijk met de oudere manieren van software ontwikkeling, concludeer ik dat AD geweldig is, een stap richting perfectie.

Code Complexity

written by Daniel Rijkhof

About

This blog entry is quick read summary about code complexity. It gives a definition of code complexity and several signs how to notice code complexity. It also provides a brief description how to measure code complexity and suggests a way to keep it low.
Most of the concepts are taken from Structure 101 help section.

This blog entry is based upon my code complexity presentation for www.greetz.com.

Definition of Code Complexity

Reg. Charney defines Code Complexity as:
the amount of effort needed to understand and modify the code correctly.
see http://www.linuxjournal.com/article/8035.

Excessive complexity: When is a code-base too complex?

Signs of Logical complexity:

• part of the design is no longer fit for its purpose
• unclear purpose
• adding new features becomes harder than it should be

Signs of Structural complexity:

• the code is hard to understand
• changes have unexpected side effects

The need to know about code complexity

Keeping track of code complexity helps to:

• Predict (future) feature implementation times
• prioritize areas of the code for structural refactoring

What happens if you don't manage complexity?
It will cost!

Manage complexity like debt
Howard Cunningham 'Mr. Wiki' compared complexity with debt:
"Manage complexity like debt"
skipping designs = borrowing money
maintenance headaches = interest payments
refactoring = repaying debt
cannot refactor = cash-flow insolvency
creating engineering policies = devising financial policies


Andy Hunt extends the analogy concisely:
"Just like real debt, it doesn't take much to get to the point where you can never pay it back, where you have so many problems you can never go back and address them."


What can we do? Measure Complexity!

Although we cannot specify the amount of effort needed. We can calculate a complexity index.

This index enables us to:

• Compare projects
• Monitor trends (getting more or less complex)
• Discover which items at what level are causing the complexity.

Measuring Code Complexity

Measure the complexity at every level of design breakout, method, class, package and
high-level package (High level packages contain only other high-level packages or leaf packages).

Measuring occurs by counting items and dependencies it immediately contains.

Method complexity is usually calculated by cyclomatic complexity.

About Cyclomatic complexity

What is Cyclomatic complexity?
A measurement for the complexity of a peace of source code.
What does it measure?
The number of linearly independent paths through a peace source code.

Cyclomatic Complexity @ wikipedia

Calculating Cyclomatic Complexity

To calculate cyclomatic complexity build a graph that describes the control flow of the program. The nodes of the graph correspond to the commands of a program. A directed edge connects two nodes if the second command might be executed immediately after the first command.

The number of linearly independent paths through the first method is 3, and the second is 2.

In Structure 101, class, package and high-level package complexity is calculated by counting items and dependencies they immediately contain (screenshot from Structure 101 help):



Design Tangles
Design tangles are cyclic, package-level dependencies. Design Tangle Metric quantifies the amount of cyclic dependencies.



Refactoring
Defenition of Refactoring:
improving the design of existing code.

Always Refactor when adding new features to you code.

Quote Chris Chedgey:
"You use personality, charisma, leadership and/or donuts to convince your team that henceforth, they will not add any more complexity debt to the code base. Now watch what happens..."

I suggest reading Chris blog item: http://chris.headwaysoftware.com/2006/10/reducing_comple.html.

The need to Rebuild
In some cases redesign and rebuild is faster then refactoring. The complexity index of the tipping point cannot be determined.

Future Vision
Chris Chedgey says: "keep a lid on it"
I agree!

• Monitor complexity (Structure 101)
• Keep complexity low (stay away from the need to rebuild)

Resources

Structure 101 Help see http://www.structure101.com
Cyclomatic Complexity @ wikipedia

Chris Chedgey blog entries:
http://chris.headwaysoftware.com/2006/09/complexity_over.html
http://chris.headwaysoftware.com/2006/10/reducing_comple.html


http://www.linuxjournal.com/article/8035