Technical Debt - Recently Updated Pages

Parallel Development

JonKern — Fri, 08 Apr 2011 10:48:27 CDT

Parallel Development allows teams to work on various aspects of the same product source code base. As you near the end of feature development for the current release, freeze the code, and begin more "formal" testing, you will find some developers with time on their hands. For this reason, it makes sense to apply this ever-growing group of developers to work on the next release. Therefore, we create a new branch:

Since we do not want to risk making changes to the "release" code we are about to deliver,
Since we do not want to simply have developers working in their own workspaces until the current release is made and they can start "checking in" code again, and
Since we want to make it easy for a developer to halt their new development and fix a bug (requiring restoring their system to match the current release code base, and reverting back again once their bug fix is approved and checked in).

Hence, we end up with a high-level view of the parallel development process as follows:

Note, for a moment, the overlapping nature of Figure 2. Except for the fact that you are likely resource-limited, development never stops. Market research, client feedback, and planning the next version do not stop either. In addition, a release comes every X months, like clockwork. It is all very rhythmic and should have a non-chaotic flow to the process. The concept of a "code freeze" is just in terms of the code base for the current release about to be shipped (to allow stabilization of the source). The rest of the developers keep right on working on the main trunk code base. Essentially, this large "outer loop" surrounding release-style development has an overlapping region. As the code is frozen on Release N and formal testing commences, new development starts on Release N+1 by part of the staff. To keep the code safe, this is when branching is done for this style of development. This requires all folks involved with developing a release to be very aware of the rhythmic nature of this process. Marketing, product champions, field engineers, documentation, trainers, management, etc., all have to be working ahead of the team in terms of planning the next release. Depending on your overall release schedule duration, you need to be aware of potential bottlenecks with some of the non-development-related issues. For example, marketing would be hard-pressed to plan for the next release at the same time as getting the current release out the door.

Triggering a Test Cycle

If the software world were perfect, all you would need to do is to create features, have them "ensured" by automated unit and acceptance tests, do some manual testing for those areas that require a human touch, and shorten the "Formal Testing" cycle down to mere days. Obviously, this is rarely the case in complex software. But a worthy goal to strive for, nonetheless. You should attempt to shrink the amount of time devoted to testing under code freeze to as short as possible. This is largely a balancing act between improving the entire development process, improving quality of code, automating as much testing as possible, making manual testing as efficient as possible, and doing the "right amount" of testing to deliver a quality product. Over time, a team should be able to improve and reach an optimal balance. In addition, you should not turn out QA builds one after the other every time a bug is found. Every time QA gets a new build, the test cycle restarts, costing $$ and time (and good will) of all the testers. You want to strike a balance here as well.

Branching

Though there are a few different ways to control software releases, the most popular method - and the one supported by most tools - is the simple branching technique. This allows teams to undertake parallel development. That is, you can have multiple project teams working in parallel on the same (in a virtual sense) code base. Of course, "simple branching" is relative. Branching can lead to high degrees of complexity and confusion if not done properly. Branching can even come about due to different reasons:

Product enhancements: patches, releases, etc.
Platform or OS: maybe you need to support a new internationalization initiative, or a new hardware platform
Roles: maybe you branch for the QA team to do their thing
Phases: you have a promotion scheme whereby the code goes from development, to QA, to release, to maintenance

The basic problem facing development is that different developers need different bodies of the "same" code at different times for different purposes. Confused? Me too. Lets assume we have a product that looks like this:

WidgetFactory
- v1.0
  - v1.1
- v2.0
  - v2.01
  - v2.1
  - v2.5
- v3.0

You need to be able to have developers create workspaces that reflect the proper body of code for their current tasks (be it development of features in the main trunk, bug fixing for v2.1, or exploratory development for a new architectural approach concept).

Some specific instructions using Eclipse found here.

The Main Trunk

Obviously, we have to start with a hunk of source code for a given application. The code from which we continue to earn a living must carry on into the future. All releases share the main trunk. In theory, you propagate applicable bug fixes for a given release (branch) back to the main trunk to the benefit of all future releases. (Merging assumes that the bug fix still makes sense in the context of the current state of the code.) An even greater theory is that you could build the application from the main trunk and that the automated tests pass with flying colors. This is a style of development that some people prefer. That is, there is an edict that the main trunk should reflect a useful state of running software (sometimes at a given point in time: daily, every Friday, etc.). The Main Trunk represents the latest and greatest. From this trunk you create the branches - hence the tree analogy. If you branch too much, you get a shrub. Avoid shrubbery.

Release Branches

You typically create release branches to establish the code base that you will eventually deliver to the customer. When the current development branch nears Code Freeze, you create the branch so that you can control changes more closely. What underlies the desire to create a new branch is simple:

You do not want "harmful" changes to the code as you are squashing bugs, and readying for a stable release/shippable build.
You want other developers to be able to continue on the next version underway. That is, they should be able to continue to add new features, undertake major refactorings, and generally do other tasks that could be very disruptive to the code base.

At the stage where a Code Freeze has been enacted, all developers know to not "break" the build. Only bug fixes that have been approved get done (that is, they are demonstrated to truly fix the problem and not wreak havoc on other parts of the system). Developers may be limited in their capability to check-in to the code base. Maybe the chief programmer must approve the changes, after which you can check in the code. Normally, you blast out a stern email message to the team, reminding them to increase their awareness to protect the release code from instability. This means things like:

Only change what is minimally necessary to fix a bug.
Be very cautious about changes in files/methods that others have many dependencies on. If unavoidable, alert QA that more testing in the affected area(s) may be in order.
Though you may spot some other neighboring code that needs "improvements" while you are making a bug fix, resist the temptation to make a "little, harmless fix." (Do it in the new branch, or log it in bug/task tracker for doing in the future.)
The less code you touch, the less likely you will create unintended consequences.

Depending on your style of development, you may do one of the following:

Preferred: Main trunk is your development stream. You create a new branch N for the release. New development continues in main trunk, and you relegate critical release development to the new branch.
You are working in the existing release N branch. You create a Release N+1 branch. New development takes place in N+1, careful bug fixing takes place in the N branch.

With multiple branches and a trunk, you have the potential for a single file to have different revisions that have diverged in content. The section on "Merging" will address what to do.

Tags and Labels

Many times, your version control software gives you the opportunity to apply a special name or symbol to a bunch of files to represent a point in time. For example, when you have finally decided to release the software to your clients, you should label all current files in the branch to be identifiable as the files that went into this official release. This gives you the ability to reliably retrieve the precise body of source code and related files that went into a shipping/deployed version. This is especially useful when it comes time to fix a bug in this version, and deliver a new "patch" or update your delivery files. When you have files tagged/labeled/marked, you can request all files with a given tag. This is kind of like turning the repository upside down and shaking it: only those files that have the requested tag will "fall out." Newer versions of files on the trunk will not be brought forth. In CVS, for example, you can create a branch based on a given set of tagged files. For example, you can create a branch labeled "ver 1 0 patch" based on files tagged as "ver 1 0." (Note: I use labels that indicate major and minor version numbers so that it is clear what I am trying to demonstrate. However, you can use tags that are code names or numeric values as well.)

Merging

The concept of merging is to take changes in one branch, and update another branch and/or the main trunk. The easiest way to visualize this is to consider a single file in your system that goes through some changes. Suppose, for a moment, that one of your customers actually found a bug – as rare as that might seem to you. You have two choices:

Fix it in the current new development branch/main trunk, deliver in the next scheduled cycle.
Make a bug fix/patch release.
Ignore the bug. :0
I know, I said two choices but gave you three. The third choice (a viable one, I might add) requires no knowledge of releases and branching, so we will stick to the first two.

Lets assume you decide to make a patch release, so you create a new branch based on the release branch. By checking out from the new "patch" branch, the developer is able to quickly get the necessary files for the release in which the offending bug exists. The developer then makes a fix to the source code, does some testing, and checks the code back into the "patch" branch. A patch build should be simple (because you also tagged all supporting files; e.g., build scripts, installer scripts, documentation, tests). You then might go through a full or abbreviated QA cycle to test the change(s). Merging Changes Across Branches:

Note how the Figure also shows changes made in the branches being merged back into the main trunk. Of course, this may or may not be done. Some times, a fix in an older version of the file may be unnecessary due to subsequent changes made to the file or the program architecture.

Branch Development

JonKern — Fri, 08 Apr 2011 10:46:29 CDT

As described in Parallel Development, we use branches in SVN to snapshot the code base that reflects the released version of the project. This allows new development to carry on in the TRUNK; while still allowing maintenance to the released version to occur in the branch.
As we find new bugs, we will decide if they need to be fixed in the branch. Those items that are deemed critical will be fixed in a special milestone. For example, current release is BETA-2, for hot fixes, we would create "Beta-2-1" as a new milestone/branch.
Branch Development Process
In short, you make your eclipse workspace be identical to BETA2.1 Branch. Do your fixes, check them into the branch. Then you decide if you want to promote the fix to the TRUNK or not (using the version control MERGE feature – include 'dry run' if needed). In general, most of the fixes also belong in the trunk.

	Benefits of SVN Merge Yes, you could copy the good file off to the side (or in your paste buffer), and then paste it into the branch file and commit. However, using the SVN merge feature is better because that sets the svn:mergeinfo metadata so that you can track the true history of a file. Copy/pasting does not do this.

Benefits of SVN Merge
Yes, you could copy the good file off to the side (or in your paste buffer), and then paste it into the branch file and commit. However, using the SVN merge feature is better because that sets the svn:mergeinfo metadata so that you can track the true history of a file. Copy/pasting does not do this.

Moving forward, developers may be required to change code in different branches other than the main /TRUNK/. If you need to do this, follow the process below to switch your project to the appropriate branch.

Project, Folder, File?
If you need to do extensive development in a branch that relies on deep compiles of code, do the whole project. Just a few stand-alone changes in a well-isolated area? Do the Folder. A simple tweak of a single file that is isolated? Just do the file...

Setting Up Your Workspace

Step 1 - Switch to Branch
Right click your project:

and go to Team > Switch:

Step 2 - Select Branch
Click Browse to search the BRANCHES in the repository:

Here we have selected BETA2.1 Branch:
Click OK

Step 3 - Verify Switch
You should see the branch name in your project:

Making Changes

Complete Your Work
Commit to Branch
Compare to Trunk
See Diff
Your Changed File	Revision 9875
Log	Add revision number/file in Jira comment -- especially important when we look to bring fixes into a branch.

Merging From Branch To Trunk

commit branch	commit your local changes to branch, including unversioned files
determine branch creation	go to your branch within SVN repository browser, right click, show log and select "stop on copy". scroll to the end (page if needed) to see the revision when the branch was copied
switch to TRUNK	Switch your working copy from the branch back to TRUNK
verify you are in TRUNK	verify your working copy says you are in trunk
merge BRANCH into working copy	merge a range of revisions in the branch into the working copy. the merge dialog FROM should be the BRANCH url and the revision when the branch was created, so you don't go back to the start of time. the merge dialog TO is the same BRANCH url and would be the head revision, but you can specify an earlier revision in the branch.
commit working to TRUNK	manually fix any merge conflicts, test locally, then commit your working to TRUNK

Merging From Trunk to Branch

This is most generally used for a file or two, though you could do it for an entire folder. Basically move to the Branch, then go to the file(s) you changed, and merge in the TRUNK changes.

NOTE: you don't have to merge just from the TRUNK, these instructions work if you want to merge from one branch to another. It is simply that we often need to know how to grab a change in the TRUNK and put it into the branch that we want to make a build from.

If you only need to merge one or two files, you can actually specify individual file locations in the from/to fields (rather than the top-level directory which would merge the entire tree). Merging single files can be safer because there could be other changes in the trunk that should not be merged.

Switch to Branch	Right click your project: and go to Team > Switch:
Select Branch	Click Browse to search the BRANCHES in the repository: Here we have selected BETA2.1 Branch: Click OK
Select Merge	From the perspective of the file that needs to be updated (either via the navigator or the editor window), right-click to choose Merge: Click Browse to search the TRUNK in the repository for the file that you are replacing: Use the "Show Log" button to find the From and To revisions: Notice that I have selected revisions in both cases. Click OK
Commit the changes	Now that you have just merged from the TRUNK to the BRANCH, you need to commit these changes to the BRANCH itself.

Iterations

JonKern — Wed, 08 Sep 2010 13:35:41 CDT

Whether every two weeks, three weeks, or once a month... The development team will peel off a set of tasks from the top of the Release Plan providing the latest business priorities.

These tasks are then re-evaluated by the developers and deeper sets of subtasks are created to enable a reasonable estimate... that is, if you haven't yet jettisoned the idea of doing estimates (see Detailed Estimation Sucks).

The Iteration Process looks something like this:

Basic Iteration Process

The Dev team and the Business get together and go through the list of desired issues.

Assign an issue
Get an estimate from the developer
Continue through the list of desired issues until the resources are used up
Modify list of issues/priority as needed based on business feedback during this process

There is a balancing act between what the business hopes to have done in a given iteration, and what the Dev team says they can accomplish. So, at the outset of the planning process, the Business is armed with their set of priorities, pulling from:

A list of all available issues (not just new features!)
The initial idea of what should be in the current iteration. (This would have been decided during the initial Release Planning session -- typically used to get a rough estimate.)
The "1st in Line" and "2nd In Line" buckets
Your own prioritization scheme

NOTE: The Dev team may also have a set of issues they want to work on (from infrastructure/automation tasks, to refactoring).

There is often a tug-of-war when it comes to planning an iteration. The Business is going to want the Dev Team to have estimates for their features. The Dev team might have questions regarding details before they can give an estimate. If this is all done in real-time, folks may think it can be a bit wasteful. The Business is listening to the developers banter back and forth about how to implement a feature and what the estimate might be. The developers may feel the need to work out some subtasks before estimating "publicly."

NOTE this section discusses estimation as part of iteration planning. Give it a good long consideration before wasting precious time on detailed estimates. Read more here: Detailed Estimation Sucks.

Priority

Another common tug of war is Business Priority. Once the business knows the cost of a feature, they may alter the priority. The Business can compute the Return on Investment (at least from a relative perspective) for a given feature. Feature X that might make a few customers happy that costs 25Z might be prioritized lower than Feature Y that will make many customers happy for only 5Z. Conversely, having Feature X's few -- but happy -- customers that turn out to be the critical partners that Marketing has groomed, need to be kept happy. So Feature X may win out after all. The simple point is to provide the Business with the "cost" of development and allow the priority to be adjusted.

In the early requirements gathering phase, you will hopefully capture this sort of critical knowledge up front should it affect design or release planning. Nonetheless, you can get a Feature X at any point in the product life with, or without warning.

If it will make the planning process go more smoothly, you may need to have a "pre-meeting" to do an initial pass at explaining the next set of desired features, and then allowing the Dev team to caucus and come up with some estimates or the need for further discussions about a given complex or large feature. In general, any issue requiring more than 20 hours should be broken down into subtasks -- both for ensuring that the task is well thought out, and to show interim progress.

Story Points or Hours?	It is a common agile practice to use something called "Story Points" so that your units of measure are abstract and not so readily tied to work hours. I have not tried using story points, but conceptually, whether you estimate an issue as 10h or 2sp and you realize that you generally accomplish about 5h or 2sp per day, you end up in the same place. Mike Cohn has excellent material on all manner of Story Point discussions. Including this one that describes using hours for short-term planning.

Story Points or Hours?

It is a common agile practice to use something called "Story Points" so that your units of measure are abstract and not so readily tied to work hours.
I have not tried using story points, but conceptually, whether you estimate an issue as 10h or 2sp and you realize that you generally accomplish about 5h or 2sp per day, you end up in the same place.

Mike Cohn has excellent material on all manner of Story Point discussions. Including this one that describes using hours for short-term planning.

Developer Estimation

The estimates should be made in Ideal Hours.

Ideal Hours	What the heck does this mean? The "ideal" time is what it should take to do the issue uninterrupted. Since many of these issues requires digging and testing, you have to account for those activities in addition to pure code/sql writing. What you would not include is time to learn SQL or the tool, or inflate it by the time you spend in the Scrum, etc.

Ideal Hours

What the heck does this mean?
The "ideal" time is what it should take to do the issue uninterrupted. Since many of these issues requires digging and testing, you have to account for those activities in addition to pure code/sql writing. What you would not include is time to learn SQL or the tool, or inflate it by the time you spend in the Scrum, etc.

Allow them to take on tasks to the tune of:

Max Developer Hours = (Work hours per iteration * Ideal Factor)

For example, a two week iteration represents 80 work hours per person. The Ideal Factor can start out at a rate of 50% to account for design time, scrums, discussion with the business, unit tests, testing, foosball. As the team marches forward, that value can be changed to account for reality.

Estimating Basics

This is a skill and some would say an "art" form (I am an overly optimistic estimator).

Use hours (e.g., 5h) for your estimating (not days)
Consider the task estimate based on "Ideal Engineering Hours"
If multiple people need to work on a task, add up the two people's hours
If the estimate is larger than a couple of days, be skeptical, break it down. (Unless you are one of those rare breeds that have excellent estimating skills.)

Logging time

Use a timer – you'd be shocked!
If you help someone else on their task (>15m), you can "Log Work" to their task.
Log your time as you complete it.

Milestone Review

Following the iteration, remaining -- unfinished -- issues are either moved into the next iteration, or moved out altogether. Then, the team should take pride in demonstrating what they have built. Sometimes it is a code review and a modicum of visible output (maybe tests only). Other times it is a nice demo to the Business, QA Team, etc.

Every two weeks
Starts with a review of visible features for QA to see
Review leftover tasks and bump to next milestone

Oops, We Didn't Finish All Issues!

That's okay. Try to improve each iteration. It usually isn't a huge problem if you missed a few issues (say out of 20 or 30). If the team is doing their best, what else do you want? Why does "management" care to hit the number dead-nuts on? BTW: Some people would call those extra few stories "stretch goals" -- which I hate on so many levels.

Jon's Agile Development Tips

JonKern — Wed, 08 Sep 2010 13:14:19 CDT

NOTE:	This set of pages will take me a while to create. I am going to attempt to post my agile process tips and techniques.

NOTE:

This set of pages will take me a while to create. I am going to attempt to post my agile process tips and techniques.

Process Introduction

Over the years, I have worked with many people and learned from masters and novices alike. Here you will read about a single, simple, slightly variable process with a handful of tenets that seem to routinely work at both getting the project off on the right foot, and providing checks and balances to catch us before/when we inevitably stumble.

Each situation is unique -- from the whole business/company culture/dev team culture/problem domain perspective. However, the overall approach to meeting the project head-on is very consistent -- just the details vary.

To some readers, this process may seem like "a lot of upfront work." All I can offer you is my advice against blindly applying some process (agile or not) without using your brain and without collaborating at a mental distance greater than just the coding aspects. There is a lot more to software development than software!

Software development of the the kind I have been involved with (and continue to enjoy doing) is always "knowledge work" and never "factory work." The projects tend to be 6 months to over a year, developing business-critical products and the team/process/culture to go with them. The more you understand about the company's business drivers (often a "level up" from your immediate project's needs), the more that you can provide innovative and explosive solutions to provide business value.

The process described herein is for just such type of projects.

Background

A conundrum of Agile Development to many laypeople is that we trade in seemingly precise process (sequential steps, roles, Gantt charts, lots of non-software-based evidence of activity) in the name of getting to production faster with higher quality -- yet mysteriously with seemingly less process.

The reaction of naive development teams is to "run wild" and do a poor job of delivering a professional solution and claim it is "agile." This gives agile a bad name -- at least to those who do not understand that software development is mostly about the quality and experience of the people and not the name of the process they followed.

Agile requires a keen understanding of sometimes subtle relationships between cause and effect in software development. This is true in many facets of software development:

people's interactions
frequent, tangible, working results
involving the customer more often
being able to embrace change

While some agile techniques -- if blindly adhered to by a newbie team -- can help provide clear feedback, there is still a challenge in our industry to identify the shades of quality. For example, there may be three shades of an architecture used to solve the same problem.

All three approaches work.
On the surface, the user would not be able to see any difference.
All three approaches require about the same time to arrive at.

A newbie developer came up with the lowest ranking architecture. A newbie might not be able to decipher the differences between the three in terms of the near- or long-term benefit. Or, they may tacitly understand what you might point out to them, but not really deeply embrace the reasoning because they just haven't got the amount of experience to appreciate the differences. A developer with more experience might recognize why the middle architecture is better than the worst, but may not be able to come up with the best architecture due to lack of experience (or skill). However, a skilled senior architect will understand the ramifications -- both near- and long-term -- behind choosing the best architecture. You just can't deny the value of working with as highly-skilled people as you can find for every aspect of the team. Ten highly skilled people with good discipline and process can do the work of 100 mediocre developers and do a better quality job on top of it. Sounds controversial? Hardly -- if you have good experience in our industry for any length of time.

Success Factors

Some mantras that we choose to work by:

Always be visible
- We work in a completely transparent way
- We are one, open team, from developer to stakeholder to gold owner
- Knowledge transfer is on an as-frequent basis as needed
- We tend to document in a natural, cost-effective manner
Business vision is critical & necessary driver
- Development is to support marketing vision and business needs
- The closer we work with actual clients, the better the results
- Technology and business works together in a give and take
- We work our magic to make the business dreams a reality
Close the "gaps" to reduce risks:
- ...of building the wrong thing by demonstrating working software in a matter of weeks, not months or years
- ...of being non-performant by testing early and continuously
- ...of being brittle by ensuring our code has appropriate level of tests
- ...of deployment by duplicating or simulating the environment(s)

General Philosophy

Our job is to do the best we can at shepherding the company's assets and resources, and the folks being assigned to the software product development.

The basic process:

Building features in an iterative manner
Having a range of tools to ensure efficiency of development and testing
Evaluation by users after each iteration
Take the reviews/critiques back to the product development team
Be prepared to throw some things away and redesign/rebuild
Repeat as needed

This will lead to a quality product.

Some typical rules of thumb:

Do not do any one aspect to excess, keep things reasonably in sync.
The only truth: running (quality) software demonstrating feature completion.
Client specifies features, dev team dictates cost and schedule.
It is far better to collaborate than to engage in antagonistic "detailed contracts"
Strive to be in a state where the software can be built and running at a moment's notice (best to have multiple environments)
Tons of detailed documentation and requirements can be wasteful; keeping everything at the conversation level can be short-sighted
Lots of fine-grained project plans for 5 years out is also wasteful

	Do Not Get Blocked You must not allow yourself to be blocked. That is, "stuck" from achieving success due to external forces. Bring this to the team's attention immediately as part of an email to the project list, or in the daily meeting or on IRC.

Do Not Get Blocked
You must not allow yourself to be blocked. That is, "stuck" from achieving success due to external forces. Bring this to the team's attention immediately as part of an email to the project list, or in the daily meeting or on IRC.

Consistency is a Quality Unto Itself

JonKern — Wed, 08 Sep 2010 11:44:53 CDT

Background

Whether we are talking about code conventions, UI, class/table names, or even sample data, consistency is a key quality.

So much so, that I would rather have a mediocre solution (say for example in the area of data management) across 5 areas of the application than 5 different solutions. Why? Because when you have consistency, folks can recognize each item as being of the same general nature. Also, it is easier to improve and refactor a consistent solution in 5 places than it is to find and fix 5 different approaches.

The Current Solution

Here is a really dumb, yet illustrative, situation. In our project, we have 3 very distinct environments:

DEV
TEST
PROD

Pretty obvious by their names alone, that that these environments serve different purposes. An important part of the application is role-based behavior of the logged-in user. When the user logs in, a set of roles/privileges is retrieved from the User database. To make it easy for developers to work and test as different users, we set up a standard set of user accounts. Standard across the environments, or so I thought.

USER NAME	ROLE
mr_author	Author
mr_reader	reader
mr_admin	Admin
mr_csr	Support

Problems

It all started out by claims of bugs being encountered. Sometimes in PROD... sometimes in TEST... and by different reporters using different logins and reporters using the same logins. Oddly enough, the bugs were against previously-tested -- and passed -- features or bug fixes. It got to the point of absurdity. Could our development team be that bad? I know many of the team members are new to the rigorous development practices associated with a public software product. But the reports were contradictory. The bug reports didn't fly with the level of maturity that I know the team had gotten to by this time in our project.

So, I began to look into whether gravity was still practicing it's basic vertical stance. Turns out it was, of course. The real culprit ended up being inconsistency in how the test logins were set up.

USER NAME	ROLE-TEST	EMAIL-TEST	ROLE-PROD	EMAIL-PROD
mr_author	Author	MR_AUTHOR mr_author@domn.com	Author	MR AUTHOR mr_author@domn.com
mr_reader	Reader	MR_READER mr_reader@domn.com	Reader, Author	MR_READER mr_reader@domn.com
mr_admin	Admin	MR_ADMIN mr_admin@domn.com	Admin, Author	MRADMIN mr_admin@domn.com
mr_csr	Support	MRCUSTSV mr_cs@domn2.com	Support, Author	MRHDESK mr_helpdesk@domn2.com

So, somehow, the users were set up differently on the different environments.

Different user names
Different emails
Different roles

What was even worse was the emails of these test logins was also inconsistent. This simply exacerbated the situation when attempting to easily find all of the test login users -- so I could check the assigned roles. When I initially specified setting up these special logins, I kept it very simple and clear -- and consistent. The user name should be the basic role name itself, the emails were ostensibly mr_role@domain.com. Unfortunately, when the users were added to the second and third environments, the same approach was not followed, as you can see in the table above. For example, the mr_csr login for the support role was entered as mr_csr@domn.com on TEST, but mr_helpdesk@domn2.com on PROD. Thus, making the SQL query that was attempting to find the test login users not work properly. Maddening, I tell you!

The Cost

It might seem trivial, these differences. Does a user name really matter if it had an underscore or not? Does a mildly different email name and email domain truly matter?
YES! When you are trying to find the data across multiple environments and databases, trivial and subtle differences make the quest maddeningly challenging -- and for no real good reason other than inattention to a simple detail. It truly costs no more effort (or at most very little additional effort) to do this sort of thing the "right way" up front versus the sloppy way. Yet the downstream penalties of doing it the sloppy way can be monumental:

Bug reports that were not repeatable and that made no sense
Developers looking in the wrong places for the cause of the bug
A sense of despair in that inconsistent, non-repeatable behavior on the DEV & TEST boxes made it seem like we had no control.
An expensive goose chase made more costly due to the challenges in simply trying to find the test logins that we created for our own use.
And we had to manually and painstakingly fix the errors on all inconsistent platforms.

Suggestions

Because there was a "User Admin" panel in another part of a related application, the developer chose to do this work manually each time.

A better approach would have been to have followed how we set up data in every other case. That is, by using written SQL insert code files that we could run on any environment to achieve consistency.

When Might This Be Appropriate?

I am not sure, but I do not think that inconsistency is ever a good idea. (Unless of course you are purposefully trying to stymie fellow developers and testers with oh-so-subtle, barely detectable, differences.

When Fancy Hurts Quality and Maintainability

JonKern — Wed, 08 Sep 2010 11:44:09 CDT

Background

Our application tangentially had to deal with a prior design around Users and Roles that was in place for an existing portal. Our online application also needed users and roles, and the customer wanted to have "single sign-on" which meant they would provide the User management with their existing tables.

Solution

Here is a model that we ran into to store Users and their Roles:

ATTR_NAME ATTR_DESC
FIRST_NAME User's First name
LAST_NAME User's Last name
USER_EMAIL User's Email
CARBON_COPY Carbon Copy for email
BLIND_COPY Blind Carbon Copy for email

ATTR_NAME	ATTR_DESC
FIRST_NAME	User's First name
LAST_NAME	User's Last name
USER_EMAIL	User's Email
CARBON_COPY	Carbon Copy for email
BLIND_COPY	Blind Carbon Copy for email

It is important to note that User attributes hold data like shown in this table:

The developers were clever in that you could add new attributes to a User without requiring the main USERS table to change.

However, there are many, many, side effects to this solution.

Problems

What's In a Domain?
Okay, since when are first and last names not core attributes of a User object?

Email

Note how the _USERS table has an attribute called "EMAIL". Unfortunately, the Email that the system cares about is found solely as a USER_ATTRIB. To get the user's email, you have to JOIN tables to find *IF* there is an email for the user.

select
bu.user_id "ID", bu.user_name "Name", bua.attr_value as "Login Email",
br.role_name as "Role", bu.user_desc "Description"
from
bus_users bu, bus_roles br, bus_user_roles bur, bus_attrib ba, bus_user_attrib bua
where bu.user_id = bur.user_id
and bur.role_id = br.role_id and bu.user_id(+) = bua.user_id
and bua.attr_id(+) = ba.attr_id and ba.attr_name in ('USER_EMAIL')
order by bu.user_name;

Just try getting the First name, Last Name, and Email for a given User. ARRGGHHHH!

(select lbua.attr_value from bus_users lbu, bus_user_attrib lbua, bus_attrib lba
WHERE lba.attr_name in ('FIRST_NAME') and lbu.user_id(+) = lbua.user_id
and lbua.attr_id(+) = lba.attr_id and lbu.user_id=bu.user_id
) || ' ' ||
(select lbua.attr_value from bus_users lbu, bus_user_attrib lbua, bus_attrib lba
WHERE lba.attr_name in ('LAST_NAME') and lbu.user_id(+) = lbua.user_id
and lbua.attr_id(+) = lba.attr_id and lbu.user_id=bu.user_id
) as "Name"

Now maybe there are better ways to grab the full data that reflects a user, but all I can say is that separating out core domain attributes like name and email into record-based data into a joined table is overkill.

Another problem arises in that you have to try all outer joins... you have no idea if data exists for these various user_attributes.

Two Emails
Since there are two emails, how are new developers supposed to know which one to use?

It looks as if the original User object was extended to accommodate new attributes -- like first and last name and email.

An effort should have been made to eliminate the no-longer-needed USER.EMAIL column. Leaving multiple solutions for EMAIL can lead only to confusion and mistakes.

Plurals
It should go without saying, that a Class name or a Table name should rarely involve a plural.

USERS is a bad name. The name should be USER. The plural comes when we have multiple records or multiple objects.

Turning SQL on Its Ear

Can you say slow performance and wacky SQL?

SQL normally performs well when you add domain properties as columns, and instances as records.

By placing individual property data as records, it basically ruins the SQL optimization.

In addition, the Users in the database did not all have EMAIL as attributes. I was given a simple sql query to look up users. All seemed well enough, until I was trying to find a new user just added by an Admin. No luck. So, after a bit of frustration, I tried to look just in the core USERS table for the new user (BTW: USER_NAME is limited to 8 measly characters). So then I discovered that I had to do an OUTER join to allow for the case where the EMAIL may not be present.

Oddly enough, the same developer that gave me this SQL, had used it in our app's code. Guess what? Some functionality of the App did not work right for some users. Go figure.

The app in which this data design exists has horrid performance whenever the code we are working on has to touch this set of tables. Even doing direct queries takes a long time.

Suggestions

Instead of making the extensible "User Attributes" cover core domain properties that are needed in many parts of the apps, properties like first and last name should be directly added to the User table. If you need to add different emails, then create an Email table and include Email Types (Primary, CC, BCC, Forward, etc.). Then a User can have 1..* Email.

Also, in a more object-oriented solution -- were these classes and not tables -- you can more easily collect such loosely-defined properties on the fly.

When Might This Be Appropriate?

In the case where you need to allow the user to extend and customize properties, "User Defined Types/Attributes" can be a handy technique. The alternative would be to add columns like "CustomProperty_1", "CustomProperty_2", "CustomProperty_3", etc. While this is more performant, you still need to allow the user to provide their custom "meaning" to the custom property, and it is limited to however many dummy columns you added to the table. Not to mention what to do if you need multiple data types (numbers, dates, text).

Calendar Date Select

JonKern — Tue, 07 Sep 2010 10:10:05 CDT

Here's a quick guide to getting started with Calendar Date Select gem from Tim Harper, et al.

Dates in forms can suck when using the standard f.date_select. Consider the following:

Model:
class RegistrationSetup
include MongoMapper::Document
key :title, String, :required => true
key :open, Date
key :close, Date
end

And a typical generated view with "date_select:"

View:
<p>
<%= f.label :from %><br />
<%= f.date_select :from %>
</p>

This results in 3 spinners:

The object save fails unless you deal with the date chunking. In addition the use of 3 spinners can be a bit laborious for the user. Naturally, in the controller, we could reassemble the constituent date parts into a Date... Blech. I'm too lazy for that. Plus I want a more modern date selector!

A pop-up calendar is a better way!

Here's a way that I found to use the calendar_date_select gem (see here)

Step 1. Install:

sudo gem install calendar_date_select

Step 2. Get code from here: http://github.com/timcharper/calendar_date_select/downloads/

unzip the download somewhere
transfer the appropriate files into your project's directories as follows:

public/images
public/javascripts
public/stylesheets directories

put the stylesheets you want (I just copied them all) into

public/stylesheets

Step 3. Add to layout/application.html.erb

<%= javascript_include_tag :defaults %>
<%= calendar_date_select_includes %>

Step 4. Add to environment.rb:

config.gem "calendar_date_select"

Step 5. Add to your UI; e.g., _form.html.erb

<div id="calendar">
<p>
<%#= calendar_date_select_tag "e_date", Date.today, :embedded => true, :year_range => 0.years.ago..1.years.from_now %>
Open: <%= f.calendar_date_select :open, :size => 15 %>
Close: <%= f.calendar_date_select :close, :size => 15 %>
</p>
</div>

And voila!

I am sure there are other ways to get this gem installed and working.

TD Blog

JonKern — Sun, 05 Sep 2010 23:07:15 CDT

The last few postings from my blog:

The Keys to Software Development

JonKern — Sun, 05 Sep 2010 23:04:22 CDT

Add photo caption or credit here.

Rails Ramblings

JonKern — Sun, 05 Sep 2010 22:56:03 CDT

As I run across items I find useful in the world of Ruby and Rails...

Calendar Date Select

Overview

JonKern — Mon, 23 Aug 2010 12:30:32 CDT

NOTE:	The "Jon's Agile Development Tips" pages are under construction. The other pages are generally complete.

NOTE:

The "Jon's Agile Development Tips" pages are under construction. The other pages are generally complete.

This site is dedicated to pointing out Technical Debt ("TD") that we run across during our work in the hopes that we all might learn:

Ways to avoid Technical Debt
How to recognize Technical Debt
The costs of Technical Debt
Prevention is better than cure (just like in your health)

The site also holds various techniques I employ for running agile software development projects. This part is a kind of work in progress, as I try to transfer stuff from other project wikis over to here in piecemeal fashion.

Look 'em Over

The examples page is a fun place to start.

Make some comments on any page as you see fit.

Join the Fray!

After you get some ideas from the examples page, consider uploading some of your own. To show us your gems, simply follow the instructions here for posting examples.

Why the term?

I have been calling this "stuff" by the term "Technical Debt" for so long, I don't even recall when, why, or how I began using the term. I have a vague memory of an audience member shouting out the term after I described the concepts in a presentation. Dunno... Doesn't matter. A quick Google seems to ascribe the term to Ward Cunningham... in his OOPSLA '92 article. No matter who came up with the term, the intent of all usages seems to be similar. Basically, the idea is to describe what can happen in the software world in common financial terms that most non-technical people can understand.

The term has the word "debt" so that most people will sense that -- like financial debt -- technical debt:

could increase over time
requires servicing the debt
needs to be paid off
can cause bankruptcy if unchecked

STD?

Should I name it Software Technical Debt, for short?

A nice Parable on Agile Concepts.

Release Planning

JonKern — Fri, 30 Apr 2010 21:13:48 CDT

Our goal is to arrive at a rough estimate for the batch of features and tasks necessary to deem a "Release." For example, a release can be version 1.0, or a smaller "dot" release. We simply work from an initial list of items to complete.
We usually have a set of minimum features we want to consider for our new release. Though we might lay out a rough iteration plan, it is subject to whatever we discover once we get into the thick of things. Sometimes we have to do some technical spikes if we are investigating new techniques.

Overlapping Streams

The hard part of this process is to show the parallel aspects in diagrams. Before the team finishes the current iteration, the business needs to be poised and ready with a plan of what they would like in the next iteration. The following diagram can be the same whether iterations are down the left-hand side, and versions replace the "Release" blocks on the right-hand side.

Ensuring stability is covered under Parallel and Branch Development.

Writing Good Issues

JonKern — Fri, 30 Apr 2010 21:03:08 CDT

I use Jira from Atlassian, so your mileage may vary. But in general, Softwae Development is nothing more complex than accomplishing a list of "to-dos" :-)

A feature is:

something the product does
think marketing brochure bullets or finding it in the User Manual
It might get bugs.
It might get improvements.
It might even be dropped.

A task is

something done one time
Is often an implementation detail for creating features
never appears on a marketing brochure or in a feature list
Examples:
- add users for beta;
- get app stood up on PROD;
- write a Flex UI training plan
- Implement the persistence layer (tables, DAO)

A subtask is

a small bit of effort that you might do to accomplish the parent issue.
It is often useful to help you track your to-dos
Is useful to estimate a larger task by breaking it down into visible chunks.
Is used solely by the dev team... that is, it should not be surfaced and scheduled by the business

A bug is

something that is not working according to plan.
should be considered when doing iteration planning
Covered in more detail in Writing Good Bugs.

Tips

Issue Type

Summary
(Title)

Make the title clear and concise. It should indicate something the user does or the system does. If it is a Group-type issue, preface with "Group: " just so that it stands out in the various views.

"Agent can select COB easily"
"Agent can view premium breakdowns and summary"

Description

NOTE: For lengthy descriptions that also serve as "corporate memory" of the major requirements, do the bulk of docs on the wiki, summarize in the issue and paste a URL.

Be sure to describe the issue fully so that the "client-valued" aspects shine through.

Here are some templates (search Google for more) to help you word the issues properly.

One template
"As a role, I should be able to feature, so that benefit"
"As an Agent, I should be able to choose class of business easily (searching by name or SIC code), so that I can quickly and accurately prepare a quote for my client."
FDD template
Another template (from Feature-Driven Development) is:
<action> <result> <object>
"Display the detail & summary premium data (to the agent)."
"Create a PDF of the detail & summary premium data (so the agent can send to client)."
Scenario Template
Be sure to also include how to know when an issue is complete and testable. Describe various scenarios if needed, when there are different conditions to consider. The point here is to make it clear to the developer and the tester what is expected.

Title	Clearly describe the context/what's different
Given...	Some context
When...	User does xyz
Then...	this happens

Examples:

Title	Show Premium Details - Single Line Of Business
Given...	A Quote for Business Owners Insurance
When...	displays the premium after successfully rating
Then...	a clear breakdown of the premium details (from ART) will be displayed in a UI, see wiki . At the bottom of the columnar-based details, show a Premium Total.

Title	Show Premium Details - Multiple LOBs
Given...	An RQA with Business Owners + Workers Comp
When...	displays the premium after successfully rating
Then...	For each LOB, show a clear breakdown of the premium details. For each LOB, show the premium subtotal. At the bottom of the columnar-based details, show a Premium Total for all LOBs.

Component

Useful to link the issue to a specific area of the application or work efforts.

Version

a.k.a. "Iteration" -- implies when the feature is scheduled for completion

Assign

Set during issue handling process. Generally should be based on person's knowledge and skills in inverse proportion to the criticality and complexity and deadline. Should not simply be any warm body.

Priority

Often known in advance. Helps team understand relative importance of an issue. Macro priority is the Version. This priority can sometimes be used to help a developer sort their work tasks in proper order.

Estimate

Developer doing the task can enter their estimate (see process)

Due Date

Used when an issue needs to be done by a date other than end of milestone (used infrequently).

Attachment

Upload any useful attachments, or reference those that are on a wiki page!

Links

If there are related issues or dependencies, add them via link. For example see Linking Issues

Writing Good Bugs

JonKern — Fri, 30 Apr 2010 20:51:25 CDT

Most of the instructions for Writing Good Issues still apply!

Meaningful title
- Indicate what is wrong
- Example: "BOP Application" as a title is too wide open to interpretation. One tip is to think of being in your user's/client's shoes and reading the title as part of the Release Notes. Ask yourself, "would the title make any sense in such a stand-alone fashion?"
  Contrast how the two would read in a "Release Notes" document/webpage:
  "Bug Fixed: BOP Application"
  "Bug Fixed: BOP Application -- Form missing fax number"
Clear description
- What is wrong
- Add expected or desired behavior
Choose the component, if known. Otherwise, development will categorize it
Record as much context as possible
- Where you were testing (if multiple environments)
- What you were doing that led to the bug
- What steps are required to reproduce the bug
- Any specific identifier for the data (e.g., an ID, a title)
Add any helpful screenshots
Indicate the severity of the bug (see below)

Severity

When we receive a PRODUCTION bug, we have to immediately determine the nature of the bug. Basically, we need to know if we should fix it ASAP because it is causing major problems for our customers, or if it can wait.
To evaluate a bug so that we can determine the severity, we need to have enough additional information in the following fashion (for example):

Severity	Impact to the user getting their job done
SEV 1	Cannot perform their job. Loss of data, time, and/or money. No workaround exists.
SEV 2	Can perform their job, but it takes much longer than it should. Workaround exists.
SEV 3	Bug is annoying, workaround exists.
Importance	Indicate the likelihood of this occurring, or the importance of getting it fixed

The SAMPLE table below outlines the response times based on the level of severity of the bug.

Severity	Criteria	Response Goal	Resolution Goal
1	Causes a Severe impact on customer's business (e.g., can't provide a quote, quote was inaccurate). Causes the Portal to cease: (a) working, or (b) working in any material respect. This includes any error in the Portal that poses harm to Customer's computer systems, or data losses. No known workaround available.	2 Hours *	Will work on the problem until resolved or a workaround is found.
2	Causes a major function of the Portal to be substantially impaired, although it still operates. In Dev Environment, causes a non-critical impact on development activity (e.g., jeopardizing production rollout). In Production Environment, causes a non-critical impact on business operations of agent (e.g., some premium details or other question data missing from a form). Intermittent disruption of portal service. No stable workaround available.	4 Hours *	Vendor will make reasonable efforts to provide a correction or workaround within three (3) days
3	Causes a minor function of the Portal to be impaired. In Dev Environment, causes a minor impact on development activity (e.g., does not jeopardize production rollout). In Production Environment, causes a minor impact on business operations of end customer (e.g., minimal degradation of functionality).	2 Days	Will make reasonable efforts to resolve the problem or provide a workaround within thirty (30) days
4	In Dev Environment, causes little or no impact on development activity. In Production Environment, causes little or no impact on business operations of end user.	1 Week	Will make reasonable efforts to resolve the problem or provide a workaround if such problem has not been resolved in the next maintenance release of the Portal

NOTE:
* Issues of Critical or High severity levels must be reported via telephone to receive these response times.

Issue Contents

Bugs should describe all that is necessary for the developer to:

know the problem via a test/demonstration (i.e., it needs to be repeatable),
troubleshoot, and
verify the problem is gone via the same (and possibly other) tests/demonstrations.

Be thorough

Reading:	MG-1243@JIRA Underwriter questions - yes/no choices When switching a yes/no choice from yes to no, the NO field does not value on the first selection. See screen printing. No to Yes is working but Yes to NO required two clicks. Happening in both "Appliances - Household" and "Apartment/Condo." I didn't check the other classes of business yet.
Leaves me wondering:	Is it the UI? Can you see this behavior in Diner? In other parts of the Qstnr 3? Is it the data? Does this happen on every UW question? ...on every question with a Yes/No? ...on only some questions? ...on only questions that have rules? ...on only questions that are made visible by rules? ...on only choices that participate in rules?

When will the app be done

JonKern — Fri, 30 Apr 2010 20:35:12 CDT

This is often a tough balance between the Dev team and the Business. "How much" and "When" are critical to the business and elusive for the Dev team at the outset.

Honesty and collaboration is the best course of action. Sure, you can answer what they want to hear, but that ain't right.

Do your best at up-front estimation, make a stab at team size and velocity (I use hours, not story points), and give a general "season" as a probably date.

As we elicit the overall requirements and rough list of features, I'll end up creating a spreadsheet that will include columns for "Estimate" and "Risk Multiplier." The Risk Multiplier is used to provide some high-lows. If we see a lot of high risk in an area, e may choose to mitigate that risk by doing a technical spike, for example.

Oh, and keep folks informed as we learn more!

Initial Requirements Scoping

JonKern — Fri, 30 Apr 2010 20:19:05 CDT

For the initial "Phase 0" or "Release Planning" effort, we want to arrive at enough of a plan to give the business some idea as to the approximate schedule and cost for what the business wants built. Normally, we use this up-front planning time to iterate with the business on just what will constitute a good first product release. That is, the business might have some ideas for features that cost more than they are worth. Conversely, the development team might have an approach in mind that makes other business value attainable. The point is that I like to work at a "level up" from the intended product. To understand the business environment and drivers surrounding the specific project is invaluable. This often allows us to come up with spectacular game-changing solutions because we aren't simply boxed in with an a priori idea of product shape (which naturally was formed within a set context).

We also let the business know that it is an "early-stage" plan, that we will continuously update with each iteration. Progress will be very visible to all participants throughout, allowing for adjustments along the way.

Business Modeling

Software is designed to solve business problems. In the beginning of a project, I like to have a series of meetings with the business to flesh out the core requirements with subject matter experts, architects, and stakeholders in the room. It is often helpful to think about the problem by working backwards from the endpoint. Many times, there is a core set of requirements needed to meet minimal business worth for the stakeholders. That is, without these features, nothing else matters. For example, what requirements will an insurance agent have for the system to allow them to place more business and manage their customers? What aspects of the system are needed for the insurance company's Underwriters to manage the lines of business rules and profitability/risk ratios? Add to this the unique aspects that the sales portal could provide (for example, through providing agents with a sales funnel concepts maybe?). The goal is to arrive at the sort of information we need to have on hand to make such functionality possible. Arriving at the core features helps us when deriving estimates and setting priorities. Non-core, or "mice to have" features can be shuffled to later in the project, or to version 2 following successful product launch.

Underlying the requirements gathering process (outlined below), we use object-oriented techniques. The domain information is placed into "Classes" with names familiar to the business, and appropriate associations. I use simple sticky notes and a whiteboard to start.

A great deal of emphasis is placed on the business model for the simple fact that this is where you make or break most software systems. By keeping the business model reflecting what you do for a living, we are more able to adapt to new feature requests or to move with the industry as changes inevitably occur.

To tie yourself to industry data formats, or database-specific calls, or user interface aspects, is to potentially cause future harm to the company. The results of doing such designs are often applications that are not easy to upgrade, apps that are hard to maintain, unpredictable schedules, and blown maintenance deadlines. All because of allowing "technical debt" to creep into the system and build up over time, unchecked.

Therefore, we tirelessly labor over getting the business model "front and center" in the application development effort. From this perspective, most other aspects "hang" off of the domain nomenclature: features, bugs, enhancements, etc., all use the same terminology

One more thing. We don't model "everything under the sun." If there is no need to record agent hair color (for example), we won't model that characteristic. If, however, the analytic logic could make use of number of times an agent's application was "referred," we would endeavor to record that in the business model.

Requirements Elicitation

As part of the process to discover the business model, we simply ask about the overarching business goals and the user's requirements. We do this in a very iterative and agile style, making successive passes until we're satisfied that we know enough to make a reasonable estimate and plan of attack.

Over time, the domain model and features list and cost estimate gain more and more information. In the early part of the process, we make huge steps with every day spent. As the process draws to a close, you will see less and less value coming from the time spent together, and fewer significant contributions to the overall understanding of the information leading to the estimate. You'll realize it is time to stop planning, and time to kick off development.

For most of my projects, the time has followed what FDD often cites: 10% of the overall development schedule. Three months for a three-year project. Two weeks for a 6-month project. Your mileage may vary.

System Purpose

It is critical to have a basic idea about what the system's purpose is, and any concrete features that it must have. At the outset of the project, it is imperative that we have folks in the room who are subject matter experts and who know what the system is intended to do. The System Purpose can be (indeed, I would go so far as to say "should be") quite simple (Peter Coad recommends 25 words or less):

SYSTEM PURPOSE:

Acme Insurance needs to build a "modern" web application to permit agencies to sell Acme insurance to the agency's clients. (1) Sell more insurance, (2) Reduce cost of sales, (3) Improve company's street cred with a modern app.

Acme Insurance needs to build a "modern" web application to permit agencies to sell Acme insurance to the agency's clients. (1) Sell more insurance, (2) Reduce cost of sales, (3) Improve company's street cred with a modern app.

Initial Requirements Gathering

You do not need to gather all of the requirements in the first go. That is, we do not need volumes of written requirements to start. We need to have enough information to make an initial stab at what the basic application needs to be, how we might architect it, and approximately what season it might be near completion (given a certain team size and make-up). Usually, the business wants a ballpark cost estimate, or wants to know what can be done for a specific budget.

NoteThe more accurate the estimate needs to be, the more work you need to do prior to giving an estimate. The Law of Diminishing Returns applies.

As we bore deeper into the domain features, we continue to add to the domain model and build up a simple list of features -- either as class methods or a simple text list to start. The wisdom comes in knowing what is important to capture, what details can be skipped until later, and what constitutes good enough for the first high-level pass. The "depth" is tempered by the up-front knowledge of how accurate and detailed the estimate needs to be. The depth is also controlled by a sense of risk of estimating. That is, if a feature sounds potentially complex and risky, we'll want to spend enough time to dig a bit deeper. Maybe go down to the next level of sub-features, so to speak, or break down what is turning out to be a "Feature Set."

Balance Tip We are always playing a balancing game. Do too much upfront, and it may be wasted effort. Do too little, and you may provide the business with a woefully inadequate estimate, or get a nasty surprise at the end when that missing feature appears.

At the outset, we just need a reasonable title for the feature, maybe a sentence or two of prose. Enough to make a reasonable hack at estimating the size. When the time comes to actually schedule and produce the feature, we get back together and talk it out in more detail – again, preferably with a subject matter expert leading the way. This will lead to a more accurate estimate as it will incorporate more project learning.

Part of the agile philosophy is to prefer individuals and interactions over documentation – especially since there is no such thing as a perfect specification anyway. And, to spend double or triple the time on writing more documentation will experience the law of diminishing returns in a dramatic fashion.

Once we get to a point of having the bulk of the initial features, we will ask for them to be prioritized -- though sometimes we have gathered that information along the way when it was obvious. By concentrating on the highest priority ("must-have") features first, we begin to deliver recognizable value almost immediately as we produce working software each day of each iteration (admittedly, a releasable product requires a bunch of features before real value is delivered).

Balance Tip How do we estimate accurately? You may not have the development team intact when doing Phase 0. You may not feel comfortable with the technology. You may not be sure of some of the requirements. You must balance knowing enough about the app's requirements yet not having all of the detail.

Typically, you have to rely on your "gut feel" and those of your trusted senior architect/developer types. If there are high risks, you may need to counter by:

Doing some investigation to reduce the risk
Add a multiplier/padding to the estimate to attempt to account for uncertainty

So one quick-pass technique I use is to ballpark the cost of developing the basic domain model components (classes, tables, etc.). Gather the list of features into a spreadsheet or convenient issue tracker (but this usually takes more time than a quick spreadsheet). Add a column for estimate in hours. Add a column for "risk factor" in case you need to multiply the estimate to account for uncertainty of general "gnarliness" of the requirement. If you have a feel for the major UI components above and beyond the feature list, throw them into the mix as well. Anything else you can think of... setting up automated processes, various environments, and so on. Balance the numbers with the gut feel folks have about the app; for example, "I think a team of six can have that implemented in 4 months."

Risk Leveling

The way I like to look at this initial period, is to consider what we are trying to accomplish. We need to provide an estimate and rough "shape" of the application to a certain level of "risk" to make the stakeholders happy. If estimates need to be very accurate, then you need to squeeze out medium and high risks. You might even need to do an initial project to build out and test some aspects of your architecture. If you need a moderately accurate estimate and you have worked in this kind of an app many times, then maybe you only need to squeeze out the high-risk areas. The point is that not all features on the list are the same, unless you make an effort to "level" out the uncertainty.

"Doing" Complements Theorizing

Part of being agile means we seek to understand enough about what we are trying to accomplish to help provide meaningful estimates. Instead of producing purely paper output and schedules made with only best guesses, we will strive to uncover enough detail about what it is we'll be building, and what complexities we may run into.

For example, there are some technical challenges facing us with:

Should we build a porting scheme for the 3rd-party back-end system (Sphnx)? Or should we access it in real-time?
What details can we find out about the web service rating engine?
We might need to scope out the Java/Flex architecture (thin slice) to see "how it feels" (and costs to develop)

How do we do this? We intend to explore a bit of the solution space itself to get a better handle on what we are up against. Our gut instinct tells us that we will want to ensure efficient means of integrating with Sphnx, for example.

By way of analogy... Think of developing an application with 100 features being akin to carrying 100 pumpkins from Point A to B by foot, one at a time (maybe 300 years ago). We could do some theorizing about how long it will take by guessing on the distance, and guessing at how much ground we might cover in a typical day. However, if nobody has ever gone from A to B before, how accurate do you think we would be? However, if we actually try to take one pumpkin to get a more realistic feeling for the complexity involved, we will likely learn some valuable details along the way. For example, we would learn about rivers to ford, possibly forests with bears. We would learn about what sorts of unexpected deviations might occur that add unpredictable variability to the equation.

At any rate, I hope you get the picture. By practicing delivering some features, even if a bit shallow at first, we will gain valuable knowledge to help the team going forward. At a minimum, we need to have a handle on the technical approach to lend credence to our initial rough release plan estimates.

Expected Results

We want the "Requirements Workshop" to help produce

A reasonable understanding of the major features needed for this application
A pretty solid business model able to take us through initial release
A realistic "feel" for the application's technical aspects
Some possible solutions for the more technically challenging aspects of this app
A "slice" of the requirements in visible form, in a working piece of software

We will also produce an initial release schedule/roadmap. This will consist of a simple list of features to be delivered in the initial release and possibly a set for one subsequent release (if we have convinced the business that some features can wait until after the initial release). We will also put a "size" and schedule around what Release #1 will require. Often, I will break down larger pieces of the puzzle for Release 1 by roughing out an Iteration Schedule of how certain larger (major) feature sets will roll out. But this is always done with extreme simplicity since we do not want to waste excessive effort on any long-term planning details.

About Me

JonKern — Fri, 30 Apr 2010 19:52:43 CDT

I live in eastern Pennsylvania in a house from early 1700s -- old for the USA <g>, but not for the Old World! I have a wife, a son (20 something), and twin daughters (teens), a Yorkshire Terrier, 2 indoor cats, and 2 barn cats.

I began life testing jet engines as an aerospace engineer. Then I discovered software on a Dec Vax. (I also discovered Adventure!) I learned IDL, FORTRAN, some C. Then I discovered object-oriented programming (with Turbo C++). It was all over after that. I cut my teeth on R:Base and learned SQL -- though I did not know it at the time. I did application development on the side. I became proficient at PowerBuilder and later Power++. I learned C as part of my graduate work in numerical computations. I was using C for flight simulation and hardware-in-the-loop projects.

I met Peter Coad in 92 -- he became my OO mentor. I bought Together/C+ + in 1994 and began to do whatever I could to help this wonderful product grow. I formed LightshipInc, a consulting company, in 1995, grew to about 10 people. We designed and built IBM's Manufacturing Execution system. I did the UI design on paper and sold the execs that we could take on the (3 year) project. We did great engineering product order configurator using OODBMS for a pump manufacturer (Ingersoll-Rand).

Throughout the 80s/early 90s, I experienced inefficiencies of doing software government-style. So I had created my own agile methods in "self defense" (besides, I don't like to waste money). I first published this stuff in a Lotus Developers book or two. (Don't ask why <g>.)

In 1999, Peter Coad recruited me to help form TogetherSoft -- with quite a few other guys. In a few years we had grown quite a bit in size. From 2000 to 2002 I commuted to St. Petersburg, Russia, to lead new product development and what not (like learn to drink vodka properly). Brilliant developers.

In February 2001, I attended the gathering in Snowbird that led to the Agile Manifesto -- as I recall (which may be flawed memory), 'agile' being suggested because I had done a lot of work with Agile Fighter research, and Alistair Cockburn did not want to be known as a "Lightweight Methodologist."

In 2003 we sold TogetherSoft to Borland. I was recruited by Compuware to come work for the OptimalJ team in May 2003. I tried my best... there was too much inertia for me to have the impact I desired. Awesome product idea, however. Brilliant developers. I got to start a blog (~35K hits/month), and a newsletter (~6,000 subscribers).

In Fall 2006, I turned private, working with a couple of friends of mine at immuexa... have worked on a SemWeb project, and on a insurance portal project with Synergy Components, among other things. We also do pre-planning software for the fire and first responders at Fire Planning Associates.

My ideal job is working on cool projects with a great team, a great customer; and to be able to travel.

My family and I travel quite a bit... we like to ski, I like to rock climb, climb mountains (Alpine), and drive cars fast (mostly Audis) around tracks and autocrosses :-)

Cheers!

Detailed Estimation Sucks

JonKern — Tue, 20 Apr 2010 21:29:40 CDT

Pretend you have a team cranking away at a production app. Part of the IT staff, so to speak.

Please have a seat. Yes, over there by the window is fine. Buckle up. You may experience some turbulence.

If you want to know "when you can get all that stuff done," sit the team in a room, show them the list of stuff, and ask them. I'll give you an hour. I suggest you have them use seasons from the sounds of the current mess :-)

If you really want to know within a month as to when the list of 1000 issues will be done (including the 250 that you don't yet know about), then wait until you are about to release, and you will have a very accurate number.

I doubt that estimating accurately or poorly will help all that much in knowing when an entire list of stuff will be done. Especially when the list changes over time. Given that, why spend any/much energy? "Don't polish a turd."

Why not simply guarantee to management that the team will do their best effort. Just grab a stack of important (i.e., prioritized) stuff to do (loosely based on a logical roadmap/release plan), track the amount of things done each iteration on a chart. Skip the up-front waste of time estimating everything in detail. I mean really... just stab at "we can do 20" and see how it goes -- adjust as needed. Should take all of about 30 minutes each iteration depending on your tools and your team collaboration.

Put an indication on the chart of number of total issues -- but be ready to continually update this as new issues/bugs arrive (it's lots of fun to watch the goal continually move further away). Or have the chart represent the bare minimum marketable features (bigger ticket items without the details). Now anyone can sense the rate of closure of the team towards the goal as each iteration piles on more closed issues. After two iterations, you can draw a straight line, after three, you can use fancier curve fits and show the least squares correlation coefficient <g>.

Instead of spending time estimating, spend time making the project progress information radiator as meaningful as possible. Believe me, software is merely working a to-do list. Anybody can watch a list of things getting ticked off and do a mental estimation as to rate of progress. Build a better view into what the project is actually doing and let people think about how it might look going forward. At least your historical data is guaranteed accurate!

The only time I would estimate "more accurately" is if I have competing ideas or ways to solve a problem via wildly differing approaches. Then maybe we might want to weigh one versus the other from a business point of view because it could make a big difference in the long run.

But for an entrenched team on an entrenched product to estimate each little feature/issue/bug fix/story at the outset of each iteration?? I question the value of the team's time being spent doing that. I gave it up for Lent.

Try no doing detailed iteration estimation. It can be very liberating to just do the work as best and as fast as you can.

At some point, you and management will realize that it doesn't matter if the team is good at estimating or not. At some point you may realize that most of the estimation process is an illusion, a grand trip into self-delusion fantasy land. The same amount of work will get done (well, more will likely get done if less time is spent estimating).

If you need to control what work gets done by when, then you prioritize mercilessly and do a good job to make development efficient (which includes smart architecture and design, among other things). If you need to speed up when the work gets done, then you need to consider resources (better or more) and/or reducing scope.

Now, if you want to spend time making the team better at estimating for their own edification (a la PSP), let them do their own estimates and track actuals and try to improve over time.

If only software were as simple as aerodynamics. I can predict quite accurately how a NACA airfoil is going to behave in 2D and 3D airflow. I have never been able to have a team predict to that level of accuracy. Predicting software is not much different than predicting a relationship.

Something to ponder... Why bother with estimates at all?

1. Who needs them and what are they for?
2. What happens if you are dead-on accurate?
3. What happens if you are off by 100%?
4. What if #2 == #3??
5. Are the workers going to be fired with poor estimates? Given raises?
6. How did they estimate before?
7. Is estimation the best use of their time?

We're now ready for take-off!

Miscellanea

JonKern — Sun, 28 Mar 2010 08:35:04 CDT

My adventure climbing the Matterhorn in 2009: Switzerland Climbing

Great bread recipe, no kneading required: Artisanal Bread

Here are some silly attempts at audio humor by using Voice Candy application:

Agile Cheese wav file
Use the Agile Force wav file
Nutty User Stories wav file
Mr. Agile Robotto wav file

Even though there is dispute over the whole right/left thing, this is humorous: "Right Brain" Test

Artisan Bread Recipe

JonKern — Sat, 30 Jan 2010 08:17:17 CST

A friend (Jacques Murphy), gave us this printed recipe. I googled for "Artisan-Free-Form loaf" and found many references. Apparently, it is from a book: Bake bread easily every day with Artisan Bread in Five Minutes a Day by Jeff Hertzberg and Zoë François.

Since the dough can be stored in a fridge, I purchased a square plastic container with a lid, that will hold a full, risen, batch (6 liters).

Preparation
The mixing takes all of about 10 minutes -- I often do it when I want a break from thinking about a tough coding or design problem :-)

Here is what I do:

Make 3 cups of 100-deg F. water, pour into mixing 6 liter container/big bowl
(Sometimes, if I have a "stump" of red wine or the like, I'll use it as part of the liquid. Going over 1cup wine will make it very wine-y tasting.)
Add 1.5T granulated yeast (== 1T + 1.5t)
Add 1.5T sea salt (or Lawry's garlic salt for even more pizazz)
Sometimes I add in 1.5T sugar -- just because I feel like feeding the yeasties some sugar
6.5 cups unsifted, unbleached, all-purpose flour. I scoop in the measuring cup and level off the measure with the straight edge of a knife.
With wooden spoon, strong spatula, stir up the mixture until there are no longer any dry clumps of flour
Let rise about 2+ hours, covered -- but not tightly sealed (yeasties have to breath, dontchaknow). Room temperature is fine, or near the wood stove, or sometimes I put it nearby on the stove when I am making stock for hours. Yeaties like a little warmth as they grow and multiply.

You could put the dough into the fridge -- we have had the dough in the fridge for as long as 12 days with no issues.

Baking
Place a rack in the middle of a 450 deg F. oven, with the broiler pan positioned in the lowest position (you'll see why below).

Prepare a baking sheet -- either with corn meal, or parchment/baking paper.

Since the dough can be pretty sticky at times, sprinkle the dough with flour (I cover my hands with flour too). Dig out a big clump -- I keep a sharp knife nearby to cut off the clump as I pull it up from the container. Work the clump into the shape you desire, additional flour is sometimes needed to keep it from sticking to your hands.

You can make any size and shape you like:

round or oblong loaf
long baguette
rolls

Score the top of the loaf with a sharp knife... Sprinkle liberally with flour (for that artisan look) or large salt crystals over top if you like.

You can let it rest for 30-40 minutes -- especially if it is from the fridge. But I have already rushed the dough into the oven (and my mouth) with no real issues.

The Power of Steam
Get a cup of water ready to pour into the broiler pan (something with a long handle is a good idea).

Place the bread into the oven.

Pull the broiler pan out a bit, so that you can quickly pour the water into the pan -- it will steam like crazy, so don't burn yourself. Quickly close the door -- which will push the pan back into the oven).

The steam will make that yummy hard crust!

Bake for about 30 minutes, or until the crust is brown and firm to the touch. Vary the time based on the size and shape...

Versatility

We also use the dough for homemade pizza and strombolis.

After a few times making this recipe, you'll have all the nuances figured out. It is a very robust recipe that can withstand lots of variations. Experiment!