Architecture in an Agile World: OOPSLA Workshop Report

Workshop conclusions

Architecture and agile are two important ideas in software engineering that ought to be used together to make software development more effective.

1. A "mixed model" of agile and architecture-driven development

The obvious conclusion in the workshop was that "pure agile" and "pure architecture-driven" approaches are not the way to go.

• In a "pure agile" approach, the system would have an "emergent architecture" -- because the most extreme agilists would consider any serious up-front architecture planning and modeling as unnecessary big design up front.
• In a "pure architecture-driven" approach, all architecture choices should be made using a top-down development approach instead of using code-level experimentation and refactoring.

We believe that some early attention to architecture planning will contribute to better communication within a development team. Even in a one-person project, architecture planning helps by giving the developer a good way to think through important development issues, instead of merely implementing his or her first idea.

The combination of agile and architecture-driven approaches is a good thing -- it creates some opportunities for discovering potential problems early in the development cycle.

• Agile development techniques are best for highlighting issues that are linked to the "uncertainties in customer requirements". Sometimes the customers aren't sure what they want -- agile practices help to delineate the areas of risk that are linked to this uncertainty.
• Architecture practices are good for surfacing some of the "technical unknowns".

So agile and architecture-driven will complement each other to aid in the discovery of project problems early in the development cycle.

2. The architect can't be an isolated outsider in the software development process

In an agile world, an architect (or any development team member who is involved with software architecture or high-level design) has to be connected with development teams.

The workshop participants have seen organizations two models of interactions between architects and developers:

• In some cases, a separate architecture group has authority over the teams that perform design and coding tasks -- they can veto design choices or they control the funding for development organizations.
• In other situations, the architects might work indirectly -- they use their status as technical experts and their contacts within development organizations to gain acceptance for the the high-level design choices they put into the architecture models and documentation.

In either case, an architect needs to overcome the perception of being an "outsider" to the development work. One "agile" way to reduce the distance between architects and developers is to include architects as members of a development team, with specific development and testing tasks assigned to them in each development iteration. One benefit of this model: it makes the architects into users of their own documents and models, so they aren't just off in an "ivory tower". Also, as a member of an agile team, they will have frequent discussions with other development team members, which should improve communication about the architecture.

On the other hand, the idea of "making an architect a member of a development team" is not perfect. Some architects have poor development skills. The distractions of the development process may limit the amount of time they can spend on architecture planning. In some large systems, it is necessary for architects to spend time thinking about larger issues -- not just the development tasks of a single development team.

3. Good architects and good agilists

Another important conclusion of the workshop -- there are many skills in common between "good architects" and "good agilists". The full workshop report has a list of key "patterns and anti-patterns" for architects and agilists.

Two key "patterns" for architects:

• An architect is a "coach/teacher" for the architecture. He/she helps the development team understand and extend the design in a way that keeps everyone organized.
• An architect is also a "student of the problem domain". It is important for an architect to keep learning about the problem domain as it changes and evolves.

There are some important patterns that are similar for agilists:

• An agilist must "respect people" first and foremost.
• An agilist is also "self-reflective" -- learning from experience and adapting the team's processes to work better in each iteration.

The architect patterns and the agilist patterns are definitely linked together.

4. Problems across the divide

Agile development teams have many complaints about architects. The top two problems:

• Agilists feel that architects are "always trying to push everyone around".
• In addition, agilists feel that architects often "overdesign".

These are valid complaints. Architects really do try to control things too much -- and an agile viewpoint could really help architects do their job better.

Architects often complain about agile teams -- some issues related to lack of forward vision and feedback:

• Most agile teams have trouble answering the question: "How much road do you have to pave ahead of you?" If the design vision only extends to the end of the next iteration, it might be easy for the design to fall off a cliff.
• Sometimes agile teams fail to get adequate feedback from stakeholders and customers. It isn't always the agile team's fault -- but teams should always try to do better.

Summary of conclusions on the post-workshop poster

In this workshop, the workshop participants explored questions related to the use of agile development techniques, the value of architecture planning, and how architecture-driven and agile approaches can coexist.

The top questions addressed in the workshop were:

• What are some of the risks and opportunities of blending agile and architecture-driven software development practices?
• How can architects establish credibility with software development teams?
• What are some key characteristics of good architects and good agilists?
• Why do agile groups have problems with architecture-driven processes and why do architects have problems with agile project organizations?

Initial thoughts

Architects play many roles in a software development organization:

• Business analyst -- understanding the business needs
• Expert in technical engineering constraints for the system
• Visionary -- the person who sees how everything will tie together, and how it will evolve in the future
• Facilitator for integration -- the person who makes sure the parts of a large system will fit together

Architecture should not always be labeled as "big bad design up front". Some early architecture planning is usually important -- even when the requirements are changing.

Risks and opportunities of blending agile and architecture-driven

There should be some alteratives to these two extremes:

• "pure agile" (no early architecture planning, only "emergent architecture")
• "pure architecture-driven" (most architecture decisions are made top-down)

• It is generally considered a good idea for a software project to "fail early" -- that means "uncover critical issues as early as possible"
  • reduces risk and costs -- by attacking the "period of uncertainty"
  • uncertainty about what the customer wants -- agile helps with this (deliver early and get feedback from the customer)
  • addressing technical unknowns -- architecture-level design work helps here (defines important design parameters to meet many of the non-functional "ility" requirements)
• some application development issues are impossible to know about in advance -- the customer might not know some factors that could affect the architecture (agility and flexibility are necessary)
• early consideration of "architecture requirements" can keep agile projects running smoothly
• in cloud computing and Software as a Service, it is essential to pay attention to both architecture and agility
  • architecture: "multi-tenancy" requires careful architecture planning to avoid issues with performance and feature interactions
  • agility: "continuous delivery" (new releases deployed to the cloud every four or six weeks) is an important characteristic of most SaaS application or component development

There are some important risks:

• Non-agile management: In some cases, the customer or project sponsor may want development progress reported as a waterfall.
• Agile teams might not want to spend enough time in early architecture exploration. For example, some Scrum teams limit their architecture work to "Sprint 0" (an initial prototyping iteration before starting the main development iterations).
• In agile projects, it may be difficult to schedule architecture improvement activities -- because the implementation of new customer requirements often take precedence over internal rework.

Establishing credibility for architects and architectures

In some cases, an architect has real authority -- the power to mandate design decisions, or direct control over development resources and money. In other cases, an architect is considered just a "technical expert" -- and his/her authority comes from the perceived quality of technical advice and the breadth of the organizational contacts in the architect's internal network.

In the agile development world, an "architect" is often viewed as an outsider:

• a person outside of the development team who imposes unwelcome constraints to possible solutions, an extra documentation burden
• architecture may be "thrown over the wall" -- the architect might not be involved in doing the development work

One approach for project organization to improve this "outsider" problem: instead of the "architect role" being an outside expert, the architect is one of the team members, with some direct design and coding responsibility.

• In an agile project, the "architect" might not even have a special title -- he/she is just an experienced team member who can coordinate architecture decisions and help all team members participate in the architecture.

One important discussion question: Should there be a "single person" who is responsible for the architecture? No consensus on this question.

• Single architect
  • Some people feel that having a single architect is good, because that person is "accountable" for important architectural decisions.
  • Architectural decisions are better made by a "benevolent dictator" than by a democratic vote of team members.
  • It can sometimes help to bring in an outside person as "chief architect" -- brings in new ideas, can be seen by team members as more objective and impartial
• No single architect
  • Experience gaps: It is difficult to get a single person who has the necessary experience in both the problem domain plus all of the facets of technical architecture, such as component architecture, development tools, testing environment, reliability issues, evolution issues, and so on.

Good architects and good agilists

In this section of the workshop, we did some brainstorming about the "patterns" and "anti-patterns" of architects and agilists.

Architect patterns

What are some of the characteristics of an effective architect?

• Navigator
• Coach/teacher
• Producer (like a "movie producer" -- makes everything work for the production)
• Puppet master (similar to Producer -- good at indirect influence)
• Seasoned veteran
• Long-bearded wise man
• Evangelist for the architecture
• Student of the problem domain (always learning about the problem area)

Architect anti-patterns

What can make an architect really awful?

• An "everything is a nail" designer
• Diva (short-tempered and high-maintenance)
• Magician
• Over-accomodating ("sure, I can change that for you")
• Chess master (he/she is always moving the pieces around)

Agilist patterns

What are some of the characteristics of an effective agilist?

• Respects people (first and foremost)
• Self-reflective (learns from experience)
• Multi-lingual (in two senses: can work in multiple programming languages and converse with stakeholders and team members in their own terms)
• good social skills (keeps on good relations with teammates)
• a "role model" for the values in the Agile Manifesto
• Change advocate
• Takes responsibility for his/her actions

Agilist anti-patterns

What can make an agilist really awful?

• Oversimplifies previously learned wisdom
• Passive/aggressive agilist ("talks the agile talk", but doesn't actually do all of the practices)
• only applies agile practices and values within his/her comfort zone
• uses agile as an excuse for not being disciplined ("coding cowboy camoflage")

One important idea for "being a good architect":

• If you an experienced architect who is an outsider to the project, you should work hard to become knowledgeable in the "problem domain".
• A good thing to say to the customer: "I'm going to work to learn enough about your problem so that if you knew what I know, you would make the same decisions."

One useful reflection on "being agile":

• The Agile community is like a political party -- it isn't necessary to accept every principle, especially because we know that Agile will evolve. But don't try to undermine its core values.

Agile groups problems with architecture-driven; architects problems with agile organizations

We started this discussion with checking version 2 of Kent Beck's XP book -- what does it say about the role of architects in an XP team?

• responsible for large scale refactoring
• write system-level tests that stress the architecture
• implement stories (high-level design stories)
• "direct" the architecture (coach the team about the architecture issues and choices)
• ensure that the system as "just enough architecture" -- "little architecture early", and as the system grows, make sure the architecture keeps pace
• architects also are involved in development tasks -- architecture is not a "separate function"

An interesting question from the discussion: "Do we need to create an architecture roadmap for a product (or series of products)?"

• The person who builds the roadmap needs lots of experience

Agile groups problems with architecture-driven

• Agilists seek to overthrow the established design process.
• Agilists believe in evolutionary design.
• Agilists think that architects are "throwing their weight around" -- using political pressure instead of technical arguments.
• Architects tend to overengineer

Architects problems with agile organizations

• Agile organizations have difficulty answering the question "How much road do you need to pave ahead of you?" How can they justify spending resources on architecture-level design work rather than delivering new customer features?
• Agile groups often fail to get adequate feedback from customers -- even though the agile principles insist on frequent customer feedback.
• Disaster story:
  • A project that used agile methods plus MDA.
  • They were able to easily change the messaging implementation, protocols, and component structure.
  • But, the real problem was that the system wasn't meeting the performance requirements -- a problem that couldn't be addressed by agile methods or MDA.

What other questions do we have?

In the workshop's initial brainstorming session, we came up with a number of other good topics to discuss. Unfortunately, because of the lack of time, we put these questions off until another future workshop:

Questions and issues on innovation for future discussion:

• Some possible approaches for integrating agile and architecture-driven, such as risk-centric model (see position paper by George Fairbanks).
• How do you measure the value of architecture?
• Are there some common "architecture breakers"? (Some characteristics of a system or its environment that will require some significant architectural rework.)
• Will agile practices impose or result in some architectural characteristics on the system being developed? (For example: more flexibility or modifiability, increased quality and reliability because of better test coverage, and so on.)