XP 2021 Report

Dennis Mancl
MSWX Software Experts
Bridgewater NJ
dmancl@acm.org

Summary

Due to continuing travel disruptions, the XP 2021 conference was a virtual conference, and it was somewhat similar in structure and content to the XP 2020 conference. Registration and attendance was lower than last year, but there was still a good crowd at the Tuesday keynote and some of the workshops.

I helped to support two panel sessions that were hosted by Steve Fraser (from Innoxec). Steve has regularly run panels at XP and other conferences, and the virtual format allowed him to pull together a multi-national panel more easily.

Steve McConnell keynote

Steve McConnell delivered the Tuesday keynote talk at XP 2021. His talk title was “20 Years is Enough! It’s Time to Update the Agile Principles and Values.” Steve is definitely an advocate of agile methods (and has published the management-oriented book More Effective Agile in 2020), but this talk made the argument that the Agile Manifesto can be an obstacle to effective agile adoption today. He believes that the Manifesto is just too old-fashioned, and it needs to change with the times.

Steve’s talk was pretty good. He is an effective advocate for opening up the Agile Manifesto to revision... His critique is based on his view that even the Agile Manifesto should be subject to the same “inspect and adapt” process that many agile methods use. He had several concrete suggestions for how to update the Manifesto to incorporate what the software industry has learned in the last twenty years. Most of his suggested changes are on target, although I think there would be a lot of resistance to making even minimal changes to a document that has such a religious following.

Steve points out that in 2001 (the year the Agile Manifesto was written), many software organizations were still hopelessly backward in their approach to software engineering. Instead of structured development, teams were still using the same “code and fix” model of development from the 1970s. But if you worked for a large software organization in a big company, there was a push throughout the 1990s to introduce a more formal and bureaucratic software development process – especially software methods that followed the SW-CMM (software capability maturity model).

The Agile Manifesto was written in the context of a software industry with these two forces at work: undisciplined chaos on one side and huge overhead on the other. Steve’s claim is that the Manifesto, with its four values statements that denigrate complex processes and tools, comprehensive documentation, contracts, and big detailed plans, is a reaction against the SW-CMM and similar overweight processes. If agile’s enemy was the SW-CMM in 2001, its enemy today is just variations of agile.

The Agile Manifesto and the Scrum Guide could also be seen as too rigid. Steve thinks it is an “undue fixation on 20-year-old artifacts.” I seem to remember that there were similar arguments over Extreme Programming in the late 1990s and early 2000s – did teams need to follow all twelve practices in the original Kent Beck book? Should we follow the updated practices in the second edition? In the 2010s, there are also questions about whether a Scrum team is following all of the rules in the Scrum Guide. (In other words, we seem capable of many different fixations.)

Steve’s suggestions about the four values in the Agile Manifesto:

Steve also spent some time critiquing the 12 Agile Principles, and these principles make an easier target. Steve found some concepts that were too specific: early and continuous delivery, business and developers work together daily, and constant pace. It may also be unclear whether a preference for “face-to-face interaction” is still valid after our pandemic experiences.

Could we rewrite the agile values and principles? I think that Steve’s comments on the Agile Manifesto’s four values are well-aimed. But I am not sure that we can succeed in creating a satisfactory Manifesto update. Any attempt to rewrite the four values will be difficult, because the poetic and rhetorical simplicity of the Manifesto’s four statements is quite powerful. Can anyone build a new formulation that is equally compelling?

Steve’s comments about the 12 Agile Principles is a bigger stretch. Many people have given more in-depth feedback on the 12 Agile Principles – I particularly like the comments by Bertrand Meyer in his book Agile: The Good, the Hype, and the Ugly. Bertrand points out that some principles are fairly empty statements, others have a lot of overlap, and they don’t have the coherence of the four values. Steve’s critique isn’t broad enough to suggest a way to make a more balanced list of principles.

The historical critique that Steve proposes is pretty good, but incomplete. The early agile community was pretty impatient with other developments in software development, not just the SW-CMM. In some large companies, there was a big investment in tools and training to support UML models, spearheaded by Rational Software Corporation and others. New language and networking technology (Java, C#, Corba, SOAP) were a big pull to adopt formal modeling in some parts of the software process. But I think many of these initiatives lost steam, mostly due to economic issues, industry reorganizations, and the emergence of new web-oriented development environments such as PHP, Python, and Ruby.

[One thought about rewriting the Agile Manifesto. It won’t be easy, and I think a better goal is to completely replace the core principles with something equally compelling. And... instead of a Manifesto with four “we value X over Y” statements, it will need to have a different rhetorical model, in order to make a clean break with the old.]

[My recurring nightmare is a new process model that emphasizes “power and control” (which will appeal to developers and managers), with the model structured as a small cluster of three or four “power areas”, and a master layer that ties them all together (“One ring to rule them all!”). Yow, this sounds pretty grim.]

Sustainability keynote

The Thursday keynote was a bit difficult - a Swedish academic (Emilia Mendes - Blekinge Institute of Technology, Sweden) who gave a discussion of sustainability frameworks and their potential connections to agile. She explained a bit about one of the frameworks: Framework for Strategic Sustainable Development (FSSD). FSSD identifies a number of potential “structural obstacles” – which it uses as a basis for defining what constitutes social sustainability. In the FSSD, a sustainable environment is supposed to meet certain principles about the treatment of individuals – characterized by potential “structural obstacles” (obstacles to health/safety, self-expression, personal development, and non-discrimination).

In the work environment, you should be able to assess (and improve) the social sustainability, and one way is to do an “inventory” of the team climate (the perceptions of individuals about tasks, safety, innovation, and vision.

My question was about top-down versus bottom-up approaches to improving sustainability. The speaker made the point that top-down was most important... setting plans and direction for sustainability at the organization or corporate level. The agile way of course is to focus on the improvements that can be made by teams.

This reminded me of a saying I learned when I worked on high-performance parallel computing back in grad school. The people working on algorithms would say “If log-n won’t do it, screw it!” If you couldn’t figure out how to design your computation in a way that operated in “log(n) time”, then you’re not solving the problem.

In agile development, the main principle of operation is “self-organization” – teams work together to solve problems and evolve their way of working. An agile team might say “If self-organization won’t do it, screw it!” In processes like Scrum, the work of the team is driven by the contents and ordering of the Product Backlog, so *any* initiatives (sustainability or anything else) should be addressed by a group decision on User Stories or Product Backlog Items related to the initiative. “Organizational support” is always welcome, and there will certainly be a need for organizations to share and coordinate ideas across groups, but a properly motivated bottom-up initiative will always get done faster.

Panel session notes

There were two one-hour panel sessions: discussions about interesting issues that have an impact on agile development. The first panel this year was “The Stories We Tell” – the panelists reflected on three different ways that researchers and practitioners create written stories about what they learn in doing software research and development. The second panel was a more wide-ranging discussion of “The Future of Software Engineering.”

Panel: The Stories We Tell

The panel featured five experts in technical communication: researchers, conference organizers, and members of the patterns community.

Steve Fraser (Innoxec) organized the panel and acted as the panel moderator.

Casper Lassenius explained how good research papers tell stories. He presented his own “COVID-D model” of getting papers accepted at the XP conference:

Rebecca Wirfs-Brock explained that the stories in experience reports are more personal than they are in research papers. When explaining real-world experience, objectivity is much less important than it would be in a research paper. Experience report authors definitely need to clearly explain their Context, because readers want to know if the experiences apply to them. But for experience reports, the “V” isn’t Validation, it is Veracity. The authors are explaining the truth as they see it, including how they felt about the experience.

Ken Power also promoted experience report writing, and encourged everyone to write their stories. He pointed out that the biggest obstacle to telling stories is “staring at that blank page” – that is, fear of starting to write. The experience reports tracks of the XP and Agile projects provide support: shepherds to help a new author get through the process: to turn the trove of ideas in your head into a coherent story.

There was a discussion of the issue of “novelty” in our stories.

The panelists agreed that there are many things in common among the three main storytelling formats (experience reports, research papers, and patterns). In many ways, these writing styles are complementary – each form emphasizes different aspects of a story (novelty, creativity, successes and failures, problem solving). Also, the discipline of telling a story in writing will help most people “discover” things that they hadn’t realized they know.

The panelists were invited to comment on how the Agile Manifesto fits into the stories we tell. Steve Fraser [panel chair[ explained that many people have called for major updates to the Manifesto after 20 years – pointing out that Steve McConnell’s XP 2021 keynote talk would be a critique calling for changes to the Agile Manifesto. There was a wide range of opinions:

Rebecca added that in the early days of Extreme Programming, the originators searched for a provocative name for their methodology – the called it “extreme” to get people’s attention. Sometimes you need to tell a story in a bold and gutsy way.

There were also several opinions about how we will tell stories in the future:

Finally, all of the panelists agreed that everyone in the audience should get up the gumption to write – to share their stories in whatever form suits them. It could be experience reports, research papers, patterns, blogs, tweets, videos, or whatever.

What is unique about each story format? (experience reports, research papers, and patterns)

More references related to the Stories panel

Panel: The Future of Software Engineering

The panel featured five experts with a wide range of experience in software technology.

Steve Fraser (Innoxec) organized the panel and acted as the panel moderator.

Each of the panelists had a different spin on the future:

There were many other discussion points throughout the session. Landon worried that we spend more effort building and learning new programming languages, instead of learning programming best practices and good algorithm design. Priya noted that there is an increased “democratization” of software development, with many low-code or no-code environments used by non-programmers. [This is quite similar to the spreadsheet boom that allowed non-programmers to build complex computation frameworks in the early PC era.] Anita pointed to significant advances in the past 30 years of software development: a better focus on architecture, improved practices and tools, agile processes, and even “DevSecOps” to make product development more fluid. But of course we are still plagued with poor quality and crashing software. When a computer games company puts out something that doesn't work, “they just say that maybe they’ll do better on the next game.”

The software industry may need some reform in the development of software products. Landon offered a specific list of actions to change the economic model for software development to reduce company incentives to ship software systems and applications that crash. Landon’s list:

Kati explained the historical path of companies she consults with. When they outsourced most of their development, they were managing for the lowest cost, so they collected many applications from different vendors. “I don’t think it’s a wonder that the whole thing is a mess.” Their current business depends on a lot of bad software, so they are desperate to in-source and hire software developers. In Finland, developers are expensive, they can choose where they want to work, and the companies do not have much experience leading and managing software projects. Some accomodation is needed on all sides. “For this to work,” according to Kati, business specialists need to know more about software and software people need to know more about business.

Bertrand thought about the future by reflecting on how the software engineering field has been impacted by significant advances and changes in the last 30 years. He brought up four technologies that have changed the world: object oriented programming, free and open source software, agile development, and cloud/microservices/DevOps technologies. Bertrand has already written a lot about what he sees as the benefits and drawbacks of many agile practices, and his misgivings about free software are also on the public record. However, Bertrand acknowledged that both of these technology areas have changed the world. Object oriented programming is great, and he thinks that cloud technology will continue to progress. Bertrand cautioned that there may be some challenges with working with cloud computing: “Traditional software engineering wisdom, principles, and modes of reasoning about the world do not completely transpose to that new world.” But Bertrand had an optimistic outlook: he sees software engineering as a field that will continue to advance.

Several of the panelists pointed to topics that should be included in software engineering education and the future of software engineering technologies.

More references related to the Future panel

Other interesting talks

I attended several Industry and Practice talks and research paper presentations that presented some valuable ideas.

Distributed pair programming

Darja Smite (from Blekinge Institute of Technology) presented a research paper on distributed pair programming, including some experience data throughout the pandemic. The authors studied pair programming practices in two companies, and their data shows that some developers actually found it easier to do pair programming in a work-from-home environment. But the pair programming process could be very fatiguing when done remotely for multiple hours. On the whole, the enforced distance of the pandemic reduced the amount of pair programming, but it also increased the interest of team members in learning and adopting new tools. It is still too early to assess the long term impact.

Note: All XP2021 research papers are included in the conference proceedings, which is “open access.” The PDF version of the proceedings is available at this link: XP 2021 Proceedings (PDF).

Code review process

Pawel Kaminski (Founder+Lightning, a UK-based software company) gave an excellent presentation on his company’s approach to code reviews. He has seen too many teams that perform code inspections in large batches, even in a supposedly agile team. His company has made an active effort to avoid the “last-minute pull request” model of doing reviews. He claims it is impossible to do a meaningful review of a 500-line code change: you will always get the answer “it looks OK to me.”

Their approach is to push changes to the main branch as quickly as possible. Any branch that isn’t reviewed and approved within 24 hours is automatically rejected. Developers need to perform code reviews for their peers multiple times a day – starting right after the daily standup meeting. Reviewing 20 lines is still hard, but the reviewer will actually read the code and find issues.

One thing that makes their process more feasible – they have tool support to check for violations of the company’s programming guidelines. This eliminates a lot of doubtful code right away, making the review process more effective.

Teaching agile methodology without code

There was an interesting research paper on education techniques to teach agile methodology to university students – “Using a Low Code Development Environment to Teach the Agile Methodology,” presented by Mary Lebens at Metropolitan State University in Minneapolis. Mary explained the basic challenge of setting up an agile course project for undergraduates: Many of the students lack skills in programming, so they get bogged down with programming details when “building a software product.” They miss out on getting a positive experience with the key skills of collaboration, meetings, and iteration planning.

They wanted a simple, easy-to-program approach for an MIS course on applications development. Her university’s solution is to use a low-code or no-code development environment – they used Microsoft Power Apps development tools, which worked very well. Power Apps can create browser-based apps and mobile apps that can connect to data sources, so it is perfect for building simple data processing applications. The Power Apps tools are part of the Microsoft Office 365 environment that all their students have available, so setup and configuration isn’t necessary. In previous semesters, the students used Mendix for the low-code environment. Students generally liked the new approach to teaching this course, and the post-course student survey showed that all students believed that they understood agile.

Note: All XP2021 research papers are included in the conference proceedings, which is “open access.” The PDF version of the proceedings is available at this link: XP 2021 Proceedings (PDF).

Inter-team coordination in large agile projects

Marthe Berntzen (University of Oslo) presented a research paper on coordination issues – “Coordination Strategies: Managing Inter-team Coordination Challenges in Large-Scale Agile.” The research paper is about the analysis of inter-team coordination in a complex project involving 16 teams – across several Norwegian public transport companies – to build a common information system framework that would support the companies’ operations.

The paper lists several coordination strategies, but the most interesting technique was the creation of short-lived technical teams to solve special problems. Temporary “task-force” teams would have members contributed from several teams – was a useful tactic when the project would run into coordination problems. Berntzen also pointed out that “communicating technical architecture” was a key mechanism to keep the teams on track.

Note: All XP2021 research papers are included in the conference proceedings, which is “open access.” The PDF version of the proceedings is available at this link: XP 2021 Proceedings (PDF).

XP 2022

Next year’s conference will be June 13-17, 2022 in Copenhagen, Denmark. The conference host will be IT University of Copenhagen. We are still not certain what the rules will be, but the team is aiming for face-to-face, with distant presentation possible. More details will be found on the Agile Alliance website: https://www.agilealliance.org/.
Last modified: Jun. 18, 2021