Technical Debt is the cost of the work you need to do because you were in a hurry -- some parts of the design and code were not clean.

• Technical Debt is a metaphor
  • it helps us think about software development choices
  • there is no "precise formula" for technical debt

• Working for old companies with new requirements (in the healthcare industry)
• bringing old COBOL systems into the 21st century
• moving to Java
• developing systems that work with individuals instead of just with employers
• software that talks to diverse systems
• we cut corners all the time
• we often have to work harder, because we didn't build the right system up front

• working in new company in an old market (telecom)
• new products and customers (IP telephony, Vonage and others)
• some work is done by contractors -> and they build what we say (but it would be better if they did some thinking -- to build something better)
  • making deployment more convenient -- one of the things to do better
  • more of a DevOps focus rather than just an application focus
  • how do you balance?

• in the aviation business -- we build and support a lot of simulators
• and we take a lot of shortcuts
• we do simulation so we can test parts of the design early
• our software has a long lifetime, and we have to support both changes and new features over that lifetime
• hiring new staff increases technical debt
• one issue in the development process: do we suspend a Sprint in order to merge things and clean up?
  • it is a challenge to do this, because it is a question of who is paying for the work
  • will the budget support clean up work??
• we have some old code bases
• it isn't easy to move to modern languages (rewriting old assembler code that was added for efficiency)
• I help developers learn about automated tests to be used to keep the software good

• people don't want to learn from previous experience
• they do want to use the latest technology
• we should invest more time (and training) to make the design better
• are there different kinds of technical debt? (depending on the process used to create the design and code?)
• one of the questions we have to answer is: Why don't we just build a scalable solution in PHP? Aaaackk...
• discussion about another metaphor: volunteer firemen who set fires... to prove that they are valuable
  • there is a temptation to do this in software development -- because when we have a product that is "too well maintained", we never get a chance to redo it (even if reengineering is really needed)

• discussion of "large telecom" systems - and barriers to reuse in that domain
  • development teams say: "we don't want you to reuse our code - it will reduce our flexibility in the future"
  • so another development team will "copy and rename" parts of the code - this is "making a fork of the original code"
• reuse requires some trust between organizations
• sometimes the cost of reengineering a software system can be avoided
  • think about solutions other than writing more software
  • for example, do some of the new functionality in the database instead
• in recent years, there is less investment in the existing code infrastructure
  • think of it as "less interest in keeping the lights on"
  • or "dwindling knowledge" of teh details of the systems that the company needs to run (original developers have retired, or development has been moved to another country)

• how good is the code base?
• are we following our own coding guidelines?
• when we acquire another company, should we assess the technical debt of their product code?
  • maybe the acquired company's product code is really poor -- the equivalent of a science fair project, not really a strong product
• part of the job of working in a complex company: bridging the gap between the innovators and the people who keep the system working
  • innovators will create something cool, but they abandon it quickly
  • other people (who have more of a long term product focus) need to be in a position to understand and support the product

Some questions:

• Technical debut: strategic implementation choice or sloppy design?
• Do we pay back debt over time, or does our debt accumulate?
• Are there different kinds of technical debt?
• Are there good metrics for technical debt?
• How do we "welcome change" even late in the process? (Agile values)

Some metrics about software quality:

• size
• defect density
• cycle time (time to implement a new feature)
• code quality (is the code clean / and is it functionality that the customers need)

Testing (or at least the process around retesting code after making code changes) creates an issue for making small changes:

• we are not allowed to change some things unless we involve a lot of people in reviews and testing
• so some code improvements get delayed, sometimes for a long time

How can we get people to pay attention to things that could be serious?

• is it as bad as the problems of batteries catching fire on the Boeing 787?
• if the issue appeared on the front page of the newspaper, would it be a public relations problem for the company?

There is less and less interest in paying the cost of building systems with high reliability

• telecom standard -- 5-nines = available 99.999% of the time = can only be out of service for 5 minutes per year
• to get this to this level, there is significant design cost, more testing, and other investment

Suppose that one of my underlying development assumptions has changed - is this technical debt?

• it depends on what you do next!
• it is OK to do something quick and dirty
• as long as we commit to revisit the design later (when we know more)
• we need to do adaptive replanning
• some people might be tempted to delete things from the list of remaining technical debt issues that haven't been worked on for six months or more -- but this is dangerous

List of potential questions to explore

• How to make Technical Debt visible?
• How to measure Technical Debt?
• How to mitigate Technical Debt?
• What to teach the next generation of developers to do better? (how to give them experience to improve what they do)
• We have already lived through the OO revolution and the Agile revolution. Are there any patterns we can use to make a technical debt revolution?
• Can we manage Technical Debt? How?
• Is Technical Debt inevitable? Do we just live with it?

Some strategies

• make software with "expiration dates" (to make sure the software gets rewritten -- this helps deal with issues of aging software, increasing software complexity, and setting up the necessary social structure that cares for the software)
• ensure we look at the "whole system", not just the software piece
  • including database, infrastructure, configurations, and people
  • note that "refactoring the data in a database" is a very hard job
  • especially dealing with "dirty data"
• do we need special strategies for real-time systems?
• power consumption?

In thinking about legacy code and technical debt, we need to consider both the "engineer view" and the "decision-maker view"

• how do you prioritize the work to be done?
• what are the costs of reducing technical debt?

There are downstream consequences of fixing technical debt

• it is like what happens when you update Firefox web browser
  • which seems to happen at least once a month (to fix security issues)
  • and it always triggers updating Flash as well
• another example: Windows XP end-of-life
• migrating from one version of Java to another (going from Java 4 to Java 5 was not easy)
• 5 years ago, we had issues with IBM WebSphere / it would work with the latest version of the Xerces XML parser
• are there any situations where Technical Debt becomes a security issue?

Agile and its interactions with Technical Debt

• does Agile improve the technical debt situation?
• or is it worse?

An observation about Agile:

• Agile is a colleciton of ceremonies and practices
• and many of them have been around for 20-30 years
• but they are now bundled together

XP technical practices

• in particular, pair programming is a good way to do "continuous code inspection"

Certification...

• it may not be the best thing for the computer science field, but many certification programs help people get employed
• because some Human Resources departments set unreasonable job requirements
• [Certification might be a good topic for a SPLASH workshop]

Is Technical Debt another word for quality?

• probably not... technical debt may just be one piece of the quality picture
• it is the "quality of the path of creating an evolving application"
• (we talk about software development using a standard saying: "it isn't the destination, it is the journey")

Final question list for the main discussion

1. Mitigation of Technical Debt and strategies (6 votes)
2. Managing Technical Debt; is it inevitable; does it matter; what are the costs (5 votes)
3. How to make Technical Debt visible; how to measure (4 votes)
4. Agile and Technical Debt (3 votes)
5. How and what to teach the next generation about Technical Debt (2 votes)
6. Can we use the Technical Debt metaphor for other choices? (acquiring other companies, decisions about retiring software products) (1 vote)
7. Is Technical Debt about the infrastructure / substrate? it might not be so much about the "apps" (which can be throw-away software) (0 votes)

1. Mitigation of Technical Debt

• strategies for keeping it under control
• (and not creating it in the first place)

Can we come up with a cost of doing a better solution? Will this help us make better choices?

What doesn't work:

• Over-engineered architecture
• (instead of building business value and learning - in an evolutionary process)
• [this is "architectural hubris" - architect acts as if he/she knows everything at the start]

Better: working together - figure out what we can ship this year (build an architecture and design to support it)

It is useful to share information about the cost of the shortcuts

• instead of hiding the shortcuts

Question: Is "forking" guaranteed to create Technical Debt?

• Answer: Yes.
• avoid duplication is an important mitigation strategy

Another good strategy: Keep a list (Technical Debt backlog)

• this is useful even if you aren't using Agile

(The idea of "identifying individual Technical Debt issues" is consistent with Philippe Kruchten's "Four Color Backlog". The four colors are:

• Green - stories that deliver customer value (normal backlog items)
• Yellow - architecture stories - don't deliver customer value directly, but you need them to be able to implement green stories
• Red - bug fixes - high priority, fixing developer blunders
• Black - technical debt stories - deferred cleanup of design shortcuts

More about "keep a list of Technical Debt issues":

• it is good to keep the list separate from the main "product backlog" (becuase it makes the technical debt more visible)
• it is OK to copy the items into the Scrum backlog in the planning process
• do you create stories for each technical debt item?
  • No - not until they are melded into the Scrum backlog
  • writing items in story form can be an obstacle, and we want the Technical Debt list to be as complete as we can make it

What is in the Technical Debt list?

• duplication
• design/coding shortcuts
• test shortcuts
• experimental and exploratory development items
• open source updates
  • it is an interesting legal item
  • when you use some open source modules, and you make changes and improvements to the modules, you are legally obligated to make your changes publicly available -- usually by checking them back into the open source repository
  • if you don't do this, you can be sued...

Does the skill set and experience of your software team have an impact on the level of Technical Debt?

• A mitigation strategy: get a team of experienced staff members (or young rock stars)
• If that isn't possible, make sure the team invests time to learn -- expand their skills and knowledge

More on mitigation strategies

Include scenario-based modles and issue tracking

• instead of just focusing on "features"
• developers and managers should also have scenario-based artifacts in their planning process == user stories, use cases, test scenarios
• Why?
  • helps you focus on essential improvements
  • it's an architectural view of the system

Another good mitigation technique: "engage testers"

Question: are testers part of your team, or are they a separate organization?

• it is difficult to have a combined role of developer/tester (developers usually don't have the best testing skills)
• there are times when we want testers to be independent
• but we also want to share information about critical scenario (and testers understand scenarios better than developers)

We need continuous test

• everyone needs info about end-to-end behavior early

Follow good encapsulation principles; write clean code

2. Managing Technical Debt

• we have it, what can we do about it?

• "There they go, the techies just want to play with more toys/tools."

One management strategy:

• plan for 80% effort on new feature work, 20% effort on refactoring
• in other words, set a budget for improvement activities

A management strategy that creates problems:

• Full traceability
• It can be an obstacle to managing Technical Debt
• because you focus on feature work over code improvement
  • "full traceability" may force any developer to "trace their proposed work" back to a customer feature request or specific item in the requirements documentation
• it takes more effort to justify each code change
• it's a "compliance mindset" (Sarbanes-Oxley and others...)
• we could try "grouping" our improvements
• should we keep a Technical Debt burndown chart?

When something blows up

• it is better to blow up early (no later than User Acceptance Testing, much better than not working in the field)
• we want to make people aware of how high the technical debt is....

Question: Is it professional to "make technical debt worse" to make it visible?

Risk management process can help:

• the process starts with identifying potential risks
• keep watching -- risk management is a continuous process
• watching for technical debt should be part of risk management

Some good books on "teaching design thinking":

• Object Design by Rebecca Wirfs-Brock and Alan McKean
  • good book on design practices - based on Responsibility-Driven Design
• Object Oriented Design Heuristics by Arthur Riel
  • good "rules of thumb" for flexible designs
• Refactoring by Martin Fowler
  • introduced the idea of "code smells"
• Refactoring to Patterns by Joshua Kerievsky
  • refactoring with a goal -- to create a design that is more maintainable, because it embodies good design patterns (so future changes can be made in a way that is consistent with the patterns)

3. How to make Technical Debt visible

How to "identify" technical debt:

• design reviews and code reviews (using humans to review)
• static code analysis tools (Coverity, Klocwork, Sonar, ...)

Static code analysis tools find some technical debt problems, especially in a large legacy code base.

Watch out for local rewards...

• One manager said "I want to be off of the list of top 10 buggy modules"
• so I asked the developers to split every module into two pieces
• This is not good for the overall quality, but it makes my modules look better

Peer-to-peer collaboration works better than central command and control (in many cases)

Of course, the management style of the organization is reflected in the architecture of the solution (Conway's Law)

Do you have people who can think about larger issues and share things across the teams?

Changing platforms -- this exposes many code quality issues

• one idea: compile all code with 2 different compilers (well before any platform changes)
• a good way to find code problems cheaply

Metric: How much is your build/test/deploy cycle slowed by Technical Debt?

Invest in early unit test

• if it is hard, you have some undiscovered technical debt
• look at the book by Michael Feathers: Working Effectively With Legacy Code

More on making technical debt visible

Keep a list of Technical Debt items

• it doesn't always tell you how much work is needed
• but it is a start

Look at future feature -- compute the development savings if you invest in fixing some technical debt first...

Watch out -- you might not need to fix the technical debt

• For example, if you can just go out and "buy a new solution", you don't need to fix the old code

There are good things and bad things about the Sonar approach to measuring technical debt

Sonar counts several things:

• violations of coding standards (it gets this information from tunable static code analysis)
• duplicate code
• McCabe complexity of each module
• unit test coverage

Why is this not so good as a technical debt measurement?

• Violations of coding standards is a weak indicator of code-level technical debt. Duplicate code is a pretty important technical debt measure. McCabe complexity is probably only weakly correlated with debt. Unit test coverage is not too bad, but this metric imposes a particular development view...
• What is missing in the metric? Design shortcuts -- impossible to measure in an automated tool.

In the code inspection process, you don't always have to see all of the code to make a partial assessment of problems:

• look at the metadata
• that is, in C++, read the header files (look at the interfaces, correct use of const, proper copy constructors, public vs. private data)
• in Java, look at the Java class signatures (in other words, run Javadoc, but ignore the comments -- only look at the data and function signatures)

More things to look at in code inspections:

• look at dynamic references
• look at references to other classes (too much coupling, for example)

If you do a performance test - can you find Technical Debt in dynamic resource consumption?

Make sure you have performance tests and load tests in place

In the analysis of test results, be aware of the root cause of the problems.

Question: Who do you make Technical Debt visible to?

• stakeholders: developers, project managers, business people, architects, production operations, testers
• explain the situation differently for different audiences
• based on what they value
• you won't be successful in all cases - it is hard to build trust

There is a lot of "underinvestment" in development

We just "bolt on" changes to an existing code base, creating a "house of cards" (from a Guardian article about a bank software crisis)

5. What do we teach the next generation?

it is a process of "educating"

people tend to think local not global

we need to convince students that TD is important

and convince faculty that it is a topic that needs to be in the curriculum

it is a challenge to teach the right thing -- not enough academic awareness

need projects to get experience

One teaching experience: doing a group project to build a small system to fill out a tax form

Problem: most problems have a scope that is too small (students would prefer to work on the problem on their own, not as a team)

but it is better experience for them to work in a 5-6 person team

One idea -- instead of a big multi-week project, use something like the old OOPSLA Design Fest

• one day event
• team of 5-6 people
• roles: team lead, recorder, designers
• work on a problem as a team, have a lot of discussions, publish the solution on the web

Another idea -- Alistair Cockburn's coffee machine exercise

• In this exercise, the instructor interrupts the teams several times with new requirements
• the solution isn't important, it is the exerience of being interrupted (which is more like the real world)

Hackathons -- good for teamwork

• many hackathons do more "non programming based development" -- create mashups
• this is a really good thing -- helps people to expand their thinking about design

Working with legacy code

• contributing to an open source project
• in the corporate world, create a pool of "internal open source" projects

What happens when you "learn the right way in school", but when you go to the work works, they aren't doing it right?

There are things that individual developers can "do on their own" to deal with technical debt:

• create a personal technical debt list
• follow clean code practices - incremental improvement
• do "self-inspection" code inspections (like Watts Humphrey's Personal Software Process)
• use "xDD" practices (test driven, domain driven, behavior driven, etc.)
• refactoring -- "learn how, get practice"