SPLASH Workshop - Technical Debt
October 20, 2014
Technical Debt is the cost of the work you need to do because you
were in a hurry -- some parts of the design
and code were not clean.
Technical Debt is a metaphor
- it helps us think about software development choices
- there is no "precise formula" for technical debt
Some discussion of ideas and experiences from workshop participants:
- Working for old companies with new requirements (in the healthcare industry)
- bringing old COBOL systems into the 21st century
- moving to Java
- developing systems that work with individuals instead of just with employers
- software that talks to diverse systems
- we cut corners all the time
- we often have to work harder, because we didn't build the right system
- working in new company in an old market (telecom)
- new products and customers (IP telephony, Vonage and others)
- some work is done by contractors -> and they build what we say (but
it would be better if they did some thinking -- to build something better)
- making deployment more convenient -- one of the things to do better
- more of a DevOps focus rather than just an application focus
- how do you balance?
- in the aviation business -- we build and support a lot of simulators
- and we take a lot of shortcuts
- we do simulation so we can test parts of the design early
- our software has a long lifetime, and we have to support both
changes and new features over that lifetime
- hiring new staff increases technical debt
- one issue in the development process: do we suspend a Sprint in order to merge things and clean up?
- it is a challenge to do this, because it is a question of who is paying for the work
- will the budget support clean up work??
- we have some old code bases
- it isn't easy to move to modern languages (rewriting old assembler code
that was added for efficiency)
- I help developers learn about automated tests to be used to keep the software good
- people don't want to learn from previous experience
- they do want to use the latest technology
- we should invest more time (and training) to make the design better
- are there different kinds of technical debt? (depending on the process used to create the design and code?)
- one of the questions we have to answer is: Why don't we just build a scalable solution in PHP? Aaaackk...
- discussion about another metaphor: volunteer firemen who set fires... to prove that they are valuable
- there is a temptation to do this in software development -- because when
we have a product that is "too well maintained", we never get a chance
to redo it (even if reengineering is really needed)
- discussion of "large telecom" systems - and barriers to reuse in that domain
- development teams say: "we don't want you to reuse our code - it will
reduce our flexibility in the future"
- so another development team will "copy and rename" parts of the
code - this is "making a fork of the original code"
- reuse requires some trust between organizations
- sometimes the cost of reengineering a software system can be avoided
- think about solutions other than writing more software
- for example, do some of the new functionality in the database instead
- in recent years, there is less investment in the existing
- think of it as "less interest in keeping the lights on"
- or "dwindling knowledge" of teh details of the systems that
the company needs to run (original developers have retired, or
development has been moved to another country)
- how good is the code base?
- are we following our own coding guidelines?
- when we acquire another company, should we assess the technical debt
of their product code?
- maybe the acquired company's product code is really poor -- the equivalent
of a science fair project, not really a strong product
- part of the job of working in a complex company: bridging the gap
between the innovators and the people who keep the system working
- innovators will create something cool, but they abandon it quickly
- other people (who have more of a long term product focus) need to
be in a position to understand and support the product
- Technical debut: strategic implementation choice or sloppy design?
- Do we pay back debt over time, or does our debt accumulate?
- Are there different kinds of technical debt?
- Are there good metrics for technical debt?
- How do we "welcome change" even late in the process? (Agile values)
Some metrics about software quality:
- defect density
- cycle time (time to implement a new feature)
- code quality (is the code clean / and is it functionality that
the customers need)
Testing (or at least the process around retesting code after
making code changes) creates an issue for making small changes:
- we are not allowed to change some things unless we involve a
lot of people in reviews and testing
- so some code improvements get delayed, sometimes for a long time
How can we get people to pay attention to things that could be serious?
- is it as bad as the problems of batteries catching fire on the Boeing 787?
- if the issue appeared on the front page of the newspaper, would
it be a public relations problem for the company?
There is less and less interest in paying the cost of
building systems with high reliability
- telecom standard -- 5-nines = available 99.999% of the time = can only
be out of service for 5 minutes per year
- to get this to this level, there is significant design cost, more testing,
and other investment
Suppose that one of my underlying development assumptions has changed - is
this technical debt?
- it depends on what you do next!
- it is OK to do something quick and dirty
- as long as we commit to revisit the design later (when we
- we need to do adaptive replanning
- some people might be tempted to delete things from the list of
remaining technical debt issues that haven't been worked on for
six months or more -- but this is dangerous
List of potential questions to explore
- How to make Technical Debt visible?
- How to measure Technical Debt?
- How to mitigate Technical Debt?
- What to teach the next generation of developers to do better? (how to
give them experience to improve what they do)
- We have already lived through the OO revolution and the Agile revolution.
Are there any patterns we can use to make a technical debt revolution?
- Can we manage Technical Debt? How?
- Is Technical Debt inevitable? Do we just live with it?
- make software with "expiration dates" (to make sure the software gets
rewritten -- this helps deal with issues of aging software, increasing
software complexity, and setting up the necessary social structure that
cares for the software)
- ensure we look at the "whole system", not just the software piece
- including database, infrastructure, configurations, and people
- note that "refactoring the data in a database" is a very hard job
- especially dealing with "dirty data"
- do we need special strategies for real-time systems?
- power consumption?
In thinking about legacy code and technical debt, we need to consider
both the "engineer view" and the "decision-maker view"
- how do you prioritize the work to be done?
- what are the costs of reducing technical debt?
There are downstream consequences of fixing technical debt
- it is like what happens when you update Firefox web browser
- which seems to happen at least once a month (to fix security issues)
- and it always triggers updating Flash as well
- another example: Windows XP end-of-life
- migrating from one version of Java to another (going from Java 4 to Java 5 was not easy)
- 5 years ago, we had issues with IBM WebSphere / it would work with the
latest version of the Xerces XML parser
- are there any situations where Technical Debt becomes a security issue?
Agile and its interactions with Technical Debt
- does Agile improve the technical debt situation?
- or is it worse?
An observation about Agile:
- Agile is a colleciton of ceremonies and practices
- and many of them have been around for 20-30 years
- but they are now bundled together
XP technical practices
- in particular, pair programming is a good way to do "continuous code inspection"
- it may not be the best thing for the computer science field,
but many certification programs help people get employed
- because some Human Resources departments set unreasonable job requirements
- [Certification might be a good topic for a SPLASH workshop]
Is Technical Debt another word for quality?
- probably not... technical debt may just be one piece of the quality picture
- it is the "quality of the path of creating an evolving application"
- (we talk about software development using a standard saying: "it isn't
the destination, it is the journey")
Final question list for the main discussion
- Mitigation of Technical Debt and strategies (6 votes)
- Managing Technical Debt; is it inevitable; does it matter; what
are the costs (5 votes)
- How to make Technical Debt visible; how to measure (4 votes)
- Agile and Technical Debt (3 votes)
- How and what to teach the next generation about Technical Debt (2 votes)
- Can we use the Technical Debt metaphor for other choices? (acquiring
other companies, decisions about retiring software products) (1 vote)
- Is Technical Debt about the infrastructure / substrate? it might
not be so much about the "apps" (which can be throw-away software) (0 votes)
1. Mitigation of Technical Debt
- strategies for keeping it under control
- (and not creating it in the first place)
Can we come up with a cost of doing a better solution?
Will this help us make better choices?
What doesn't work:
- Over-engineered architecture
- (instead of building business value and learning - in an evolutionary process)
- [this is "architectural hubris" - architect acts as if he/she knows everything at the start]
Better: working together - figure out what we can ship this year (build
an architecture and design to support it)
It is useful to share information about the cost of the shortcuts
- instead of hiding the shortcuts
Question: Is "forking" guaranteed to create Technical Debt?
- Answer: Yes.
- avoid duplication is an important mitigation strategy
Another good strategy: Keep a list (Technical Debt backlog)
- this is useful even if you aren't using Agile
(The idea of "identifying individual Technical Debt issues" is
consistent with Philippe Kruchten's "Four Color Backlog".
The four colors are:
This is a useful way to make sure that technical debt reduction
activities aren't completely lost in the list of new customer-facing
- Green - stories that deliver customer value (normal backlog items)
- Yellow - architecture stories - don't deliver customer value directly,
but you need them to be able to implement green stories
- Red - bug fixes - high priority, fixing developer blunders
- Black - technical debt stories - deferred cleanup of design shortcuts
More about "keep a list of Technical Debt issues":
- it is good to keep the list separate from the main "product backlog" (becuase it makes the technical debt more visible)
- it is OK to copy the items into the Scrum backlog in the planning process
- do you create stories for each technical debt item?
- No - not until
they are melded into the Scrum backlog
- writing items in story form can be an obstacle,
and we want the Technical Debt list to be as
complete as we can make it
What is in the Technical Debt list?
- design/coding shortcuts
- test shortcuts
- experimental and exploratory development items
- open source updates
- it is an interesting legal item
- when you use some open source
modules, and you make changes and improvements to the modules,
you are legally obligated to make your changes publicly
available -- usually by checking
them back into the open source repository
- if you don't do this, you can be sued...
Does the skill set and experience of your software team
have an impact on the level of Technical Debt?
- A mitigation strategy: get a team of experienced staff members
(or young rock stars)
- If that isn't possible, make sure the team invests time to
learn -- expand their skills and knowledge
More on mitigation strategies
Include scenario-based modles and issue tracking
- instead of just focusing on "features"
- developers and managers should also have scenario-based
artifacts in their planning process == user stories, use cases, test scenarios
- helps you focus on essential improvements
- it's an architectural view of the system
Another good mitigation technique: "engage testers"
Question: are testers part of your team, or are they a separate
- it is difficult to have a combined role of developer/tester (developers
usually don't have the best testing skills)
- there are times when we want testers to be independent
- but we also want to share information about critical scenario
(and testers understand scenarios better than developers)
We need continuous test
- everyone needs info about end-to-end behavior early
Follow good encapsulation principles; write clean code
2. Managing Technical Debt
One criticism by managers who don't believe in Technical Debt:
- we have it, what can we do about it?
- "There they go, the techies just want to play with
One management strategy:
- plan for 80% effort on new feature work, 20% effort on refactoring
- in other words, set a budget for improvement activities
A management strategy that creates problems:
- Full traceability
- It can be an obstacle to managing Technical Debt
- because you focus on feature work over code improvement
- "full traceability" may force any developer to "trace their
proposed work" back to a customer feature request or specific
item in the requirements documentation
- it takes more effort to justify each code change
- it's a "compliance mindset" (Sarbanes-Oxley and others...)
- we could try "grouping" our improvements
- should we keep a Technical Debt burndown chart?
When something blows up
- it is better to blow up early (no later than User Acceptance Testing,
much better than not working in the field)
- we want to make people aware of how high the technical debt is....
Question: Is it professional to "make technical debt worse" to
make it visible?
Risk management process can help:
- the process starts with identifying potential risks
- keep watching -- risk management is a continuous process
- watching for technical debt should be part of risk management
Some good books on "teaching design thinking":
- Object Design by Rebecca Wirfs-Brock and Alan McKean
- good book on design practices - based on Responsibility-Driven Design
- Object Oriented Design Heuristics by Arthur Riel
- good "rules of thumb" for flexible designs
- Refactoring by Martin Fowler
- introduced the idea of "code smells"
- Refactoring to Patterns by Joshua Kerievsky
- refactoring with a goal -- to create a design that is more maintainable,
because it embodies good design patterns (so future changes can be
made in a way that is consistent with the patterns)
3. How to make Technical Debt visible
How to "identify" technical debt:
- design reviews and code reviews (using humans to review)
- static code analysis tools (Coverity, Klocwork, Sonar, ...)
Static code analysis tools find some technical debt problems,
especially in a large legacy code base.
Watch out for local rewards...
- One manager said "I want to be off of the list of
top 10 buggy modules"
- so I asked the developers to split every module into two pieces
- This is not good for the overall quality, but it
makes my modules look better
Peer-to-peer collaboration works better than central
command and control (in many cases)
Of course, the management style of the organization
is reflected in the architecture of the solution (Conway's Law)
Do you have people who can think about larger issues and share
things across the teams?
Changing platforms -- this exposes many code quality issues
- one idea: compile all code with 2 different compilers (well
before any platform changes)
- a good way to find code problems cheaply
Metric: How much is your build/test/deploy cycle slowed by
Invest in early unit test
- if it is hard, you have some undiscovered technical debt
- look at the book by Michael Feathers: Working Effectively
With Legacy Code
More on making technical debt visible
Keep a list of Technical Debt items
- it doesn't always tell you how much work is needed
- but it is a start
Look at future feature -- compute the development savings
if you invest in fixing some technical debt first...
Watch out -- you might not need to fix the technical debt
- For example, if you can just go out and "buy a new solution",
you don't need to fix the old code
There are good things and bad things about the Sonar approach
to measuring technical debt
Sonar counts several things:
- violations of coding standards (it gets this information from
tunable static code analysis)
- duplicate code
- McCabe complexity of each module
- unit test coverage
Why is this not so good as a technical debt measurement?
- Violations of coding standards is a weak indicator of code-level
technical debt. Duplicate code is a pretty important technical debt
measure. McCabe complexity is probably only weakly correlated with debt.
Unit test coverage is not too bad, but this metric imposes a
particular development view...
- What is missing in the metric? Design shortcuts -- impossible
to measure in an automated tool.
In the code inspection process, you don't always have to see
all of the code to make a partial assessment of problems:
- look at the metadata
- that is, in C++, read the header files (look at the interfaces, correct
use of const, proper copy constructors, public vs. private data)
- in Java, look at the Java class signatures (in other words, run
Javadoc, but ignore the comments -- only look at the data and
More things to look at in code inspections:
- look at dynamic references
- look at references to other classes (too much coupling, for example)
If you do a performance test - can you find Technical Debt
in dynamic resource consumption?
Make sure you have performance tests and load tests in place
In the analysis of test results, be aware of the root cause
of the problems.
Question: Who do you make Technical Debt visible to?
- stakeholders: developers, project managers, business people, architects,
production operations, testers
- explain the situation differently for different audiences
- based on what they value
- you won't be successful in all cases - it is hard to build trust
There is a lot of "underinvestment" in development
We just "bolt on" changes to an existing code base, creating
a "house of cards" (from a Guardian article about a bank software crisis)
5. What do we teach the next generation?
it is a process of "educating"
people tend to think local not global
we need to convince students that TD is important
and convince faculty that it is a topic that needs to be
in the curriculum
it is a challenge to teach the right thing -- not enough
need projects to get experience
One teaching experience: doing a group project to build a small
system to fill out a tax form
Problem: most problems have a scope that is too small (students
would prefer to work on the problem on their own, not as a team)
but it is better experience for them to work in a 5-6 person team
One idea -- instead of a big multi-week project, use something
like the old OOPSLA Design Fest
- one day event
- team of 5-6 people
- roles: team lead, recorder, designers
- work on a problem as a team, have a lot of discussions, publish
the solution on the web
Another idea -- Alistair Cockburn's coffee machine exercise
- In this exercise, the instructor interrupts the teams several
times with new requirements
- the solution isn't important, it is the exerience of being
interrupted (which is more like the real world)
Hackathons -- good for teamwork
- many hackathons do more "non programming based development" -- create
- this is a really good thing -- helps people to expand their
thinking about design
Working with legacy code
- contributing to an open source project
- in the corporate world, create a pool of "internal open source" projects
What happens when you "learn the right way in school", but when
you go to the work works, they aren't doing it right?
There are things that individual developers can "do on their own"
to deal with technical debt:
- create a personal technical debt list
- follow clean code practices - incremental improvement
- do "self-inspection" code inspections (like Watts Humphrey's Personal
- use "xDD" practices (test driven, domain driven, behavior driven, etc.)
- refactoring -- "learn how, get practice"