The Risks of Agile Transformation and How to Control Them

“Agile methods are ubiquitous but not necessarily aligned with the way organizations have evolved”, writes GCBA co-founder Haydn Shaughnessy.

Every now and again a term gains such deep traction that it takes over a whole layer of the corporate dialogue. “Value chain” was one such term, reaching back to the 1980s and the publication of Michael Porter’s Competitive Advantage. Business process re-engineering (BPR) ignited change in the 1990s. Agile is the latest. Beginning 20 years ago, it is tightening its grip on the corporate mindset by the day.

These terms are not fads – corporate fashions that come and go. They bite deep and gain permanence, altering how we think about organizational design and behavior.

Haydn Shaughnessy, GCBA Founder & Flow Practitioner

They are significant because something in them fundamentally captures the needs of the day. Porter was writing as companies shed the shackles of state ownership and protectionism and were let loose to compete. BPR won the day because it addressed the benefits of globalization. We know now that agility is going to be a prerequisite for survival as Chinese and other Asian companies use their scale, cost advantage, low-margin pricing and educated ingenuity to dominate global markets.

However, these terms also raise an important and much neglected issue. If every organization had understood Porter’s concepts and applied them skillfully, would we have seen so many great businesses exit the Fortune 2000 or S&P 500? Business concepts are often grasped at rather than embraced, hooked in where possible rather than becoming part of the DNA.

The same goes for agile. Your organization is more than likely applying some form of an agile program or transformation, and I can guarantee all but a handful of you are doing it wrong.

Some leaders of the agile revolution are already calling for people to distance themselves from “agile” as it is now practiced. As agile advocates attempt to spread the religion beyond software development, one of the founders of the movement, Ron Jeffries, said in May this year: “Developers should abandon agile.”

Have you any idea what risks you are running through bad agile implementation? Do you even know how to make that judgment?

What Is Agile?

Agile has its roots in software development methodology. It is a reaction against an old method, referred to as Waterfall, that was largely built on the principles of project management. Those principles derived from weapons’ manufacture in the 1950s or construction projects, both of which had clear “critical paths.”

Once software analysts began designing “big projects,” it turned out that many stakeholders would try to add their requirements into it, and projects became bloated and often difficult to complete. Think back less than a decade, and software projects like Microsoft’s Sharepoint could take three years just to update.

Today the emphasis is on getting features into users’ hands early so they can try them out, and then building on what works. However, this is not “agile”. Other methods, including Lean StartUp, can lay claim to a build-experiment-develop philosophy.

Agile is, in fact, a highly structured approach to software development, embodied in frameworks like Scrum, SAFe, LESS and Scrum Kanban.

Agile is usually marked by a series of phased, two- to three-week sprints that should have very defined value goals. Sprint teams are supposed to be multidisciplinary and autonomous.

Scrum Agile is probably the most common form of agile. It is only fair to point out that the Scrum Alliance reports that the majority of scrum projects are successful and “organizations choose Scrum primarily to deliver more value to the customer. In this year’s [State of Scrum] survey, 85% of respondents say Scrum continues to improve quality of work life.”

Though these views are a powerful endorsement of Scrum Agile, they are at odds with what many agile leaders are seeing. The evidence, anyway, suggests an acute dilution of agile skills as demand has grown.

What Is Going Wrong?

There are several problems with agile that have been created just by virtue of its success. There are others that are more to do with how the nature of work and the economy is changing.

Agile is suffering from the sheer volume of its growth over the past decade. In the U.K., the Agile Business Consortium reported a 33% increase in the uptake of agile project management qualifications between 2015 and 2016. The Dice quarterly tech job alert reported a 14% year-on-year increase in demand for agile software developers, also in 2016. In the U.S., by 2012 agile jobs’ demand was already outstripping candidate supply by 4.6:1.

These figures would be alarming from a quality perspective if they related to jobs that had an established high-grade training environment. But many people qualify as ”agile” after two-day Scrum Master training courses. An undated TechBeacon articlequestions the value of the two-day certifications:

“Opponents contend that agile certification programs have lax vetting, training, and exam requirements. Beyond that, they say, certification only proves that the graduate is book smart, but it reveals nothing about experience or expertise.”

That complaint has a corollary in the domain of agile coaching. People can become agile coaches even if they have followed no previous training in how to coach.

Hiring Issues

In short, it has been impossible for the agile community to keep up with demand. Meanwhile, enterprise CIOs need to hire agile-trained personnel come what may.

Against this background, recruiters are forced to recruit employees who are accredited “agile” even when that means “not very experienced.” Firms accept low-grade qualifications that don’t speak to how a person manages in a software project, let alone how they might fare in the wider business. Boosting the number of people who are agile trained, or talking up an agile transformation, is also a line to feed into the PR machine.

Enterprises are now being encouraged to take agile beyond IT to the whole organization. But is this the right strategy?

The Changing Nature of Work and the Economy

There are fundamental changes taking place in the way we are able to work. Agile methodology doesn’t necessarily capture the benefits of these changes, and to understand why look at the schematic below. It illustrates in broad terms key changes that have been taking place in many organizations.

Traditionally, firms sponsored large waterfall projects – single-entity software builds such as “a platform for the company’s web and e-commerce presence.” Software teams would attempt to design and architect that and build it out over months, anticipating first where problems might lie, and second where value might be accrued. Usually, value was pushed months or years into the distance.

In Scrum Agile, those large projects are broken into sprints, or smaller three-week projects that should have a value goal as their output.

The problem in scrum is that teams might wander off in different directions. Their code may be difficult to integrate, which leads to a lot of rework. But probably worse is the absence of continuous appraisal of the value of the work itself. Value is accepted as part of the ROI from the business, rather than being a topic for debate as evidence emerges.

Smaller Work Units

In the new phase, work is broken down into much smaller units. Developers more or less code directly to a main code line (via a testing pipeline), so there are no code collisions from the work of different teams. And ideally, the much smaller units of work would be created with an eye on getting them to customers to test their value early on.

As work units are small, not much is lost if they don’t stick with customers. Discussions about value can become more commonplace as nobody has so much as stake that they will hide a loss or fail point.

The importance of these changes cannot be overstated. They are about the very fundamentals of how you organize around value, or not.

When people talk about agile, the work organization they are thinking about is the second of those blocks in the diagram above.

But times have moved on, and we now need to think of this new world. Not only is it a place where tasks are broken down to a lower level. It turns out that increasingly, the amount of code needed for a given feature is also reducing. In some cases a good IT department can give business people the building blocks to create their own services without code. And in all cases it should lead to a reappraisal of value.

The Risks of Being Old Agile

You need three cultures. Many organizations cannot get away from the need to embrace multiple work cultures, in this case legacy, agile, and devops or Flow. They want to move away from legacy, but legacy is there for a reason. They are trying to become agile and maybe implementing scrum, and that carries risks. They really need to move on, but there is no point in leaving agile behind. The value-centric nature of good devops, or what my colleagues and I call Flow, can improve scrum and agile. But the organization needs to know how to maximize all three.

Low qualifications. There are problems of quality that risk managers need to be aware of. Software is integral to how we work, how we create value, and what we sell. Coding may not be difficult, but the proof of time and experience is that organizing this value process is challenging. It becomes more difficult if your organization is building up inexperience in the form of two-day certified “experts” and coaches that don’t have human development experience.

The effect of being rules bound. Equally, agile can be deployed in a rules-bound way, despite the best intentions of people who provide certification. Employees do not get the attention and space they need in order to become more autonomous; and they tend to gravitate back towards old methodologies and behaviors as a safe haven.

Being software-led when business is becoming software-independent. Companies are becoming more software-driven, but the move within software is to create services that business people can define without code know-how. The potential exists for companies to become a community of start-ups. What tends to happen instead is that the business looks to emulate scrum techniques and cast around itself rules and processes that are not necessarily helpful.

What Can Be Done?

Risk managers who have the enterprise in their care need to make themselves better informed about how the enterprise creates value. As important, they need to know how the enterprise is changing its value-creating methods. Some of the steps the less experienced risk manager can take are:

  1. Audit the coaching experience of people appointed to coach your change teams; not to make an aggressive stance, but to figure out where additional training is needed.
  2. Likewise, audit the onboarding of people with agile certification and work with HR to figure out how much experience of agile working that adds up to, so you can nurture people over the deficits
  3. Review the IT portfolio with a Portfolio Wall. (Find examples of Portfolio Walls and a discussion of cycle time in 12 Steps to Flow) That is, lay out all the IT work-in-progress on a wall and lead a discussion about the value of projects, especially those that are swallowing large amounts of resources.
  4. Ask your IT Director about the use of retrospective meetings (stand ups) and the pattern of issues and challenges that is emerging from them. Go as far as to ask for an analysis of this.
  5. Conduct a risks-and-issues survey among staff in one-on-one interviews.
  6. Check if you have feedback loops that formally channel customer issues into different parts of your work process, or are you taking value for granted? The feedback and retrospectives can tell you where your agile program needs more support.
  7. Ask for a measure of your IT cycle time, the typical time it takes to complete IT tasks. Ask for regular updates. You want to see this trend downwards from three weeks to two days, over time.
  8. Gently audit work in progress for value, asking whether the original assumptions still hold true and what measures are being used to validate that.

These simple steps will help you begin to build a picture of how change is being managed and in which direction. They will also allow you to make an initial assessment of whether your company is managing good agile or high-risk agile. Don’t wait too long to find out.