December 7, 2013

The concept of Agile Analytics is a close parallel to Agile software development methods.

In a recent blog post at, Anand Krishnaswamy compares Agile Analytics to Agile project management. Earlier this year the TXO Blog also discussed Agile Analytics, and Mr Krishnaswamy’s article is therefore a very interesting read.

The concept of Agile Analytics is a good parallel to Agile software development methods. In the age of Big Data, these two disciplines are tightly interwoven, as analysts depend on computer technology to collect, store, handle and examine structured and unstructured data. Meanwhile, analysts and data scientists are needed to design these systems, feed them content, interpret results, and report findings to decision makers.

Actually, analytics and development might be even more similar than Mr Krishnaswamy suggests. In IT projects, user requirements aren’t always as clear as you would think, and data sources are sometimes unknown or ambiguous at the outset of the project. This is particularly true when developing internet-based solutions that are targeted at a wider population of users, and never truer than for start-ups creating brand-new user experiences.

In analytics projects, the end-user requirements may also be vague and high-level. A customer might ask you to find out how people are talking about the company’s brand in social media, without any idea if it’s positive or negative mentions, the amount of data available, or which social media channels are relevant to study. In another case you may be asked to examine thousands of internal documents to find patterns indicating significant business trends. As you work through the project, you might discover that the documents are in a wide range of file formats, sometimes incompatible with today’s software. Or you could discover patterns that the project owners had no idea existed, and end up studying other topics than originally planned.

Mr Krishnaswamy points out that an Agile project is all about failing fast. This way you avoid building out full-scale solutions that don’t give the answers you’re looking for, or basing a project on incompatible data sources. In our experience, to get through the try-fail stage and arrive at a workable solution, the analytics team will need to work closely with both the business side and IT.

In a typical Agile analytics project, the initial steps may include:

  • Get the customer’s high-level needs and goals
  • Transform customer needs to searches / queries
  • List possible data types and sources
  • Identify needs for file transfers, document conversions, scanning/OCR
  • Hook up to data sources and get sample data
  • Run queries and analyze results from samples
  • Tune queries, adjust sources and reschedule content retrieval
  • Run more tests and make further adjustments

The outcome of this initial stage will usually be a specification for a full-blown project, with description of query terms and content sources, and a prediction of the type of analytical results the client should expect.

For an Agile approach to be feasible you need flexible search and mining tools, that can be easily pointed at relevant data and quickly come back with digestible results. The technical solution must, however, be possible to scale up to a fully operational state. And keep in mind that even in the full-scale project you might have to make rapid and frequent changes in the project scope.

As important as a great set of analytics tools is an analysts’ team that understands the Agile method and acts accordingly. The analysts start out with a set of hypotheses, but must be ready to give them up if they fail in favor of new and better assumptions. The central guideline will always have to be: how to meet the customer’s goals and requirements.

Agile Manifesto for Software Development at

TextOre, Inc.