We frequently hear the terms “Proof of Concept” (POC),
“Prototype”, and “Pilot” used interchangeably. This is unfortunate because they
mean different things and serve very different functions. In fact, I think that
if you want to successfully implement technology, you should be doing all
three. The good news is that, at least to some level, you probably are. But the
extent to which you are benefiting from POCs, prototypes, and pilots depends on
the level of consciousness and deliberateness you practice. By “deliberateness,”
I mean having an understanding of purpose. I use “consciousness” to say that
you pay attention to the process and are aware of the results.
Proof of Concept
A Proof of Concept (POC) is a small exercise to test a
discrete design idea or assumption. Software developers tend to do POCs
instinctively when they experiment with technology. An example of a POC is
testing whether one technology talks to another. To be deliberate, a POC should
clearly state what it is to be proven and to what degree. For example, it may
not be enough that a message can be transferred from system A to system B if
the complete solution calls a specific level of performance and reliability.
You may need to test under certain conditions. POC’s often take the outward
form of a simple “Hello World” but that may not be enough if the system needs
to support an extended character set.
The results of a POC need to be measurable so that they can
be fed into the decision making process. Your decision should be based on
thresholds like a performance level that is acceptable. That’s the
consciousness part. It may not be enough to for someone to just say “yeah, we
tested that and it works.” You want the results of the test to see how or how
well it works. A POC may involve secondary research if you trust the source.
Maybe another group tested the same concept. If their process was as rigorous
as yours, you may be able to use their results. There are also user interface
POCs that explore the usability of a specific UI control or visual metaphor.
These need to be designed and tested with the same degree of thoroughness.
Prototype
A Prototype is a more fleshed out system that tries to
simulate the full system or at least a material part of it. Prototypes are used
to test the viability or usefulness of a system or subsystem. You plunk an
intended user of the projected system in front of the prototype and he/she/it
(the user might be another system) should be able to visualize the experience
of using the complete system. The completeness of the prototype depends on what
you are testing. For example, you might want to build a three dimensional
physical model to test an innovative new shape for a telephone handset. If
there is question as to whether the person can hear a speaker which has been
designed in an unconventional place, the prototype may need a speaker. Or
possibly, you could do a POC of just holding a speaker in that position and
testing audibility in different conditions. Pretty much all manufactured goods
go through a prototyping process because that is usually the best way to figure
out the feasibility and cost of manufacturing the product. For software,
prototypes can have varying degrees of fidelity from visual mockups to working
software.
You need to be deliberate in defining what you hope to gain
through building the prototype and the minimal level of completeness that it
would take to get that information. There is some disagreement as to whether a
prototype should be constructed to be thrown away or whether it should be the
initial building blocks of the complete system. I think it depends on a number
of things. You should invest as little as possible in the prototype otherwise
the sunk costs might make you stick with a bad idea longer than you should. If
that means building it out of a technology which is agile to use but unsuitable
for full scale deployment, then use the easy technology with the expectation of
throwing it away. However, if you are dead sure on the core idea and you just
want to prototype different variations, it may be more practical to build the
core technology framework and then build prototypes off of that.
If you go through the hassle and expense of building a
prototype, use it. Get it in front of your intended audience and listen to what
they say. Don’t just listen. Hear them. If you are not doing to do that, don’t
call it a prototype. Call it an iteration.
Pilot
A Pilot uses the full production system and tests it against
a subset of the general intended audience. The reason for doing a pilot is to
get a better understanding of how the product will be used in the field and to
refine the product. Maybe there are some open ended questions about scalability
that only a live audience using the product can answer. If you are doing a
Pilot (or Beta) for a reason other than to build hype, make sure that you
target the right group of people and you are prepared to collect the right
metrics. You may consider prepping your pilot group to evaluate the full
breadth of the application and give them channels to provide feedback. If there
are particular areas of the application that you really want to test, call
attention to them. Determine what kind of quantitative metrics you will be
collecting and have the tools and time to analyze them. If you are going to do
a Pilot you should be prepared to act on the information that you collect and
fold refinements into a general release. You should probably have a process to
prioritize enhancements ideas and a threshold for determine what enhancements
to implement.
POCs, Prototypes, and Pilots are all important tactics in
implementing technology. However, just using the words is not enough. You need
to engage in these activities deliberately and methodically for them to be of
any value. And you will never realize that value unless you consciously record
the results and turn this data into usable, actionable information.
No comments:
Post a Comment