Now that I’ve reviewed Forrester’s Wave report on ECM, let’s tune into some specific criticism. Specifically Jeff Potts spoke up and took them to task on how they weight what they measure. Being a fan of Alfresco, his post is a little on the defensive side, but that doesn’t eliminate his points.
Jeff on Forrester
First, I want to look at some specific criticisms leveled by Jeff. Some are a little off target. When reading this, remember I am not a fan of the report. I actually enjoy Jeff’s blog. I just want to make sure that facts/perceptions are straight.
First, he starts to criticize the weight of several items in the report. He complains that Platform Readiness is only 2.5% of the total score. Couple things to note here. First, it is actually 5% of the Current Offering category (the vertical axis). If one is going to combine the multiple dimensions, then all three should have been taken into account (1.7%). It doesn’t really matter though. Alfresco got a 5. As did EMC, IBM, Oracle, and everyone else but SAP and Vignette. So no matter what you make the weight, it washes out. As Forrester defines that category, Alfresco at least deserves the 5 unlike others (See the previous Forrester post for details).
To be fair, he continues the divide by 2 error everywhere, so it isn’t an effort to make the numbers work for his argument. It is just a math issue.
Jeff also complains about Enterprise Search being measured as part of Extended Capabilities. He questions the whole inclusion in the evaluation. He likens the breadth of the ECM definition to that of an Imperial Star Destroyer. While I like his analogy, I think this is an issue of the transition of the ECM definition. Most of the world uses the AIIM definition. I think Jeff has a similar vision for the ECM 2.0 as I do, but he is expecting Forrester to follow that definition when nobody else really does. Forrester defines Extended Capabilities as features that are related, but not necessarily core, to ECM. I favor the inclusion based upon my experience. The weighting may be a little heavy, but more on that in later.
I think some of Jeff’s problems stem from dealing with customers that just need a Content Management system and not an Enterprise Content Management system. To be fair, many organizations don’t need the full power of a traditionally defined ECM system. However, I have a few clients that do and actually value some of the things that Jeff discounts. Remember, we are evaluating ECM Platforms and their ability to handle everything thrown at them, not the needs of any specific client.
Finally, Jeff doesn’t agree with the scores for several capabilities, such as BPM and Collaboration. I can’t speak to that. I can only suggest looking at the products that scored higher and determine if there is a rational reason. There may not be one. This whole report seems to be based on surveys, so I expect mistakes. I like to pretend that there is no intentional score adjustments.
Riding a Wave of a Different Kind
Okay, now that I’ve upset Jeff and the Alfresco world. Let me say HOLD ON! While I think that most of the items should be considered, I think that the weights are wrong. Extended Capabilities should be, for example, half of what Forrester rates it. That is the beauty of the Forrester report, you can take the existing categories and adjust the weights to measure what you actually need. In fact, Forrester tells you to do just that:
This evaluation of the ECM suites market is intended to be a starting point only. Readers are encouraged to view detailed product evaluations and adapt the criteria weightings to fit their individual needs through the Forrester Wave Excel-based vendor comparison tool.
Not having their Excel spreadsheet, I made my own and tweaked the scores. I could only adjust the high level scores as that is what I had for everyone. I also only adjusted the Current Offering dimension, so everything adds up to 100. I’m not mucking with the Strategy scores without the details.
- Core ECM Capability Breadth: I knocked this from 40% to 25%. I want all of the functionality there, but it isn’t my primary factor. For a specific client, I would try and go deeper and trim the scores on individual components.
- Platform Readiness: As I have stated, the scores here are worthless. I took this from 5% to 0%. Everyone is hurt pretty much the same except for Vignette and SAP. However, if I know it is wrong for IBM, OpenText, and EMC, I can’t use the score for anything. It is a shame as I believe Alfresco may actually deserve their 5. If this was accurate, I would have given it a 10%.
- Additional ECM Capabilities: I took this from 20% to 15%. Not a large shift. Once again, I would adjust the sub-categories for specific clients. Having these capabilities as part of the ECM platform can help quite a bit.
- Extended Capabilities: Same logic as stated for Additional ECM Capabilities. I took it from 20% to 10%. For some clients, I may even remove this. However, when evaluating a platform, we need to leave this in.
- Core Capability Architecture: This is big. This is the core of ECM 2.0. I moved this up from 10% to 50%.
- Miscellaneous: As I stated in my last post, this is a joke. Dropped from 5% to 0%. I wish these scores were real. I would add 5% back to it.
If you look, I would have given 15% back if the Platform Readiness and Miscellaneous scores were actually accurate and not throw-ins. I would have taken 5% each from Additional ECM Capabilities, Extended Capabilities, and Core Capability Architecture.
So what does the Pie Wave give us? Here are the scores and some quick notes:
| Vendor | Pie Wave | Forrester | Change | Rank |
| IBM | 4.47 | 4.21 | 0.26 | – |
| OpenText | 4.05 | 3.99 | 0.06 | – |
| EMC | 3.87 | 3.76 | 0.11 | – |
| Oracle | 3.82 | 3.65 | 0.17 | – |
| Vignette | 3.52 | 2.88 | 0.64 | +1 |
| Interwoven | 3.42 | 3.15 | 0.27 | -1 |
| Alfresco | 3.27 | 2.46 | 0.85 | +2 |
| Xerox | 2.98 | 2.42 | 0.47 | -1 |
| Microsoft | 2.93 | 2.42 | 0.51 | +1 |
| Highland | 2.87 | 2.51 | 0.41 | -2 |
| SAP | 2.50 | 1.74 | 0.76 | – |
- The four leaders changed the least. This is from having balanced scores across many of the categories.
- Alfresco scored the biggest improvement both in ranking and in score.
- SAP and Vignette improved so much because they were the only ones who didn’t get a 5 in one (both for Vignette) of the tossed categories.
The lesson, adjust things to match your needs and then rank. I just went on the whole ECM platform ranking. I have never had a customer that needed it all. However, I’ve needed almost every component at one time or the other.
A Parting Thought
Alfresco will always score poorly on the ECM 1.0 reports. Their strategy seems to be very ECM 2.0 focused, which I applaud. They are trying to be a platform that support Content Management for the Enterprise. They aren’t there yet, but they are relatively young. However, they aren’t adding all the functionality being scored by Forrester or Gartner. They are working on allowing the experts in Search and IRM to be added on top of their platform. Alfresco is focusing on what they know and allowing others to just plug-in.
We’ll see an accurate measure of ECM 2.0 in about five years when we are working on implementing ECM 3.0. Maybe by then I can be an analyst and evaluate products that I’m never used.
Word of Pie 