As a follow up to my open letter on Adobe Certification (which again, never garnered an official response), I have an update to share with my readers. Before I do so though I have a lot of preparatory text I need to get out of the way. The following blog entry is based on a meeting I attended (non-NDA) for Adobe User Group Managers and Community Experts. It was specifically on Certification. This meeting left me frustrated and - frankly - a little bit more angry. I should perhaps have waited before blogging - but I wanted to share the notes while they are still fresh in mind. Just keep in mind that this may not be my most rational response. I'm emotional now (scratch that - I'm angry) and if you want to stop reading, I won't mind.

Note that when I use quotes it should not be considered exact quotes. Rather it is my notes and I wanted to visually differentiate between the slides in the meeting compared to my thoughts.

Software is Adobe's core business, not certifications. Anything they do outside of software though is meant to help the core business.

Very fair point and I agree completely. I wouldn't fault Adobe for a poor grammar choice in their docs for example.

Certifications serve three main purposes: insure adequate technical knowledge for certified instructors
ditto for adobe solution providers
provide credentials to help developers get more business and better jobs

Makes sense to me.

Certification is expensive. Costs include development, validation, localization, publishing, and tracking. The cost is around $50K per exam and every change is a $2.5K cost. (Unless they just turn off a bad question. Adobe does not make money on certifications.

Ok, no big surprise here, but frankly, I don't care. Really, I don't. The whole point of this issue is the quality of the certification and the fact that Adobe is selling something they don't stand behind. When the presenter was pushed on this, he said some internal forces were pushing back against just pulling the exams. Frankly those internal forces are wrong, in my opinion. Writing certifications are hard too. But darnit - if you can't stand behind it - don't sell it.

The creation process involves writing objectives (which is started fresh for each product version). Then they find individuals (like me) both internal and external to help write them. They are then reviewed (again, by folks like me, both internal and external). Based on feedback, Adobe sets a passing score for each exam.

For my thoughts on the write/review process, see the old blog entry.

After publication, how does Adobe verify the certification? They check pass rate. They aim for 60% of folks passing. They check the amount of time it takes to pass the test. They also do a 'point by serial correlation' check where they can look at a question that folks missed even though they passed. That's a good indicator of a bad test. Unfortunately it costs for Adobe to look at these.

Nothing much to say here - but again - I don't care about the costs (admittedly, easy for me to say).

Now we come across the slide that made me see red. Just ask the guys on my IM buddy list. They can't print here what I was saying:

Bad things can happen, and did. The first bullet point of blame was on the writers and reviewers. The last bullet point was on the vendor. The vendor was outsourced and looked good on paper.

So. Um. Can you see where I got angry? Hint - it's the second sentence. To say I was a bit upset would be a major understatement. Myself, and the the two reviewers, were very vocal. I also emailed Adobe. One of my contacts on the CF team, who I will not name, basically said it was the cert divisions problem, not the CF teams problem. So when the blame was laid right down on me personally, I kinda flipped.

Now the presenter, at the end, came back and acknowledged that that wasn't what he meant to do. He personally apologized to me. (And I'm not going to name him because he is an old timer, an Allaire guy, someone I know, and he was in a very bad position here and I don't want to shoot the messenger.) And let's be frank - you guys have seen my code. You know I make mistakes. Hell, I love to blog my mistakes because I figure people can learn from them. (Just ask me about how I messed up this CF server today. It was a real bonehead mistake.)

That being said - that was one of the worst slides in history. The number one bullet needed to be that the blame was squarely Adobe's fault. Stop passing the buck!

When pushed on the "if you won't stand behind it, don't sell it" point, again, the buck was passed, this time to internal forces. Apparently these internal forces won't let them stop selling the certs.

Adobe repaired as much as possible, but did not rewrite the certifications. One test was pulled. One test was not required for instructors anymore (Dreamweaver). One was totally rewrote (InDesign). On another test, it had a lot of issues but Adobe left it be since folks tended to pass because they figured out the right answers.

Not much to say here. The last sentence is a bit hard to believe though. Folks shouldn't have to figure out a certification.

The tests will not be rewritten for the current versions.

Last note - there are no plans (as of yet) for a Spry certification.

I don't know how much I can summarize this. I'm trying to clear my head a bit. I can say it is extremely frustrating. Extremely. I try not to say I have a "big" blog, because I figure as soon as I do my traffic will drop (and frankly, that guy with the big arms is kicking my butt ;) but I did expect to hear something on the blog from my last entry.

It sounds like Adobe really gets that they have a problem. But the #1 thing I got from this meeting was one big giant buck passing. Blame the writers/reviewers (and again, I'm not taking this as personally as I was initially). Blame the vendor. It's the cert team fault. Etc. None of this sounds good to me. I'm more than willing to say I know more about My Pretty Pony than large multinational corporations and economics, but at the end of the day, I don't believe Adobe is doing the right thing here. I've been called an Adobe Fan Boy on more than one occasion but I got to speak up here.