Your Bots Are Only as Good as Your Docs

Job Thomas didn’t come to customer experience through support or operations. He came through the classroom. He lectured Greek and New Testament studies in South Africa and coordinated a secondary-teacher training program in Belgium before he ever touched a support queue. When he went looking for a change, he wrote a job description for the customer-education role he thought a company he admired was missing, and told them to hire him for it. They did. Nine months later that company was folded into Automattic, and over the next eleven years Thomas went from training manager to Happiness team lead to CX leader for WooCommerce, and now interim CXO across all of Automattic’s products.

That educator’s path is the tell. Thomas runs AI in support the way a good teacher runs a classroom: the knowledge base is the curriculum, so you get it right before anything else; the point is the learner actually being helped, not moved along; and the system only gets good through relentless review and feedback. It’s why several of his convictions run directly against where the industry is heading.

From the Classroom to Customer Happiness

The job Thomas invented was in customer education, and the company was WooThemes, later folded into Automattic. He arrived barely technical, knowing a little WordPress and WooCommerce and not much else. What changed him was a culture that expected everyone to get their hands dirty. He remembers the frustration of writing up a long bug report for something as small as a copy change inside a plugin, then learning he could just submit the pull request himself.

“It was scary as heck at first, but very formative. You’re shifting the relationship between support and product from a signaling function to an active contribution to building the product.” — Job Thomas, Interim Chief Customer Experience Officer at Automattic

That experience became a philosophy. If learning to make the change himself felt empowering, he reasoned, it would feel the same to customers, so his teams work to make it easier for people to solve things on their own. It’s the same instinct a teacher has: the win isn’t answering the question, it’s leaving the person able to answer it next time. When AI arrived, Thomas already knew what he wanted it to do.

What’s Actually Working

About three years ago, when everyone was certain AI would change support but nobody knew how, Thomas gave one team member free rein to figure it out and ship something fast. Her conclusion still governs how the organization builds: start with the documentation, because a bot with no brain gives wrong answers. She restructured a product’s docs before anyone built a bot on top of them, a mantra a colleague later turned into a conference talk.

“Your bots are only as good as your docs. Make sure your documentation is up to date and easy to digest for all the different LLMs and AI tools, because that’s the brain the agent is reading from.” — Job Thomas, Interim Chief Customer Experience Officer at Automattic

His teams have since built a tool that watches the product’s code repository. When a new WooCommerce version ships, it runs through the documentation, compares it against the actual codebase, and lists everything that needs updating with a suggested fix, so the docs stay ready for both humans and AI without waiting for a person to get to them.

The second conviction is sharper: step away from deflection as a metric. Thomas thinks the industry’s obsession with how many customers a bot deflects is backwards, and easy to game. Trap someone in a bot loop and your deflection rate looks great while your customer stews.

“We don’t want to deflect customers, we want to help them. Our success is in them being helped, not in us chatting to them less.” — Job Thomas, Interim Chief Customer Experience Officer at Automattic

The third is that shipping a bot is the start of the work, not the end. Across product lines, Thomas’s teams run close to 500 to 600 quality reviews a week, feeding corrections back into prompts and skills so the AI keeps improving. He’s blunt that a bot is human-like in one specific way people don’t expect: it won’t be perfect out of the gate, and treating it as a finished mechanical tool is the mistake.

The Mistakes Along the Way

Thomas is candid that his teams learned these lessons by getting them wrong first. Early on they chased deflection like everyone else, and they fell into the trap he now warns people about at conferences: setting up the prompts and the skills configuration, shipping the bot, and considering the job done.

“People say, we’ve set up our configuration for the bot, it’s shipped, and that’s it. And I’m like, no, that’s the start. Then you need to teach it how to improve, do regular reviews, and feed that quality back in so it keeps developing.” — Job Thomas, Interim Chief Customer Experience Officer at Automattic

There’s a subtler friction he’s still working through. For transparency, every AI reply is labeled as AI-generated, which is the right thing to do. But it also pushes some customers to demand a human even when the bot’s answer was correct, adding unnecessary friction for both the customer and the team. The honest fix isn’t hiding the machine, it’s rethinking where humans belong in the flow, which is exactly where his next bet lives.

What’s Next

Thomas wants to break the linearity of support. Today almost everything escalates in a straight line: basic bot, better bot, tier-one human, tier-two, tier-three. He thinks that’s a relic of an era when AI was only good at the simple stuff, an era he considers over. The bot can search 11,000 lines of code in minutes where a human would spend hours, so the question is no longer who’s more capable at each step, but who should hold the wheel at each moment.

“Almost everything in CX is extremely linear. We need to rethink where we still stand out as humans, and where we can hand it back to the bots when it’s clearly headed the right way.” — Job Thomas, Interim Chief Customer Experience Officer at Automattic

He’s exploring proactive, almost self-healing support, too. Many WooCommerce stores are self-hosted and outside Automattic’s network, but his teams have early tests that can read a site’s health from the outside, catch a critical error, and, for customers who’ve granted permission, help fix it or roll back a bad update before the store owner even notices. For a business losing money the moment its site goes down, that’s the difference between a vendor and a partner.

The throughline back to the classroom is the outcome he keeps pointing at. As AI has grown, his team hasn’t; the business grew while headcount stayed flat, with no layoffs, and the freed-up time has gone into proactive work like migrations and onboarding they never used to have room for. His advice to CS leaders is pure educator, the same instinct that had him ship a bot that was right 60 percent of the time because it beat the automated “we’ll reply within 24 hours” email it replaced.

“Keep asking one question: is what you’re creating today better than what you had yesterday? If it is, ship it. Don’t wait three months for perfect.” — Job Thomas, Interim Chief Customer Experience Officer at Automattic

The Sonora Project captures how CS leaders are navigating AI — the real experiments, practical wins, and hard-won lessons shaping the future of customer success.