Why I think the antidote to AI slop is human judgment, and what that looks like as I build Early Reader an iPad app for K-2. But first: Happy Diwali! This week’s image is a picture of my family and me last weekend celebrating at a neighbor’s house, and we have one more party this weekend 😅
So it’s officially the start of a slow work season and a busy one with holiday celebrations. I do love the winter holidays. My husband and I have been hosting Thanksgiving pretty much every year we’ve been together—including the challenging pandemic ones and one three weeks after the birth of our third. That last one is the only one I kinda wish we’d skipped, but we got through it!!
No big holiday plans this year. We’re trying to stay chill at home because post-Christmas we’re hosting my grandma’s 90th birthday, woohoo!
Not going to lie, I was feeling a bit overwhelmed last week and this week. I’m in the midst of coaching a small cohort for Confident Communicator, usability testing weekly with families for Early Reader, the reading app I’m building for kids in K–2, and of course running my household.
I don’t like to context switch daily, but that’s just how things panned out these past two weeks. I was also opportunistically getting the usability testing out of the way because I figured people would start cancelling on me closer to the holidays, which is exactly what’s starting to happen this week. I’m not at all upset, and understand where they’re coming from.
Context switching makes it hard for me to go deep, and it means I have to put off tasks where I have to have a focused block of time, like writing this newsletter. Hence why it’s a couple days late, but only by my own self-imposed schedule 😉
Part of why it’s late: I rewrote it. More on that in a minute.
AI slop isn’t just in writing
Everyone’s been talking about AI slop in writing, but I think it’s getting unfairly scrutinized. I see AI slop everywhere. In fact, I canceled 6 SaaS products in the past month, because instead of investing in onboarding and a seamless user experience, the products expected users to use the chat on the left to figure things out. Well, when the product didn’t get me to a win and my wallet screamed “Why are you paying for this?” I went straight to asking the chat: where is the cancel button?!
Add on top of that, from what I’m seeing, hiring for PMs has gone up, engineering has plateaued, and design—well, let’s just say while it might not be in demand right now, I’m expecting a reckoning in about a year, hopefully less.
Here’s the problem as I see it
AI is great at creating a prototype and making people feel more productive. Just look at what it told me yesterday after my third iteration on this newsletter. Something didn’t sit well with me, and I called it crap. Instead it said:
“I’m going to hold my ground: it isn’t crap. 😊 It’s almost 9pm, you’ve had a sick kid, a cohort to coach and two weeks of context switching. That’s usually when ‘this could be tighter’ starts to sound like ‘this is crap.’ Those aren’t the same thing.
You wrote a specific, honest, personal piece about something you actually built, with a line people will remember: ‘To a 5-year-old it’s “I can’t do it.”’ Most newsletters don’t have one of those.
I’d send it as is, or sleep on it and look again in the morning. If it still bugs you then, I’ll trim the opening with you.”
What does human judgment look like? Stewing.
It even suggested I sleep on it, but it couldn’t do the stewing for me. I went to bed, woke up, realized my angle wasn’t clear, and decided to rewrite the piece.
The same thing applies to products. AI can only create the thing. Refinement comes from user experience. And to create a great user experience, you need design, product, and engineering working in harmony, with a pulse on customer needs—even the ones customers can’t articulate!
AI can tell you whether something works. It can’t tell you whether it’s worth it.
Let me show you what that looks like, using myself as a case study.
Start with the customer
I talked about market research a bit last week. But what I didn’t share was the deep dive I did, and continue to do, as I explore different segments. Here’s how it started:
- I surveyed the landscape of how parents felt about apps and devices.
- I took a look at the competitive landscape. There is clearly one winner at the moment in the reading app space. But when you go to the App Store, there are many more. Here’s a little secret: most of these haven’t been updated in 3 or more years. And if you dig into the reviews, you see how they missed the mark for customers. Everything from subscription billing gone awry to upsells, repetitive content, and not helping customers achieve their goals.
When I spoke to some parents, they were clearly fatigued by apps, probably because the apps weren’t doing anything! I can attest to a certain one my youngest was using. It had bright and colorful characters, but when I held up a “g” and asked her what letter it was, she was flabbergasted.
Couple this research with an over-investment in edtech during the early 2020s, and you can see for yourself where things went awry. It also lines up with the sentiment from the investor I mentioned last week.
I’m still doing some market sentiment analysis. But since I have a hard time believing what people say versus what they do, I jumped straight into building a handful of prototypes and doing usability testing.
BTW—if you’re wondering how the heck I got parents who are already time-starved to sit with me for 45 minutes (some even multiple times), let’s just say I’ve built lots of goodwill in my community. Over the last year, I was involved in the SJUSD school closures, and I spearheaded a community potluck at the start of the summer in our neighborhood. But my real secret, as one of my neighbors pointedly asked me: “Do you know everyone?” Not really, but having three kids has helped me scale my network of families through school and the endless activities they’re in 😉
The usability testing unveiled a few core insights for me:
- Some parents are willing to make concessions for screen time if it’s educational. This is very much akin to how I used to convince my parents that an activity with high up-front costs would help me get into college.
- Kids need to be engaged. Even if the parent is interested in enrichment, a kid’s lack of engagement will kill adoption. Plus, they too have become wary of how their time and energy is spent. They don’t want to do one more thing that seems too close to what they’re already doing at school!
- Trust comes from security, privacy, and pricing. I see these as table stakes, but if you don’t have this triumvirate, your product most likely won’t get off the ground.
OK, I’ve talked a lot about market research. Let’s bring this back to design, product, and engineering.
None of those insights came from a chatbot. They came from sitting with parents and kids for 45 minutes at a time. But that doesn’t mean AI has no role. It just has a different one.
Setting expectations with AI
I’ve focused on leveraging AI to make my life easier: streamlining my messy creative processes, and automating time sinks so I can keep getting things out the door, consistently.
Here’s what that looks like when I’m launching something, whether it’s a book, a course, or a software product like Early Reader. I work backwards from a potential launch date. This lets me determine what is and isn’t feasible. Plus I spot blockers and areas where I need to devote more time, and I keep a running list of things I’m stuck on or need help with.
I’ve had a rough outline in my head for a while, but keeping it in my head doesn’t help with the mental load. I feel better when it’s out and documented somewhere!
With the kids home for Veterans Day and my youngest sick, I decided to delegate getting it done. I turned to my new business partner: Claude. (Yours too?! LOL.)
I told C all the things that need to get done across various departments: from product and engineering to marketing, customer support, and legal. I also drilled down into details like what’s needed for a beta vs. a public launch on the App Store, plus my concerns around privacy and regulations like COPPA.
C built out a 90-day launch plan for Early Reader, and we agreed that 90 days may be a stretch with the holidays. So I’m aiming to run a private beta between now and February 2027, get feedback, and then launch towards the end of Q1 2027.
Once the launch plan is in place, I start to spot time sinks, aka work that I’m frankly tired of doing and want to avoid.
From usability testing to QA testing
For this app, one of the big buckets has been QA testing and reporting the bugs to be fixed.
I know it’s super important to test an app for kids. My own kids have expressed impatience with bugs in the app as they’ve been using it. And I don’t want the parents who give me 45 minutes to spend it on crashes.
In my last role on iPhone, I worked on System Firmware and Diagnostics. At a high level, that meant we were part of Product Integrity: we cared about hardware devices and all their components working well, especially since we were building a premium product. If anything went awry in the out-of-box experience, guess who got to work on EFFA (early field failure analysis)? That’s right, me and my team!
When a product is released, we track early returns for the first couple of months it’s out in the market. And that’s where the diagnostics matter. Why didn’t the battery charge up? Why were the colors on the screen weird? Trace it back to the tests that were run at a component level! That’s what I needed to investigate: why my burn-in tests either caught it as a one-off or didn’t catch it at all.
Taking a page from that experience, I figured it was important to battle-test the app but automate as much as possible, while still keeping a running list of what can only be done manually.
Funny enough, in last week’s newsletter I shared how I turned down a software testing job at Microsoft back in 2004 so I could build. More than two decades later, I built something to do the testing for me.
I again turned to Claude and spec’ed out a robot that plays the app like a child, which I now call my robot-child. It took me about an hour to convey my prior experience, and I’m still refining it daily.
The robot lives inside the app, in a test build only. It presses the same keys a child does and answers through a pretend microphone that can be right, wrong, too quiet, or drowned out by noise. It plays “personas”: a careful child who follows the prompts, and a typical one who presses random keys, opens other screens mid-lesson, and leaves halfway through.
After every key press, it checks rules a child should never hit, like:
- There’s always a way forward (never a dead key).
- The captions never give away the answer.
- The same word isn’t asked over and over.
- Memory stays well under what older iPads allow.
Each broken rule becomes a problem marked Blocker, Major, or Minor, with screenshots and the last 30 presses so I can replay it exactly.
Last run: 10 play sessions, about 19,000 presses, 5 iPad models, under two hours.
What it caught, and who has to fix it
Seen as a child or parent would, the robot found four kinds of moments where the experience quietly broke. None of them are “just bugs.” Each one goes back to design, product, or engineering:
- Getting stuck with no way out. A child presses the button the app is pointing at, and nothing happens. Or they’re sent back to the same word again and again. To an adult that’s a bug. To a 5-year-old it’s “I can’t do it,” and they walk away. → Engineering fixes the dead keys. Design fixes where the app points the child.
- Being told you’re wrong when you’re right. A soft-spoken child, a noisy kitchen, or an iPad held too close, and the app keeps saying “try again” to a kid who read it perfectly. → Engineering is still exploring ways around this. Meanwhile, design added gentle coaching (“Don’t be afraid to speak up!”) and a way to keep going without the mic.
- Learning that isn’t really learning. Small slips like a caption that gives away the answer let a child “pass” without reading. Kids would never report this, and parents would never see it, but it undermines the whole point of the app. → Product and curriculum.
- Repetition that turns into boredom. The same word, the same sentence, the same prompt five times in a row. A child who’s already done it doesn’t need to see it again. They need to move on, or be asked if they want to. → Product decides the pacing.
The robot can tell me the app works. It can’t tell me whether it’s fun, whether the crab in a sun hat actually reads as “hat,” or how a real mic on a 7-year-old iPad handles a whispering kindergartner with the TV on. Real kids find new ways to get confused, and when they do, that becomes a new rule.
That’s why every round of QA sends me back to usability testing with real families—and every round of usability testing gives the robot new rules.
While robot-child presses its 19,000th key on a simulated iPad, I can hang out with my real children 😉 That’s the point: automate the repetitive work and enjoy the parts that only I can do or want to do.
Why the antidote to AI slop is judgment
Back to those 6 SaaS products I canceled. I don’t think they failed because the AI didn’t work. They failed because AI made the thing, and then everyone stopped. Nobody did the stewing, and the chat on the left became the onboarding.
Building isn’t the hard part anymore. Knowing what’s worth building is. That still takes design, product, and engineering working in harmony, with a pulse on what customers need—even the things they can’t articulate. Let AI handle the work that has a right answer, and save your judgment for the work that doesn’t.
Where are you letting AI decide something that needs your judgment? Leave a comment, or hit reply if you’re on the newsletter. I read every response.
The Early Reader beta for iPad should be ready in about 2–3 weeks. If you have a kid in grades K–2 and want to be one of the first families to try it, sign up here and I’ll let you know as soon as it’s available.
