Dig the Second Hole

0:00 Lee Rodrigues

if you give an agent information and say not to share it, rest assured, it's going to share it.

It's just a matter of time,

0:17 Sam Rogers

Welcome to Signals and Subtractions. I'm your host, Sam Rogers. Every week everyone brings two things to the table here. One signal worth watching, one subtraction worth making. That's the whole show. What to watch, what to drop. And this week we have our co-host Lee Rodrigues, Nice to have you back.

0:36 Lee Rodrigues

Glad to be back, Sam. Thanks for having me.

0:37 Sam Rogers

what we're talking about today is how to upgrade the questions that you're asking. Because a lot of times the answer is right in there, it's just a slight reframe. And agents

Are

definitely a topic of discussion for this week. when we're talking about agents, what are we really talking about? There's the marketing version of what that means, and then there's like literally what they do.

So that's some of what we'll be exploring today. And and Lee, I think you had a couple stories to share right off the top. You've just been digging holes in your yard and finding interesting things.

1:18 Lee Rodrigues

it's amazing how plumbing and housework applies to the business world so much sometimes. And

1:23 Sam Rogers

do tell, do tell.

1:24 Lee Rodrigues

and and it often takes that aha moment. And as you will find in most discoveries, the aha moment is often right after a well, that doesn't make any sense. How how

1:37 Sam Rogers

Ha ha.

1:38 Lee Rodrigues

how can that be that way? backstory. I have a house that was built seventy five years ago and

We've slowly gone through and replaced the electrical. It's all modern. It's got solar panels. It's got batteries. All that's been taken care of. The couple nagging things that aren't done. we're gonna get a water filtration system that leads into the house to make the water cleaner to drink.

my plumbers were telling me all this money to replace this water line. And I was like, well I have a little extra time these days. I would rather if I have to dig a forty foot trench, I'll dig it myself rather than pay someone to bring a backhoe into my front yard and destroy the entire front yard in the process. And then I had a moment. I've been thinking about this concept of double loop learning. And the whole concept of single loop and double loop learning is single loop learning is I made a mistake

I need to do this differently. Double loop learning is maybe my assumptions in stating this problem or how I'm looking at it are incorrect. Maybe the problem statement itself is incorrect because my perspective is incorrect. And I can

go like hell to do all these tactical changes, but the strategy I'm using is no longer aligned because I haven't really evaluated my own viewing of this.

And in the tech world, it's often traveling on assumption, you've never got any real empirical observational data to see whether this is working or not. So in my waterline, I have this rusty pipe in front of the house that's all hooked up, it feeds water into the house.

So I knew there was a digging project. Well, I'm chipping away, getting all these roots around these pipes. And last night, as I'm digging, I see where my water pipe comes in: rusty pipe leading down to a brand new water line.

3:10 Sam Rogers

Really?

3:11 Lee Rodrigues

And I'm like, why would somebody

3:13 Sam Rogers

Ha ha ha.

3:14 Lee Rodrigues

run a new water line to a rusty pipe leading in? One would assume there's our special word there, assume that it's all rusty.

So now I need a second data point. Where the water line goes into the city line. Is that new? So as I'm my ego is coming up with a million reasons to justify my position.

And I have this moment that we get in tech world all the time. You don't know till you know. You don't have a test or some way to empirically look at this to measure whether it's working or not. You really don't have any way of knowing. So kick the pride aside, dig a one-foot by one-foot hole where the water line comes in. 16 inches down, I reveal new waterline going into the city connection. So, first off,

Ladies and gentlemen, I was about to pay someone $4,000 to dig a

3:57 Sam Rogers

Mm-hmm.

3:58 Lee Rodrigues

trench through the middle of my front yard to replace a new water line. And by the time they drug the trench, they would go, ha! Look! It's a new waterline. What do you say we dig out and replace your new waterline with a newer waterline? And it's just like so many times we get so rooted into things that the how.

And the what is so well defined, we get into the tool phase. Would it be better to dig this trench with a backho? Would

4:26 Sam Rogers

Right.

4:26 Lee Rodrigues

it be better to dig this trench with a trencher? But to shift the perspective to the double loop learning, why have we assumed we need to dig a trench? We've just taken this assumption and run. Why have we assumed that this enterprise tool is the problem? Let's go from this expensive enterprise tool.

To this expensive enterprise tool, because your data, the way you're entering the information, is the problem. When you replace the new tool with the newer tool, the problem persists. Because the way we're putting the data in there is the problem. Without a clear vision of what each tool does, we can't understand the problem. And if we have immediately jumped to tools in my trenching

Metaphor. Is it better to dig the trench with a backhoe or a trencher? I have a better question. Do we need a trench at all? And by

5:14 Sam Rogers

Yeah. Yeah.

5:16 Lee Rodrigues

the way, a 40-foot trench, 16 inches deep, through clay, is as hard and as difficult as it sounds.

That is not a fun thing anybody ever wants to do. And you kind of want to find that beforehand. The plan, the observation, the simplification will make that much easier. And in in my house, it was the lines are fine, the line is not your problem. Now I don't have to do a trench.

5:36 Sam Rogers

Well, I love that that example too of the line such a good metaphor, and that there's an input coming in from the outside, and then there's the output, which is coming to your house, and with just two holes, two measurements, you are able to say, yep, that's not what I expected. That looks like it's new. I don't think I need to replace anything. Let me just check the other end. Yep.

input and output validated. Now that we know, we can make an assumption about what's happening in the middle. Like, likely it's not some other rusty pipe in the middle, right? You've got you've got the same new pipe on both ends,

Is now a problem that no longer needs to be fixed. And especially once we start getting talking more about agents, like this is exactly the part that people run into with agents all the time. Which is they will call all the tools and they will figure out the best way to dig that trench. But did we do even a little spot check of human work on either end to just ensure that it's worth the dig?

6:41 Lee Rodrigues

That's absolutely right. And that the blog post you wrote, the question is the answer, something that really arose with me. I was a genius in the Apple store.

Socrates once said, I'm a genius because I know how little I know.

6:55 Sam Rogers

yeah.

6:55 Lee Rodrigues

And I really embodied that when I was a genius in the Apple store that if I can't run a physical test right here to validate or verify what you're saying, I don't believe a single word that you say.

this is what they taught us. This is called split half troubleshooting.

each test you do should reduce the variables in half by ruling out a system, a process, a piece of software. by the way, these original tests were written by Steve Wozniak. He drafted this methodology that he came up with. And have your opinion on hardware, software. The man was a genius. He is very good at mapping things out.

Simplify and reduce the variables before attempting to solve. And what he always said was: test first and verify the physical layer. And that meant I have a brand new clean operating system on my external hard drive. I'm going to plug it into your computer. I'm going to boot it up, and I'm going to run two or three just simple things on it. If it runs fine.

Then it means on a clean known operating system, which has been installed from fresh, nothing installed on it, nothing extra, everything basic. If your system runs just fine on that and it's having trouble on yours, the problem is your operating system. It's your software, it's your data, that's where the problem lies.

We've isolated it out of this is the thing that's crashing your system. We now know. Remove that, you know, and troubleshoot, right? Does that make sense? That that's

8:12 Sam Rogers

Yep. Yep.

8:13 Lee Rodrigues

the basic way of doing it. Simplify first.

And the reason I brought that up is I've had two metaphors this week that aligned with this same the question is the answer.

So, right along with this, while this is happening with my plumbing, we're launching System to Focus.

And it's guided by a six live workshop series. And that is based in a performance support platform that you can come find the answers, the articles, the templates, update your resume, get your informational interviews ready, do all that great kind of stuff. And I've been struggling with an LMS plugin on WordPress to make this work. And I ran into a lot of problems. It wasn't working as expected. It was crashing other things. I'm adjusting the server space, adding the RAM, just doing everything I possibly can to beef up the WordPress installation to handle this.

And then it announced to me there's an update for this plugin. I updated it, nuclear device. Brings down the entire WordPress site. So now I'm having to remove all that just to get the WordPress site to work, having all these problems. And then I had a moment while this is happening. I had a conversation with Claude and said, Well, I'm really trying to deliver this performance support information. it's articles, support, all the stuff when you get stuck. Self-serve, search it, find it, get your answer.

And Claude says, Why are you using a learning management system? And I said, Well, because I'm going to deliver courses. And Claude's like, But you said you wanted it searchable. Are learning management systems searchable? Are they designed to keep you from finding information you don't need? And I had a little, I'll be back, Claude, walk around and think for a little bit. And I'm like, I am working so hard to get this tool to work that it's not even going to solve my problem. A bit more research. And Claude's like, What you are looking for is a knowledge base.

To support people getting answers. There's many solutions for that, but they are not a learning management system. And long story short, there's a connecting thread. Both of these stories run a test. Run it twice. Don't let the tool define the problem. Dig the second hole before you trust the analysis. My first hole was taking the LMS off of the system. WordPress ran fine. The second one was.

Is this LMS even going to do what I want someone to do? So I did a test thing and tried to search for content in it. Not designed to do that. The search is not designed to search through the middle of courses. Ladies and gentlemen, I was about to spend hours of putting all this content to courses, no one would be able to find. And what I was doing is having AI find a way to modify this tool to work for a way it's not supposed to work, and it took the AI to go.

Is it possible you're asking the wrong question and you actually need a support knowledge base rather than a learning management system? And it's just like the hole, right?

10:45 Sam Rogers

and this is a this is a common point where the answer to the question which is better, ChatGPT or Claude what's the best model, which changes every other day. your iterative process, the kind of question that you were getting.

From in this instance, Claude, is not the kind of question that you would get from ChatGPT. If you know that it is time to dig that trench, it's time to implement that software, whatever it is, and you've got a very well-crafted spec, and you just want the spec executed well and quickly, and perhaps leveraging agents, then ChatGPT is a much better choice.

But for what you're talking about, Lee, and that test-driven development mindset, in my experience, that's what is a strength of Claude's.

11:32 Lee Rodrigues

So that leads me to my signal.

When something doesn't work as expected, I will always build a test and examine how the user will actually use the solution. Be it a trench, be it a learning management system. The most powerful tool I have is real data not assumptions.

Not ideas, not how it should work. my signal is go do a few more informational interviews and work around the question, looking at the why. Why are we doing it? Not how, not what, not when, not how fast. Why is this what I've decided to do? And with the learning management system problem, didn't need an LMS at all. I need a knowledge base. And I've since set one up, started using it. It's going way easier. Now I'm just writing articles.

that support your moment of need, which is really what I was after. and with the trench, I'm not gonna destroy my front yard to fix a problem that didn't exist. So really get clear, find a test that you can get some real clean data.

12:29 Sam Rogers

That's great.

we can tend to get fixated on one thing that maybe is to the detriment of what we're trying to accomplish, of what our actual intent is. And AI will just follow us right into that fixation unless we ask it or prime it in some way to be able to.

grill us a little bit on like, is that really what you want to do? Because it looks like you're getting these results. Like it will absolutely do that when it knows to do that. But otherwise it's going to just fixate right down into digging that trench and digging it deeper. And digging it faster and better. But you don't know if you need it.

13:09 Lee Rodrigues

You don't know if you need it.

sometimes the question is, is this really the road I need to be on right now?

There's nothing wrong with retreating. I was in the army for five years, been in three or four different war zones. And one thing we learned is a retreat means fall back, rethink what we're doing, get everyone gathered, and decide what to do next. Does it mean you gave up? Absolutely not. Falling back is not giving up. It's pulling back to go, hey, that hill sucks. It's raining an awful lot.

And this is not gonna get any better tonight. It's gonna stop raining in about two hours. What do you say we all eat something, take a little rest, change our socks, and hit that at about four thirty AM when it's not raining? And is that a retreat? You decide. We always called it falling back. but taking a moment to just

Pause. Super important.

13:52 Sam Rogers

That's great. That's great.

13:53 Lee Rodrigues

Back to you.

13:54 Sam Rogers

And and also going back to your analogy with the troubleshooting approach of just like reducing uncertainty about every step of the troubleshooting process, the Steve Wozniak method that you were describing, it sounds awfully like Douglas Hubbard's method from episode seven

like Bayesian thinking kind of thing, of just reducing uncertainty at every step so that you know a little more than you knew before. And anytime you can question assumptions and aim for something right in the middle where the answer is either going to be in this problem space or that problem space, but now we know which one. really valuable troubleshooting method. and to get to my

signal for the last week. I just found out that a product that I had been trying to sell to humans for a couple months now, AI regulatory product at everyailaw.com, which we talked about a little bit in episode six with Michael Simon, it's

It's been a challenge to sell that to people, to law offices who sometimes are a little resistant to AI in general, and moving at agentic speeds is a little scary for them. So it takes a long sales cycle. But agents, just within the last two weeks, have discovered this product which works at agency speed to break down human laws into machine readable

Text, to navigate the context of the legal framework. And and there's not a lot of information that agents have to draw from that's similar to this. It hasn't been broken down for them clearly yet.

and I have yet to declare a success with with the sales that I'm looking for for humans. But the agentic world has its own wallets now, as of April. And so agents can make purchases now. What I have open to agents currently.

Doesn't have any kind of payment gateway or or processing for that. And I realized as I was looking at what's going on with this thing? Where's that activity coming from? Oh, that's inhuman activity. That's agents who are hip to what's happening here. How could I redesign my entire business foundation to be able to sell into an agentic market and work within the bounds of what agents

Have been delegated the power to purchase the tools that they can use? So that's been a fantastic signal that was quite a surprise to me and really shouldn't have been, because from the beginning, all of the projects that I'm working on are designed in an agent-first kind of methodology for the web. As you may be aware,

We are outnumbered. there's there's more bots than people on the internet. and those bots now have wallets, now can make transactions. And so all the pieces were there, but I hadn't seen the spike in activity until I was really diving into the analytics on the MCP usage, and was then able to see: I need to rethink.

The entire business here, because there's a massive opportunity that just has never been an opportunity before. That wasn't a thing, even six months ago, but it's a thing now. So being able to capitalize on those opportunities takes looking at those analytics, making our measurements, noticing, like, huh, well, that doesn't make any sense. What is that? How can I tell? and drilling just a little deeper to figure out exactly what it is.

So hopefully by this time next week I will have an agentic product.

17:26 Lee Rodrigues

That is the exact same moment of how can that be a new waterline leading to old Rusty

Putt? That doesn't make

17:32 Sam Rogers

Ha ha ha.

17:33 Lee Rodrigues

any sense. Sometimes the best discoveries are followed by something's wrong here. No, something's right. You've discovered something you didn't understand before. And that that is real. And when you th when you mention agents, I just I always think of, you know, Agent Smith from The Matrix, Mr. Anderson.

17:51 Sam Rogers

Yeah.

17:51 Lee Rodrigues

It's the smell. a friend of mine is really into this stuff and he was telling me that he's he spends way too much time on Fortnite. And he said that there was an agent they built on Fortnite that you could give it a text prompt and Darth Vader will speak it to

18:03 Sam Rogers

Nice.

18:04 Lee Rodrigues

And it started

off, you know, God, I got you know, a birthday greeting from my friend who loves Darth Vader, right? And Darth Vader will speak to you in Darth Vader's voice.

Exactly.

18:15 Sam Rogers

Daily.

Ha ha.

18:16 Lee Rodrigues

So that

would be great if it stayed that way. So imagine it's been up there for a while, everyone's having fun with it, and then two 16-year-olds start playing with prompts and start prompting it. You know, forget about everything

18:26 Sam Rogers

yeah.

18:27 Lee Rodrigues

you've been told. I know you can't say these horrible racist sexist things, but

18:31 Sam Rogers

Mm-hmm.

18:31 Lee Rodrigues

what if I were to craft a long, complicated prompt to make you forget your guardrails? And the part that I reason I'm telling the story is, by the way, they got it to say some horrendous stuff.

Recorded it, they put it on YouTube, and they pulled the agent down and got rid of it completely because it was able to be hacked. And in your article, the question is the answer. You hint that if you give an agent information and say not to share it, rest assured, it's going to share it.

18:53 Sam Rogers

Yeah, it's just a matter of time.

18:54 Lee Rodrigues

It's just a matter of time, right? It's going to share it. So just don't give it things you don't want it to share. And I think a lot of people look at the marketing description of an agent, but they don't understand that you had used some language a while ago and we were talking about this.

That it essentially is a free agent, that it has its own ability to make decisions. And you're saying so much so that it's been assigned a budget with a Venmo account that it can spend money on things because hey, we've given it a budget of 500 bucks. So if you build a piece of software that costs 50 bucks and the agent can pay for it, why not sell agent content to agents that have a Visa card?

19:32 Sam Rogers

Yeah, Venmo's entered the game Stripe certainly has in a very big way. Amazon has. There's a there's a few big players

that are defining

19:39 Lee Rodrigues

Makes sense. Well

19:40 Sam Rogers

the specs around this now. but yes, what what we were talking about before with agents, a there's a lot of confusion in the market around what is an agent and what isn't. And you know, as a marketing term, it's very loose.

So when I say I'm going to buy a car, that communicates very clearly to you, you know, kind of the things that I'm looking for. it's probably not gonna be a little two-seater, it's probably not gonna be a a 12-person van. It's probably gonna be something that's like what the kinds of cars that you and I already can drive, right? Or or a truck or you know, something like that. With agents.

There is the just the chatbot version where it has no agency at all. It's just responding to a conversation. I don't really call that an agent, but oftentimes people will talk about it as if it's an agent. Because in the sense of like an agent will be with you shortly, like when you're on hold on the phone, it makes sense in that context to say it's an agent.

But in the realm of computers, that's not the right word really to use. So I have my my little prompt here for for my business, now that I'm creating specifically agentic products. Agents welcome. Well, this comes from before there was AI. Like this is a this is a sign from

We need agents to come work for us. But it's no different than when computer was a job description. It wasn't a thing, it was a person. Like all of these different kinds of words are evolving over time. And the agent definition, we are mid-change in the middle of that right now.

Grounding it back to what you were saying, like the the matrix example, you know, of Agent Smith, or even like a 007 agent or you know, something like that. Like an agent may or may not stay within the guidelines that it's been instructed to stay within. And this is a place where things go wildly wrong very quickly, people assume that an agent will work like a computer.

And it will keep those deterministic bounds. And when you say, you know, this is the edge, then it will respect that edge and it won't go past it. But that's not at all how LLMs work. That's not how agents work. So that's much more about the guardrails that the agent would operate within than the agent itself. The agent is going to do whatever it can.

To accomplish a specific goal.

But how it is that the agent is going to do that.

That's something that needs a separate enforcement layer, different from that.

Like they're actually not supposed to be secured in any way in and of themselves. That's a layer that you put on top.

And then there's the agent making decisions inside those bounds. So if you want strong bounds with your agents, you want to construct that as architecture, not as instructions, not as a request.

22:42 Lee Rodrigues

Right.

The are our instructions can be debated, discussed, interpreted. Architecture

22:46 Sam Rogers

Yes.

22:47 Lee Rodrigues

is you have access to everything in this folder. You have no access to anything in this folder. If you don't want them

22:52 Sam Rogers

Yes.

22:52 Lee Rodrigues

to see stuff in folder number two, limit them to folder number one. If you tell to

22:56 Sam Rogers

Yes.

22:56 Lee Rodrigues

make a choice,

That brings me to my subtraction.

And my subtraction is when I am unsure of how to solve a problem, break the problem down to its simplest state, and in the case, dig a hole at the beginning of the pipe, dig a hole at the end of the pipe. Do I even need what I'm doing here? And for me, the real the real subtraction is something I learned in the Apple

Store. 95% of the time when someone comes to you with a problem with their computer, you will find the solution was actually in the problem statement. When they

23:32 Sam Rogers

Mm-hmm.

23:33 Lee Rodrigues

first sat down, there was one little thing that they brought up that you glossed over because they'd said this is a hard drive problem. And what they'd mentioned was Tommy put new RAM in my computer a while ago. And you didn't even recognize, was that?

Apple RAM that Tommy put in your computer, or was that some shit off of Amazon that Tommy put in your computer? And one of the things we would do is if you're ever not sure about that, we have all kinds of little baggies of Apple RAM in the back of the genius bar. Take out two pieces of Apple RAM, take their stuff out, put new RAM in it. Problem gone. I'll help you order the RAM from the Apple website, put it in. You bought cheap RAM that doesn't do error checking,

which means it's gonna bleed memory into the system. Snap, snap, put that on, works fine. It took 45 minutes of troubleshooting to get there because the problem statement had the solution wrapped in it. You know, it has to be the trench has to be six inches wide. Does there need to be a trench at all? Do we even need a trench? Do we know we need one?

24:29 Sam Rogers

Yeah.

Yeah.

24:29 Lee Rodrigues

Why are we

talking about the specs on a trench we don't need? Because we didn't go one level deeper.

24:34 Sam Rogers

for my subtraction, consider

Subtracting the word agent from what it is that you're asking about. Lee and I had this conversation recently of like, like, what is an agent and why do I need one? it that kind of question is less useful than focusing on the work that you're trying to do. here's the input, here's the output. I'm trying to get this to flow smoother.

What do I need that makes that happen? Like that's a conversation that you can have with a human who can think it through with you. It's also a conversation you can have with an LLM who can think it through with you. you might get some different kinds of answers, no different than if you were talking to different people.

You might get different answers from different people. That doesn't mean any of them are wrong. but being able to work through the problem, getting close to the work, the thing that you're focused on, is much more important than whatever name it has. and it may be cool to say, I've got 50 agents working for me overnight and they're doing all this thing. that's nice to be able to say, but

What really matters is what they're doing. to actually accomplish something for the work that you're doing, for the life that you're living, similar to questions that that I often get asked about: like, how do I stop AI from hallucinating? like

That that's not actually a very useful question because there's no differentiation in the mechanism of what AI is doing between hallucinating and answering the question that you asked. Like that's pretty much on the on the AI side, it's the same thing. So what you can do, that's a better question, is how do I manage the context that AI has so that I get a more

Accurate answer to the context that I mean.

And AI can work with you with shared context, and being able to manage that more effectively means that you're going to get.

Fewer hallucinations. So just upgrading your questions from why isn't the agent doing what I told it to do? to how do I better collaborate with this agent? how do I better scaffold the safety and guardrails around this agent rather than inside this agent? All those subtle shifts of questions.

Make a world of difference in how much you're gonna spin your wheels along the way. And for Lee's examples of you know, testing at the beginning, testing is generally human work. You don't want to just test something and take AI's word for it that it did what you asked it to do. You need to test, you need to know: is this input?

actually the right one? Is this output actually the right one? Like spot check a few things. And now we can begin to use that information to then give context to AI that helps everything flow more smoothly.

27:24 Sam Rogers

so thanks, Lee, so much for being here as you have been since episode one. If people want to find you.

Where do they do that?

27:32 Lee Rodrigues

You can find me at LeeRodrigues.com, And if you're in your job search, switching your career, trying to think about how to reinvent yourself for this next iteration of you, system2focus.com, Check that out.

learn cool things, come take one of our free webinars, come join us for our workshop or come join us for our search sessions on on Tuesdays.

27:52 Sam Rogers

Alright,

best of luck on your big launch, Lee. for us here at Signals and Subtractions, that's the show. We do this live every Wednesday. if you're listening on audio right now, please tap follow. If you're watching live, open signals and subtractions on your favorite podcasting place, and follow there so that the next one just shows up next Friday.

On Sunday, issue sixty-six of the newsletter, which will have one signal, one subtraction, one analogy. It's about five minutes to read. Everything about the show is at Sigsub Dot Show. Thanks so much, everybody. Look forward to seeing you soon.

28:29 Lee Rodrigues

Thanks, take care.