ChatGPT: Understanding The Limits and Possibilities

ChatGPT: Understanding The Limits and Possibilities

Tim Fitzpatrick
Over the last three months or so, ChatGPT, and similar AI technology has been a super hot topic and continues to be. ChatGPT represents a huge leap forward in artificial intelligence. I have a special guest with me today, and we're going to delve into the limits and the possibilities of this incredible technology and what it means for business owners. Hi, I am Tim Fitzpatrick with Rialto Marketing, where we believe you must remove your revenue roadblocks to accelerate growth, and marketing shouldn't be difficult. Thank you so much for taking the time to tune in. I am super excited to have with me Sam Tomlinson from Warschawski. Sam, welcome and thanks for being here, man.

Sam Tomlinson
Likewise, thanks for having me, Tim. Excited to be here.

Tim Fitzpatrick
Yeah, I'm excited. I'm looking forward to digging into this. I certainly am not a ChatGPT expert, but I know you have spent a lot of time thinking about this. I know when we initially connected for the pre interview, I was super impressed, man, with some of the things that you had to share. I know you're going to drop some serious bombs on us today. Before we do that, I want to ask you some rapid fire questions. Help us get to know you, ready to rock?

Sam Tomlinson
Let's do it.

Tim Fitzpatrick
Okay. When you're not working, how do you like to spend your time?

Sam Tomlinson
I would say two years ago, I would answer that by playing golf. Now I have got a two year old daughter, so I love spending time with my kid. It's honestly a lot of fun.

Tim Fitzpatrick
Isn't it funny how our hobbies change as we have kids?

Sam Tomlinson
Oh, my God. It's so huge.

Tim Fitzpatrick
I used to play a ton of golf, and then it's like, Oh, my God. I don't have five or six hours to just go off and do this.

Sam Tomlinson
I feel like I need to hire my nanny on a Saturday and be like, Can you go play golf, please? That'd be great.

Tim Fitzpatrick
What's your hidden talent?

Sam Tomlinson
I love to cook. I love to cook. I hate to bake, but I love to cook.

Tim Fitzpatrick
Love to cook. Anything in particular you like to cook?

Sam Tomlinson
I run the gamut. I think if I... In a different life, I might have started a restaurant, been a chef, but I feel like if I ever would have done that, I would have hated to cook. You know what I mean? It's fun because it's a hobby. But I love classic French cuisine and Italian cuisine, so those two for sure. After that, whatever.

Tim Fitzpatrick
What's the best piece of advice you've ever been given?

Sam Tomlinson
Probably that you pay to learn. My grandfather always told me that. He's like, Everything seems free, but you always pay to learn. So you're either going to pay in time, money, or suffering. So just understand how you're paying.

Tim Fitzpatrick
That is actually a very good piece of advice because it's so true. I think so many of us default to like, I'm not going to pay for that. But damn, you spend a ton of time so you pay one way or the other.

Sam Tomlinson
You always pay. So it's a really good mindset shift for me to be like, okay, actually free doesn't mean free. Free just means no dollars, but there's another cost associated. And if you don't understand that, I think it's really hard to be successful over the long term.

Tim Fitzpatrick
And oftentimes too, if we pay to learn, actually pay by paying money, it helps us get where we want to go much faster.

Sam Tomlinson
Yeah, it's often the cheapest way to make cash.

Tim Fitzpatrick
What's one thing about you that surprises people?

Sam Tomlinson
That one's tough. I think a lot of it is that I probably read more than anybody else, which surprises people a lot. I probably try to spend two to three hours a day reading stuff.

Tim Fitzpatrick
Do you read fast?

Sam Tomlinson
Yeah. But it's a combination of... And I read the same thing probably five or 10 times. I've read The Intell Investor, I think, cover to cover seven times.

Tim Fitzpatrick
You always pick up something different, right?

Sam Tomlinson
You do. I think I pick up something different, and sometimes I look at something different, which surprises people. And a lot of it is just... Okay, I'll give you a very simple example. I was at a Bruce Springsteen concert this weekend. I love Bruce. I love Bruce Springsteen. And there's a song he plays called The River. And when I was a kid, it was a warning song because it was about a young couple that they got pregnant at 18, and he had to abandon all of his dreams of getting out of his town and whatever. It's actually about Bruce Springsteen's sister. And when I was a kid, that song was a warning. If you make bad choices, you're going to get stuck in this town forever. I looked back on it as an adult and I'm like, It's a love song because this guy gave up a son of dreams, and now he has a daughter that he loves and he's married to his wife and he loves her. This is a great song. But teenage me was like, This is horrible. Adult me is like, It's beautiful. I think I get the same thing from books. You look at it from your current perspective and the lessons that you may have never seen this. When I read it five years ago, I read it now. Huh, that's clever.

Tim Fitzpatrick
That's so interesting because it takes... You're in a different place every time you read it or every time you listen to that song. And because of being in that different place, maybe having a different perspective, having more experience, you pull something different out of it. I love that. What does success mean to you?

Sam Tomlinson
I think that's another one where the definition changes. Obviously, we all have financial goals and I want to get to a place where I can do all the things that I want to do in terms of travel, spend time, not work until I'm like 90. That seems good. But I think more and more to me, it's being able to do most of that, 90 % of that, while being 100 % present for my kid. So that's always a tension for me, but it's evolving, we'll say it that way.

Tim Fitzpatrick
Where's your happy place?

Sam Tomlinson
Generally speaking, on a golf course. There is nothing that beats four hours with no cell phone ringing and just you and your thoughts.

Tim Fitzpatrick
Yeah. You got to concentrate just enough where if you're thinking about something else when you're trying to hit a golf ball, you're in trouble.

Sam Tomlinson
It doesn't go well. But I think there's also something to be said about it's just you versus a golf course. There's nobody else. It doesn't matter. But it's just you versus something else. And it's an opportunity to just be there and have fun. And I like that that it's just peaceful.

Tim Fitzpatrick
What qualities do you value in the people you spend time with?

Sam Tomlinson
I would say for sure, drive success. I want people that are driven. I think you're probably the sum of the five and seven people you spend the most time with. You're the product of that. So I want people that are driven, that are ambitious. I don't want people that settle. I value people that are loyal, at least they'll do what they say they're going to do. I guess it's probably a better way to put that. Blind loyalty is terrible, but I think a commitment to honoring your word is good. Intelligence, for sure. And creativity. Anybody can find a reason not to do something, but I like people who can find the one way to do something. I'm not interested in the 99 ways it can happen. I'm very interested in the one way that it can. So that's it.

Tim Fitzpatrick
I love it.

Sam Tomlinson
If we're smart, driven people who are committed to doing what they say they're going to do and we'll have a great time.

Tim Fitzpatrick
So Sam, tell us more about just what you're doing at Warschawski. I know you've got a personal brand as well.

Sam Tomlinson
So Warschawski, we're global boutique marketing agency. We tend to work with midmarket to enterprise brands. We do work with some startups via our venture fund that we launched in 2020. But primarily it's performance marketing, growth marketing for brands that we love, for brands that are trying to do something interesting, which is fantastic. I'm a person that loves to solve problems, so we get to work with some really cool companies who are doing really amazing things. We work with dual packaged foods, we work with you know, CFG Bank, who just completed the naming rights for an arena here in Baltimore. We've got everything from high growth B2B to niche B2C and D2C. So it's really interesting for me to be able to see a huge array of to be run, again, like plaintiff's lawyers to bankers to private credit institutions to see so much and help these companies grow. So most of my focus is on our digital, our analytics, and our marketing science teams, but we do a lot of things in house and we built a remarkable team here. So really, I'm thrilled about the work we're doing. And then on the personal side, I just got done with the Spring Conference circuit.

Tim Fitzpatrick
So that was busy.

Sam Tomlinson
It was a lot of cities in relatively few days, but I did a keynote panel at SMX in Munich on ChatGPT of all things. I was in Vegas for Shop Talk. I was at Social Media Marketing Summit, a few others as well, but been busy. Now I'm looking forward to a little bit of a slower season.

Tim Fitzpatrick
Yeah, a little bit more time at home.

Sam Tomlinson
A little bit more time at home, a little bit more time to work on some projects. Just a little stuff here and there, but it's fun.

ChatGPT is an (Over)confident High School Sophomore?

Tim Fitzpatrick
So let's talk about ChatGPT. In a recent newsletter, you had said ChatGPT was an overconfident high school sophomore. I love this because I think most of us can start thinking back to when we were sophomores. Can you explain what you mean by this?

Sam Tomlinson
Yeah. I think if you take a step back, the first thing you have to understand is what LLMs are and what they are. An LLM is, I think, the easiest way, which is not a perfect data science explanation, but I think it gets you close enough, is that if we think about machine learning is like having a million interns that can look at something and find patterns at scale. Machine learning is basically pattern identification at scale. Sometimes those patterns are interesting and sometimes those patterns aren't interesting, but that's what it fundamentally is. LLM is an inverse of that where we resolve a linguistic question into a statistics problem and then we pattern match it. So if I ask ChatGPT a question, it looks for things It looks for an answer that is the most probable to something that looks like my question. It tries to find the answer to my question based on that matching pattern. So it's less about, I would say, answering the question. It's more like creating something that looks like an answer to the thing that looks like your question.

Tim Fitzpatrick
Okay. And LLM stands for Large.

Sam Tomlinson
Language Model.

Tim Fitzpatrick
Model. Okay.

Sam Tomlinson
Correct. Yeah. So I think when you think about it in that context, ChatGPT doesn't know anything. It's just taking a linguistic problem and turning it into a statistics problem and then answering the statistics problem, which perfectly well and good. And candidly, it does a really nice job of providing a decently fine response to a lot of questions, just like a high school sophomore would. But then if you're truly an expert in that area, you'll look at it and say, This actually isn't right. It sounds good. It looks good. But underlying that, there are problems because, again, you've taken a linguistic question, potentially a factual question, and you've turned it into a statistics problem. I think that's the inherent limitation. And that's why you end up with these hallucinations, for instance, where I ask ChatGPT to summarize a conversation, and it injects somebody into that conversation that wasn't there. Or just yesterday, I was asking it about a medical study for a heart condition, and it cited an article talking about... We have a potential investment opportunity in a cardiac implant thing, about the efficacy of these things. When I went to go find that article in the New England Journal of Medicine, it didn't exist. ChatGPT made up an article that said it was in the New England Journal. And if I hadn't taken that thing, I was like, Let me look at this. Let me actually put this into Google. Let me see what's actually there. I never would have known. The title sounded good. I was like, Oh, okay. Efficacy of cardiac implants in whatever patients with COPD, right? Or whatever the one that was. That sounds like an article in the New England Journal of Medicine. I don't have a medical degree, but when I looked it up, I was like, Oh, that doesn't exist. That's a little problematic.

Tim Fitzpatrick
You've got to check sources if it's listing sources.

Sam Tomlinson
100 %. Or you have to ask for where it's coming from or any of that information. I think when you do that, you start to see, Maybe this isn't this great. And I know there was a Washington Post story, I believe it was, last week about a law professor who ChatGPT accused of sexual harassment. The problem was that that law professor had never taught at the college where he allegedly did it and was never actually accused of anything.

Tim Fitzpatrick
So another example, ChatGPT made it up.

Sam Tomlinson
Made it up, along with a Washington Post story allegedly corroborating it, that never ran. And then the Washington Post subsequently investigated the Washington Post's investigation of the thing that never happened.

Tim Fitzpatrick
And it's doing that because of how it's structured?

Sam Tomlinson
Exactly. It's a probabilistic model. It looks at the word that is most probable to follow the sequence that it has. And sometimes that's another name. Then you got a hallucination, and then it runs with it. And sometimes that's a fake article, like we saw with the American New England Journal of Medicine article or like this Washington Post. And I'll send you the links, you can put it in the show notes.

Tim Fitzpatrick
Yeah, cool.

Sam Tomlinson
It's fascinating that you'll see some of these models just invent stuff that's just not true, which is really interesting.

Tim Fitzpatrick
I'm still thinking about this confident high school sophomore, right? So with these examples, it's like, Hey, what do you think? And the high school sophomore is just spitting it out with complete confidence, and it's total crap. Correct. Sometimes it's total crap. It's not always. But we can't take what we're getting from ChatGPT at face value.

Sam Tomlinson
No. And I think the best way to describe this is basically ChatGPT is giving you an answer that looks like what the answer to a question that yours should be like. It's trying to pattern match. Just like that machine learning technology identifies patterns in a data set. And maybe that pattern is really interesting. And maybe the pattern is just that women tend to have babies, like, duh. That's the pattern. And if you look at that pattern, a machine learning model will find that because it is a pattern and it will identify it. But it's not an interesting pattern. It's not a useful pattern. It's just a pattern. And here, we're doing the same thing. We're finding this late in pattern in terms of what your question looks like, and we're formulating an answer based on what we think the answer to that question looks like. So it's fun. I think it's a really interesting it's an interesting piece of technology.

The Significant Limits to ChatGPT that May Not Be Obvious To Many

Tim Fitzpatrick
What do you see as some of the significant limits to ChatGPT that may not be obvious to a lot of us that we're not thinking about?

Sam Tomlinson
So I think the big one is obviously that these are not... An LLM is not an information retrieval system. It's not. Search is an information retrieval system, and it's a really good one. An LLM is not, by definition, going to do that because the LLM generally works on some... It works on a probabilistic model. So we all know that there's a lot of garbage on the internet. All that garbage gets fed into the model and the outputs of the model are largely contingent on the data that was used to train it and the data to which it has access. Now, in most cases, ChatGPT, generally speaking, if you put in an answer like, Hey, go to the website this, it was like, I don't actually have access to the internet. Because it's not trained on that data. So when you think about what's it good at, it's not going to be information retrieval because number one, it doesn't always have an up to date corpus of the actual internet. And number two, it has no way to distinguish between what was written on Reddit by a random, say, neofascist user and what was written in the New England Journal of Medicine by a world renowned cardiologist. It's like two pieces of information. You know what I mean? The ability of the model to understand and properly weight different sources of information is still not great. And then I think when you start to get into it even more, you're like, Okay, what else is it not particularly great at? One is any minority opinions, or a marginalized group, or anything like that where the machine just doesn't understand that it's talking about certain things. I think it's more of a societal mirror. ChatGPT takes in all the information that we have on the internet, and some of it's really ugly, and it gives it all back to us just reflected. So I think anytime you have a marginalized group or you talk about racism or sexism or ableism, ChatGPT is just as terrible as some people are. And it's nice as some people are. It's just right. So I think when you think about it, is this going to properly represent marginalized groups? Is this going to produce content that is free of some of those discriminatory things? Not as much as the Internet is going to. And then when you think about the other stuff, I would say it's not particularly good at any time of high level reasoning or strategy. It's not. It's not particularly great at logic or reasoning in terms of the way a person might be as good at it. But that's something that can probably improve, but that's good. And I would say the other one that's a little controversial is ChatGPT doesn't understand. I think people think of artificial intelligence, like, Oh, I put in a thing in ChatGPT understood. No, it didn't. It just matched the pattern. That's it. It's the same thing if you go to midjourney or Dolly, the image in the platforms. Let's say, show me a picture of a person. It doesn't know that a person has two arms and five fingers per arm, and it doesn't understand any of those things. It just understands that this is the way pixels look for people. And that's why there was a recent lawsuit, I believe it was in January, it was filed by Getty because Midjourney was... I'm sorry, Dolly. Let's not slander Midjourney. Dolly was putting the Getty watermark on images it was generating because they trained it on a data set of Getty stock photos. It didn't understand that it was putting a watermark in there. It was just like, images that look like images that looked like this tend to have this weird pixel set up in the corner with a little line through it that says stock photo.

Tim Fitzpatrick
So we're going to put that there.

Sam Tomlinson
Because that's what's in the pattern. It matches the pattern. So I think that's that. And I think the last one is an L is great for anything where you can tolerate 5 % error.

Tim Fitzpatrick
Got it.

Sam Tomlinson
Which is if you're talking about social media copy? Sure, who cares? If you're talking about the procedure for craniotomy? Yeah, 5 % error seems bad.

Tim Fitzpatrick
Yeah, it's not good, right?

Sam Tomlinson
If you're talking about options trading, 5 % being 5 % wrong about how a long straddle works, that's not acceptable. If you're talking about what's the best way to make a paper airplane? Yeah, who cares? It's a relatively low leverage bet. But there are plenty of cases where this also applies to code. If 3 % of your code is wrong, that thing doesn't work. You can't tolerate that. So not great. It's just, I think, one of those things where you have to understand the tolerance for error and the impact of that error on the output.

Tim Fitzpatrick
Well, one, the current data that ChatGPT is pulling from, is it 2021 or 2022?

Sam Tomlinson
So I believe ChatGPT 4.0. I believe that model includes, I think, up to. Early.

Tim Fitzpatrick
2022.

Sam Tomlinson
Early 2023. Okay. Is the GPT 4.0.

Tim Fitzpatrick
So whoever's when you're watching or listening to this, this is a moving target. My point in asking that question is ChatGPT is not pulling live data. It's not pulling stuff from the internet. No. It's got a data set that it's working with. So it's also, I think, important to keep that in mind as you're...

Sam Tomlinson
Yeah. And I mean, there are some cases where some applications like Bings' iteration of this can access some things on the web selectively, which is interesting. But by and large, most of these models they ingest and then they produce. They're not... The same way Google crawls the internet 24 7, 365.

Tim Fitzpatrick
Yes.

Sam Tomlinson
And LLM is not. And I think one of the things when people don't realize is the cost to then to crawl the web and then to train an LLM on the latest generation of the web is astounding. We're talking probably orders of magnitude more than what Google currently pays for search. So at this point, it's not economically feasible.

Tim Fitzpatrick
Tthat's an interesting thing to keep in mind long term, right?

Sam Tomlinson
Yeah. Everyone was talking about how chips are dead. It turns out chips are very much a very good business these days. These things go in cycles.

The Top Use Cases of ChatGPT for Business Owners

Tim Fitzpatrick
What do you see as some of the significant limits to ChatGPT that may not be obvious to a lot of us that we're not thinking about?

Sam Tomlinson
So I think the big one is obviously that these are not... An LLM is not an information retrieval system. It's not. Search is an information retrieval system, and it's a really good one. An LLM is not, by definition, going to do that because the LLM generally works on some... It works on a probabilistic model. So we all know that there's a lot of garbage on the internet. All that garbage gets fed into the model and the outputs of the model are largely contingent on the data that was used to train it and the data to which it has access. Now, in most cases, ChatGPT, generally speaking, if you put in an answer like, Hey, go to the website this, it was like, I don't actually have access to the internet. Because it's not trained on that data. So when you think about what's it good at, it's not going to be information retrieval because number one, it doesn't always have an up to date corpus of the actual internet. And number two, it has no way to distinguish between what was written on Reddit by a random, say, neofascist user and what was written in the New England Journal of Medicine by a world renowned cardiologist. It's like two pieces of information. You know what I mean? The ability of the model to understand and properly weight different sources of information is still not great. And then I think when you start to get into it even more, you're like, Okay, what else is it not particularly great at? One is any minority opinions, or a marginalized group, or anything like that where the machine just doesn't understand that it's talking about certain things. I think it's more of a societal mirror. ChatGPT takes in all the information that we have on the internet, and some of it's really ugly, and it gives it all back to us just reflected. So I think anytime you have a marginalized group or you talk about racism or sexism or ableism, ChatGPT is just as terrible as some people are. And it's nice as some people are. It's just right. So I think when you think about it, is this going to properly represent marginalized groups? Is this going to produce content that is free of some of those discriminatory things? Not as much as the Internet is going to. And then when you think about the other stuff, I would say it's not particularly good at any time of high level reasoning or strategy. It's not. It's not particularly great at logic or reasoning in terms of the way a person might be as good at it. But that's something that can probably improve, but that's good. And I would say the other one that's a little controversial is ChatGPT doesn't understand. I think people think of artificial intelligence, like, Oh, I put in a thing in ChatGPT understood. No, it didn't. It just matched the pattern. That's it. It's the same thing if you go to midjourney or Dolly, the image in the platforms. Let's say, show me a picture of a person. It doesn't know that a person has two arms and five fingers per arm, and it doesn't understand any of those things. It just understands that this is the way pixels look for people. And that's why there was a recent lawsuit, I believe it was in January, it was filed by Getty because Midjourney was... I'm sorry, Dolly. Let's not slander Midjourney. Dolly was putting the Getty watermark on images it was generating because they trained it on a data set of Getty stock photos. It didn't understand that it was putting a watermark in there. It was just like, images that look like images that looked like this tend to have this weird pixel set up in the corner with a little line through it that says stock photo.

Tim Fitzpatrick
So we're going to put that there.

Sam Tomlinson
Because that's what's in the pattern. It matches the pattern. So I think that's that. And I think the last one is an L is great for anything where you can tolerate 5 % error.

Tim Fitzpatrick
Got it.

Sam Tomlinson
Which is if you're talking about social media copy? Sure, who cares? If you're talking about the procedure for craniotomy? Yeah, 5 % error seems bad.

Tim Fitzpatrick
Yeah, it's not good, right?

Sam Tomlinson
If you're talking about options trading, 5 % being 5 % wrong about how a long straddle works, that's not acceptable. If you're talking about what's the best way to make a paper airplane? Yeah, who cares? It's a relatively low leverage bet. But there are plenty of cases where this also applies to code. If 3 % of your code is wrong, that thing doesn't work. You can't tolerate that. So not great. It's just, I think, one of those things where you have to understand the tolerance for error and the impact of that error on the output.

Tim Fitzpatrick
Well, one, the current data that ChatGPT is pulling from, is it 2021 or 2022?

Sam Tomlinson
So I believe ChatGPT 4.0. I believe that model includes, I think, up to. Early.

Tim Fitzpatrick
2022.

Sam Tomlinson
Early 2023. Okay. Is the GPT 4.0.

Tim Fitzpatrick
So whoever's when you're watching or listening to this, this is a moving target. My point in asking that question is ChatGPT is not pulling live data. It's not pulling stuff from the internet. No. It's got a data set that it's working with. So it's also, I think, important to keep that in mind as you're...

Sam Tomlinson
Yeah. And I mean, there are some cases where some applications like Bings' iteration of this can access some things on the web selectively, which is interesting. But by and large, most of these models they ingest and then they produce. They're not... The same way Google crawls the internet 24 7, 365.

Tim Fitzpatrick
Yes.

Sam Tomlinson
And LLM is not. And I think one of the things when people don't realize is the cost to then to crawl the web and then to train an LLM on the latest generation of the web is astounding. We're talking probably orders of magnitude more than what Google currently pays for search. So at this point, it's not economically feasible.

Tim Fitzpatrick
Tthat's an interesting thing to keep in mind long term, right?

Sam Tomlinson
Yeah. Everyone was talking about how chips are dead. It turns out chips are very much a very good business these days. These things go in cycles.

Tim Fitzpatrick
So we've been talking a bit about some of the downsides to this. Let's talk about the positives. What do you see as the top three to five use cases in its current form for business owners?

Sam Tomlinson
I think there's a ton of really interesting stuff. And I think a lot of this is there's a mental model called the zone of genius. And it basically says you have four quadrants, so to speak, of how you resolve problems. You have your zone of incompetence at the bottom, which is basically stuff you're just terrible at, and you don't know how to do it. Your zone of competence, which is you can do, but everybody else can do it too. You have a zone of excellence where you can do it better than most, and you have consistently high feedback. You're pretty good. And then there's a zone of genius, which is where you have something that you love doing and that you are one of the best in the world at doing it. If you think about that mental model, I think ChatGPT allows you to spend much more time as a business owner in those zones of genius and excellence, because a lot of the stuff that you are either not good at or you're just merely competent at, you can push on to some really ChatGPT or a ChatGPT powered device, powered application. So when you think about this in practical terms, that means content summarization. I don't do that anymore. You can give ChatGPT a thing and be like, Hey, summarize for me the Intell Investor, in 2,000 words. Give me four pages on what this actually means and the principles underlying it. It will do a pretty good job of that. It may not be perfect, but again, you can tolerate 5 % error on Intell Investor. Cool. Personalization as a business owner. I know a lot of business owners, they do sales. You're constantly reaching out to prospects. You're constantly doing this. You can give it a general template like, Hey, these are the services my business offers. This is what makes us unique. Provide a custom cold email to a business owner in the marketing space about why they should use my whatever thing based on this information. And ChatGPT can write you a very nice little cold email personalized directly to the owner of that business. And there's even Google Sheets ad on to do this for you. You can literally have it go through. Here's the list of my prospects. Here's their name of their company. Here's their website. Okay, ChatGPT, based on this information and here's information about my business, go write me a custom pitch for each of these. It'll do that. Super big time saver. Pretty cool. Really huge time saver. As a business owner, that means that you can five, ten X out your cold outreach. That could probably result in a lot of money if you're somewhat competent at sales. I think language translation, it's great at translating stuff. We use this sometimes. We have a couple of clients that are international and we need to translate something real quick. Hey, here's this press release in English. Make me a version in Mandarin, please. And it does a job that is 98 % acceptable. And there might be two things that are slightly off or I missed a little nuance here or there. But for translating a press release in, say, three minutes, fantastic. I think it's great for, at the very least, some types of research. If you're like, Hey, find me the seven biggest objections to hiring a head of marketing for my startup. I need to present this to my board. I want to understand what their objections are going to be so I can be read. Be ready. It will do that. It will give you seven things that people object to and why they shouldn't do it. And you can come up with responses. Same thing for if you're a writer or you're looking to produce content. Here it is. Write me a blog post on this based on this. It does that. It's great for content generation. I think it needs editing for sure. But it gives you a sense of what the minimum acceptable threshold is for content and where the state of the market is. So as a business owner, as a marketing executive, as a whatever, you can look at this and say, Okay, this is the baseline. How can I take this to the next level? What did I miss? So now you can spend your time, again, doing that higher leverage activity, which is adding high value content into what ChatGPT made versus trying to start from a blank page and figure out what I want to write.

Tim Fitzpatrick
Yeah, it gives you a great foundation to build from.

Sam Tomlinson
100 %. It's a productivity enhancer. Which is awesome. I use, for all of my Zoom meetings, I use Otter AI, which is like a note taking app, right? But then I can feed the transcripts from Otter AI into ChatGPT and say, Hey, this was my 12 o'clock. Provide me a two paragraph summary of the meeting. I get a two paragraph summary of the meeting based on what Otter AI transcribed during that hour long call. And now I've got all that information right here, which is super helpful.

Tim Fitzpatrick
It seems to me like one of the... You touched on this, right? The data that it's pulling from the quality of that data dictates the quality of the output. But the other big thing that determines quality of the output is the prompt you actually put into ChatGPT. And it's interesting to see that evolve because I think a lot of us are still pretty new at this. And it's like, oh, then probably didn't do a very good job on that prompt. But what I'm seeing, too, is there's people that are writing prompts that already you can load right into chat GPT and play with it. So there's a lot of really exciting things going on there that's cool to see unfold.

Sam Tomlinson
Yeah. I think for the next, say, 12 to 24 months, prompt engineering is going to be a big thing. Because to your point, the outputs are so conditional on the input that it makes sense to understand the right ways to craft those inputs. But I think once you get past that, you're going to start to see some of those potential, we're not sure about this, go away. You'll start to see some of that either in the form of the platform suggesting like, hey, actually, this is a better prompt. Are you okay with that? Or, hey, we noticed your prompt is missing context. This is the context that it actually needs to produce an acceptable result. So can you please give us piece A of context, piece B of context, piece C of context so we can generate a more relevant response. And once that starts to happen, I think the incremental value of prompt engineering declines.

Tim Fitzpatrick
It goes away.

Sam Tomlinson
But for the next 12 to 24 months, I think you have a huge arbitrage opportunity. If you are better at it than everybody else, you will get better content out of ChatGPT faster, which, if you go back to that four models framework, spends, again, more time in that zone of excellence or the zone of genius because you don't have to worry. You don't have to put in seven prompts to try to get 15. You just know exactly what to put in there. You get in there, you get your answer. And by the time guy who doesn't understand the prompt finally gets an acceptable response, you're on a task seven or eight. You're running down the road ahead of them. But again, I don't think it's going to be a durable advantage.

Tim Fitzpatrick
Yeah, it's going to be interesting. You touched on this, too, where there's so many different tools out there that are starting to incorporate ChatGPT. I know Google's got theirs. I can't remember the name of it off the top of my head, but..

Sam Tomlinson
Bard.

Tim Fitzpatrick
Yes. So you can bet Google's not too far behind.

Sam Tomlinson
I think Google is probably... This is a controversial thing to say, but I think Google, from a technological standpoint, is probably ahead. But from a go to market standpoint, I think they're behind. But then again, you look at ChatGPT it's based on a transformer. Google's deep mind is the one that actually wrote the original paper on that. They're the ones that invented it. And they did it five or six years ago. They did it in 2017, I think. Google's got, I would say, probably the deepest AI team in the world. It's one of those cases where I'm looking at like, All right, this is a really bad metaphor, but there's only so long to keep LLM over on down. Yeah, you know what? Maybe Open AI did get a couple of games on him real quick, but it's like, all right, they're catching up really quick and they've got a huge advantage in terms of data. They've got a huge advantage in terms of talent. They've got a huge financial advantage.

Tim Fitzpatrick
Yeah. We're only in the first ending of this ball game.

Sam Tomlinson
To be very clear, Open AI won the first ending. They won. The fact that ChatGPT has gotten the adoption that it's gotten in the period that it's gotten it is remarkable. And they've certainly moved fast and they've done some great things. But I think before we anoint them winner of the AI Wars, we should probably at least see a few more in these baseball.

Tim Fitzpatrick
Yeah, totally agree. Well, it's interesting, man. I signed up for... Somebody sent me something about an email list where the guy sends out just AI updates daily. And what I found fascinating is I think this guy started this thing three months ago. He has over 100,000 email subscribers at this point. And he started his aside gig. And he just put out an email going, Hey, I just left my job so I can focus on this full time. But it just goes to show you how much demand and interest there is in this. Right now, it is a super hot topic. It is changing quickly. I just think we need to be open minded as business owners about it and stay on top of it and embrace things as they change and use it to our advantage. Because like you said, I think from a productivity enhancement standpoint, right now, where it's at, it can save us so much time. We just need to be open to how it can actually do that.

Sam Tomlinson
100 %. I also teach at Hopkins in Georgetown. And I know in academia, there's this whole thing about ChatGPT where we don't want it. It's scary. It's bad. We're going to ban it from assignments. And I'm like, But why? You don't get to close Pandora's box. Students are going to use this one way or the other. So your options are either teach them how to use it and give them the tools to succeed in the modern workforce, or try to repress it and lose because you will lose. So in my classes, if you use one, use ChatGPT, go for it. I don't care. But you're held to the same standard. So ChatGPT makes a mistake and you don't catch it.

Tim Fitzpatrick
That's a problem.

Sam Tomlinson
That;s a problem. That's a huge problem. It's no different than a math class. We don't make you do multi variable calculus by yourself. We have rule for them. It solves it for you. But you have to understand what to put in and how to evaluate the response and determine if the response is correct based on what you put in. Same principle. So I think it's about, like you said, understanding how to use it. And I'm hoping that more colleges get better at it because we're in a place where ChatGPT is a reality. It is not going away. It will never go away at this point. Pandora's box is open. So as employers, we want people that can come in to actually understand the technology and understand how to use it and can do some of these things like prompt engineering and can do some of these things like understanding the right context to give it so that we can get the right response. And then when they get the response, being able to look at it critically and add in those higher order bits that ChatGPT doesn't get below, like big picture strategy, like logic and reasoning, those are big picture bits that we need a person to come in and make it better. And that's a skill. That's a big skill, at least for... I'm not sure what your business, but in our business, that's a huge skill. So I want people to be better at it because it's better for us.

Tim Fitzpatrick
Yeah. And like you touched on this, too. ChatGPT and similar technologies, at least at this point, from a strategic standpoint, it's not there.

Sam Tomlinson
No. I don't think it's. Candidly, I don't know if it ever... I don't think an LLM ever gets there.

Tim Fitzpatrick
I Don't think so either.

Sam Tomlinson
Because of the inherent limitations of the model itself. Now, obviously, if you start to Frankenstein these models together and you start putting in an entity or identity graph like what maybe Google has been building for the last decade, wonder why they invested a ton of money in that. Who knows, right? And then you start adding in a reasoning engine and a logic engine, you start being able to combine these things together in a really interesting way, then I think you get closer to something that can think strategically. But until then, ChatGPT is a high school sophomore. Maybe it gets a little smarter, maybe it becomes a junior in a year or two. But there's a reason high school juniors generally don't run multibillion dollar enterprises.

Conclusion: ChatGPT: Understanding The Limits and Possibilities

Tim Fitzpatrick
That's right. This has been a great conversation, Sam. Any last minute thoughts you want to leave us with today?

Sam Tomlinson
I think A lot of it is just is what I said at the end. This is a part of what we're doing now. Pandora's box has been opened. The Bell has been rang. So now we have to figure out how to live in this world and how to make the most of what we have in this world. I hope people spend time playing with it. I hope people spend time using the models and understanding the limitations because if you can, you'll get ahead. And if you can't, sorry, you'll lose. But it's exciting.

Tim Fitzpatrick
It is exciting. Where can people learn more about you?

Sam Tomlinson
Sure. Best place, I have a website, samtomlinson.me. I write a newsletter every week called The Digital Download, which is one of those additions is what spurred this chat. So you can sign up. It's free. I don't do ads or anything. It's just a newsletter. I work at Warschawski. If you have any marketing questions, you can help with advertising, warschawski.com. But that's it. And if you want to talk to me personally, I'm on Twitter, digital Sam I am.

Tim Fitzpatrick
Cool. I love it. Connect with Sam, reach out. We'll make sure those links are in the show notes. I have enjoyed connecting with you twice now. So thank you so much for doing that. Like I said, I'm super impressed. So reach out, connect with Sam. Those of you that are watching, listening, I appreciate you. You can always connect with us over at rialtomarketing.com. If you want to discover which of the nine revenue roadblocks are slowing down your growth, you can do that over revenueroadblockscorecard.com. It takes less than five minutes to discover and assess which roadblocks are slowing down your growth. So thank you so much. Until next time, take care.