Who is Maciej Chmurkowski?
- Who he is
- Maciej Chmurkowski, Enterprise Product Owner at Senuto, an automation and robotics engineer by training.
- What he does
- He has analyzed hundreds of thousands of answers from ChatGPT, Gemini and Perplexity to check whom AI really recommends, whom it cites and how long that lasts. Before joining Senuto he ran his own affiliate business for over a decade. His two favorite words are automation and scaling.
- What we talk about
- What brand visibility in language models really is (mentions, citations and sources), why you cannot measure a position with a single check, what entities are, how to study LLM answers, what influences whom AI recommends, and what to do on Monday morning.
- Where to find him
- LinkedIn (Maciej Chmurkowski) and talks at SEO conferences, including I Love Marketing in Poznań.
Best quotes
„Chats are non-deterministic, meaning the answer changes so often that when we check once, we cannot tell how visible we are.”
„You would have to simply check systematically and look rather at the presence of your brand, not at the position.”
„Buying brand mention publications just to be visible in LLMs, without surgical, precise targeting of topic and intent, simply makes no sense.”
„A transcript is very important, because it is the transcript, not the video, that gets into these models.”
Recommended materials and sources
Guest’s pick
Related conversations
A Winner’s Mindset in Business: “Competition Is My Fuel” A Conversation with Michał RochwergerMichał RochwergerShared topics: SEO
AI Implementation Strategies in E-commerce with Damian SałkowskiDamian SałkowskiTheme: AI visibility
AI Is Not an Add-on, It Is the Core. What Does an AI-Native Agency Look Like? | Robert NiechciałRobert NiechciałShared topics: SEO
Artificial Intelligence and the Future of Marketing with Kas SzatylowiczKasia SzatyłowiczTheme: International growth
Transcript
Host: Rafał Moszkowców. Guest: Maciej Chmurkowski.
Introduction
Rafał: Hi! This is Rafał and a new episode of Globalne Horyzonty. Today we talk about how to build visibility in LLMs and what that even means. It turns out that almost everyone understands it differently. And my guest understands it probably best in Poland, because instead of guessing, he went and counted all of it. Maciej Chmurkowski, Enterprise Product Owner at Senuto, a man who analyzed hundreds of thousands of answers from ChatGPT, Gemini and Perplexity to check whom AI really recommends, whom it cites and how long that actually holds there. Before joining Senuto he ran his own affiliate business for over a decade, and by education he is an automation and robotics engineer. So when he says his two favorite words are automation and scaling, it is not just a LinkedIn pose. Maciek, welcome to Globalne Horyzonty.
Maciej: Good day. Thank you for the invitation.
Rafał: I am extremely glad you are here and that we have such a very nice topic to discuss.
Maciej: Oh, I am very happy too, because this is a topic where I feel like a fish in water, so we have a lot of data in the background. That data is confirmed. And you know, when you talk about something that is certain, it is completely different. You do not have to stitch anything together, so to speak.
Rafał: Certainly, 99 percent, I assume.
Maciej: Certainly, 99 percent. Okay, I would also add two things to this introduction right away. There is also Michał Suski, who studies this too, on a different scale, and I would like to say hello to him from here. We also have very nice conversations with Michał on this topic. And I would add a few more platforms, because there is already Copilot, there is AI Mode and AI Overviews. There are quite a few things being added there.
Rafał: Yes, and it keeps growing. Recently I heard a request like: "Hi Rafał, listen, we would gladly buy from you a service of positioning in LLMs", whatever that means. I admit I do not know myself, but fine, "but we would like to be in DeepSeek. In DeepSeek is where we would want to be."
Maciej: Oh my! We also have an offer to do monitoring even in Claude, which really only has an API, so there is no public version. Maybe let us start with simple things: the monitoring we do, the data comes from the public versions of the chats, meaning we scrape it. It is not an API, as someone sometimes suspects, so it is a different kettle of fish, and it is the answer closest to the one a human can get. Closest, not the same, because here I have rambled on a bit too.
What visibility in LLMs means: mentions, citations and sources
Rafał: No, no, nice. Because actually I wanted to ask this question so that we start from there.
Maciej: Right.
Rafał: Around this, as everyone who listens to this podcast, and you have probably heard it too, there are very many myths and very many people do various strange things, and in fact nobody knows what is true. So I wanted to start from zero and hear what you think. When someone says "my brand is visible" or "invisible in ChatGPT", what does that actually mean? Because I seriously think that as many brains, as many people understand it completely differently.
Maciej: Well, where to start? Maybe I will start with the basics, the way I divide it all, because very many people confuse citations, brand mentions, meaning brand visibility, and also sources, to complicate it even more. So we have this answer in ChatGPT and we can really have three places, three areas where our brand can appear. Brand mentions is really a mention, the name of our brand mentioned in some context. Citations is the list of attached pages from which, for example, the citations for this answer come. And sources are the search sources. This answer is also built so that if the model is not sure about the answer or web search is turned on, the model also searches the internet. Those are three places. And it depends what visibility we are talking about. Because brand visibility, that is probably what matters most to us: how a given model describes our brand.
Rafał: Give an example, so that we get it well. An example where the brand is pulled out, so that we understand these differences.
Maciej: Generally, for example, you ask: what are good running shoes for asphalt. And the model answers that the best shoes for asphalt are such and such, but that Adidas works best. The second choice would be, for example, Nike. And these names appear without a link. But there is also so-called shopping, meaning we have a list of attached products from Merchant Center and so on, with links to stores. We also have links to sources. Adidas can also be mentioned in citations. So part of the answer can come from an Adidas subpage. That is a citation. And sources is when the model, searching for this answer, wants to complete it and also finds, for example, an Adidas page, some subpage, because it does not have to be the same subpage. So those are already three places.
Rafał: I know this is a bit hard on this podcast, because we would need a chart. We have to do a webinar.
Maciej: Yes, exactly. Let us focus for now on the brand mentions. That is, whether the brand is mentioned and in what context.
What should an online store care about in LLMs?
Rafał: Okay, let us try to ground this this way. Say I have e-commerce with products. What should I care about most?
Maciej: In my opinion, what you should care about most right now is shopping, which in Poland and in Europe is still limping because of European law. But in the States, ChatGPT has this shopping in most answers. Besides the product being mentioned and its price being shown, after you click the product a side panel opens where there is already even a bit of a ranking, because there is a first place, three URLs to stores. So here it would be worth not only being mentioned. I mean, probably if the brand is mentioned in first place, that also guarantees that you will be in that shopping. As for e-commerce, I think that is what to focus on.
Can you talk about position in LLMs at all?
Rafał: Okay, because you mentioned ranking, position. The next question comes to my mind right away, so we are going off script, let us go.
Maciej: Right.
Rafał: As always when we meet. In classic SEO we have position. We have an index, in Bing or in Google, we have some positions. Without going into details, how it changed over time and so on, we have positions, we are able to measure it. We also have tools for measuring SERPs with already twenty years of history. And here, in the context of LLMs, can we really talk about any position at all? To add this for our listeners: I know you understood what I mean, that you are able to measure: okay, this is organic position number one, and this is a tool that will measure you an averaged position, for example over the last 24 hours for the query "running shoes" in Polish.
Maciej: Yes, because there the position simply does not change from hour to hour, from minute to minute.
Rafał: So we will eliminate personalization.
Maciej: Yes, in chats we also eliminate personalization here, because we are not able to reproduce it.
Rafał: Okay, but how do you talk about position in ChatGPT at all?
Maciej: I will divide this answer into two parts, because it is very important, probably one of the most important things I would like to tell you today. Let us start with the fact that you can measure these positions, because first, you can see which brands are mentioned one after another, and in that context it is best to be mentioned as high as possible, because often this answer, if it is long, is collapsed and you have this above the fold. I do not know if people talk about it and have noticed, but it exists. For example there is part of the answer and some people will click only on what is at the top. Not everyone will read to the end, expand. That is one thing. And there are these positions. Citations have their positions too. Although here I honestly do not know: you are cited, your page is cited, that is fine, but do you have to be in first place in those citations? Practically nobody clicks citations, rather sources.
Rafał: Right, sure. And is there research on that? Whether anyone clicks or not?
Maciej: We do not have that analytics and that is the biggest problem. But the other side of the coin, which is just as important, also corresponding to those places in SEO rankings: chats, not only GPT, all these elements, this is not the new SEO. Here you cannot measure position deterministically. That is a very important word. Chats are non-deterministic, meaning the answer changes so often that when we check once, we cannot tell how visible we are, whether we are visible. You know well, because you saw my presentations in Poznań. We will probably talk about numbers in a moment, but first this kind of foundation. A single measurement of your brand's presence in an answer may show that you are in first place, and it may show that you are not even in this answer at all. So if you hit that answer where you are, well hurray, you are happy, you are an agency and you make a report for your client: sir, we are here. The client enters the same prompt and where? These positions bounce around too. When I give you the numbers, we will probably talk about it, this is also impossible to determine with a single check.
Maciej: So all these myths, what is happening, I am a bit horrified that the industry is going toward this being the new SEO, that we monitor position, report, that we will be positioning. Nonsense, bullshit. You cannot monitor this with a single check. You would have to simply check systematically and look rather at the presence of your brand, not at the position. Of course a higher place is nice, because you are mentioned sooner. But more broadly, how does this model perceive your brand?
Rafał: Are you recommended?
Maciej: Sentiment almost does not exist, we can talk about that for a long time too. There is no negative sentiment, and positive and neutral, I have a study on this, are so close to each other that it has its moment. Above all, I have now focused on whether your brand is recommended, whether it is the first choice, whether it is recommended as an alternative, on what topics, in which entities, around which entities your brand appears, because that later gives us the ability to fill those gaps. If we measure an entity where the competition is and we are not, we can turn the situation around very quickly, find the places where we are not, and publish the right materials. Because there is one more thing.
What is an entity in AI search?
Rafał: Oh God, I agree in a moment. Maciek, you are on a roll, these are important things, but just tell me, what is an entity?
Maciej: I define it as a being, an existing thing. It is a kind of node that is defined. Quickly, my favorite example: Barcelona is an entity, because it is a city. The football club Barcelona is also an entity, because it is a being. But every entity has its relationships. Barcelona as a city has a relationship to the football club Barcelona, because that club is located in that city. Lewandowski has played in the Barcelona club. Lewandowski is also an entity, because he is a being, but he has a connection, because he plays for the club. Lewandowski has a wife, Anna, that is also a connection. So it is a kind of being that is defined. It is a person, a city, it can even be some process, and every entity has to have a relationship to another entity. Our digital world is described by entities, and that is true both in SEO and likewise in LLMs. Unless you want to add something.
Rafał: No, no, no, I agree, but I am not much of an expert in this matter either.
Why is simply buying publications not enough?
Maciej: Okay, and now we were at those gaps, at how these models see your brand. And now buying brand mention publications just to be visible in LLMs, without surgical, precise targeting of topic and intent, simply makes no sense. And that is again a myth, because it is growing in strength now: buy brand mentions, publish. I was also in Lithuania, at a big international SEO conference. Every second presentation said: here you can buy a hundred publications. Preferably from us. Bullshit, simply. And will those hundred identical publications give visibility? No. You have to be visible on different topics, in different intents. That is where the dog is buried. And on top of that, imagine all of this together with the non-determinism of the models, meaning you can be mentioned, you can not be, alternating.
Rafał: I am afraid that what we have now is "positioning" in quotation marks, I will call it that, because there are 1,500 names for it, in LLMs. Just as SEO was hard to explain to business, you would probably agree that this is a very hard service to explain to someone who does not understand it or never had closer contact with it. Now it is even worse, because before you could say: you know what, Mr. President, you will be in high positions, and those high positions are between 1 and 5. And today we can say: a hundred thousand is needed and we do not know at all what will happen. Is that not dangerous?
Maciej: Dangerous in my opinion, because it carries the threat that if someone does not understand, then maybe it is too hard, but the truth behind it will not be visible. So we can complicate it like that, make ourselves a new service and rake in the cash. Sorry, I will be speaking today, I am poking a stick into an anthill, because I think the direction our industry is taking is very bad. I mean I do not like it, it is fleecing people, especially since this traffic is for now...
Rafał: ...a few percent, supposedly. True.
Maciej: ...because we cannot measure it, it converts quite well. I do not have a store, you know, and I do not run an agency either, so I have no clients, but I suspect that analytics probably already exists in Analytics or somewhere, right? In conversions, right? So that is at least something.
Rafał: Yes, those are still small numbers, because analytics is also quite lame, as it always was, now it is lame, shot in the knees, and I think we are all waiting for how to arrange it better. But when it comes to B2B, because with stores it varies, here, I run a small agency, so I admit, guilty as charged, in the last half year, maybe even longer, all the most aware clients, who came to us already truly educated about us, about me, about Radek Paluszak, whom, by the way, I warmly greet, a sharp guy, came from LLMs, mainly from ChatGPT and from Gemini. That is why I wanted to talk about this, since I know everyone asks us about it, but I am still curious myself how else we can improve it so that our visibility is greater.
How are LLM answers studied?
Rafał: But let us go a bit further. We talked, when we last saw each other, and I know you have analyzed hundreds of thousands, or maybe already a million AI answers, meaning results for prompts. Let us call it that. Maciek, how do you do something like that technically? After all, ChatGPT does not have search results you can download, and you probably did not type it in by hand. How do you do it?
Maciej: We will go a bit fast at the very beginning. I started saying that we actually download the public version, meaning we scrape. Scraping is downloading the full answer, the HTML of the whole page. So that is possible in ChatGPT, Copilot, Gemini, AI Mode and so on. Claude is not a good example here, because it has no public version, you have to be logged in. We cannot download this information after logging in, because it is against the terms, and then we are really stealing data. So what is public and available, there is no problem scraping it.
Maciej: And once again there is a huge difference, because each of these models has its API. Some tools, which is also a horror, are based only on the API, they query the language model directly, which is a slightly different environment, and sometimes a different model altogether. Because here we have ChatGPT, it is our layer on top of that model, and it has its settings, algorithms, its system prompt, which changes completely the answer we get. So if we want to be as close as possible to how people see our brand, we cannot use the API, because that makes no sense. I did a study with a partner, I hope the report will come out soon. We compared it to human answers, because they collected them. And indeed the API comes out worst. Scraping is closest to those human answers, but listen, it is still not one hundred percent, because there is the persona.
Maciej: Are you logged in? Most people are logged in to chats, really. So this persona, I looked into this a bit too, is built not only from search history, but also from conversations. And the model builds your profile. It builds it in such a way that it knows even your favorite phrases or what you do. It is able to determine what kind of person you are. And this profile, only we do not know how, is probably part of the prompt, say thirty-four lines, that is passed to the model and it also influences this answer. It is based, for example, on your experience. So if someone asks about running shoes, and the model knows they are a runner, it will not recommend models for beginner runners. And that is already implemented. And in some systems we want to simulate this too. We will probably create such personas and study it that way, to get even closer to this answer. But everyone who does monitoring has to know that this is how it looks and that the client gets this report. Let us say it is close. The closest it can be under current conditions.
Where is search in ChatGPT heading?
Rafał: And where do you think this will go? Will there be any precise data on this? Given that, for example, OpenAI introduced ads in ChatGPT?
Maciej: Yes. I watch it very closely, because I also have a lot of clients from the US market and I watch many prompts there.
Rafał: I really like the direction it is going.
Maciej: It is the new Google, listen, because there are already maps in the answers, there are recommendations, so a bit like reviews. So it starts to grow into, you know, a large search ecosystem, and always what is introduced in the States later comes to Europe, like Google also does a bit, only here it of course depends on the law. The European Union has somewhat different rules and it is easier for them to introduce things there. They already have a lot of those snippets, additional things in answers, but it suggests which way it will go, how it will develop. So I think we are only at the beginning of this road and it is still worth taking an interest in it, because it will only grow and grow.
Rafał: I mean, gosh, search in ChatGPT is growing like an avalanche, basically. I observe it in myself too: when I search in a classic search engine, I search in Bing, if anything. Google seems totally overloaded to me, AI Mode gets on my nerves, but I have an agent built, I mean based on DeepSeek, which uses that browser and chooses and searches for me, it orchestrates these things for me, so it also has additional memory and so on. I think for somewhat more advanced users this will be the future, but I wanted to ask about something else, actually. Do you see ChatGPT becoming more of a new Google, a more personalized one? And Claude basically just for, I do not know, writing code, for the terminal, and for something else?
How to prepare a website for LLMs: structured data and meta tags?
Maciej: You know, there is one more area we have not talked about that may determine all of this. It is about how the architecture of websites itself is changing. I am not talking about llms.txt, I am talking more about preparing a website for robots, which is how it is said, but really for LLMs. And that is very important, because I already see a very strong impact of structured data, which I am testing. We recently ran some tests with entities placed in structured data, and we are checking how that affects presence in LLM results. That is, we give the answer faster, it can be pulled out in an easy way.
Maciej: But from what I read, new standards are also coming, in which of course Google is also trying to outdo others. It is still not a uniform standard in which a website will simply be saved in Markdown in separate files, with some kind of map to those Markdowns. Your organization, what is on the site and so on, would be described in a different way. If that comes in and starts to work, it will turn out that websites will not be needed by us at all, because every model will know their content. It will be more convenient to just chat with the chat than to look for that information, for example, on a page. That is a long way off, but if it arrives, in my opinion we will move from this kind of Google search, apart from you and me, to LLMs for the most part. So it seems to me. And I am already starting to do it myself, because lately I have been playing a bit with SEO, but also with the kind of SEO that will work well in LLMs, and I am starting experiments with it and I think it will work.
Maciej: I take care of those meta descriptions and titles, which are the first proxy that the model sees right away and does not have to scan the page. By the way, I will tell you something interesting, because I also study and play a bit, as I said, and I noticed that in many cases the model searches for answers in web search, for example in Gemini or in Google, without entering websites at all. It simply pulls a list of pages from the SERPs, it has the title, meta description and the domain there, and on that basis it hallucinates a bit, but tries to predict what is on that page. It saves its resources. That confirms once again what I said in October at the SEO festival, that meta description and title are very important, because they have to say good things about your page, I mean summarize what is on your page. And imagine also structured data, which is easy to pull: pulling all the JSON-LD scripts. Every crawler will do it and skips all the HTML junk and other JavaScript. It needs few resources in itself, so all of this is already starting to work.
Rafał: I am laughing, Maciek, because there was a time when the main optimization consisted of headings, meta description and H1. Yes, and we are coming back. Then we stopped doing it, because Google will choose on its own, it is not that important anymore, and now we are coming back, healthily, because it is important. Recently, internally with the guys, I mean with the girls, with my crew, when we sit together, we laugh that basically we have SEO 2008, only we can do it much faster, we have more data and we are a bit smarter, but it is a bit similar to how we started.
Maciej: Yes, only we have tools such that instead of sitting for a month, in one day we create what we need.
How can SEOs level the playing field against algorithms?
Rafał: Exactly, and here I will share my philosophical thought. I always had this conviction that we as SEOs have a totally unequal fight with the algorithms. I mean we have our brains, a few crippled tools, we write ourselves a scraper and try to figure out what works, what does not work, to maneuver between all this, still working with market expectations, with competition and with those algorithms that have infinite money, those companies invest in them and still do not tell us what they do, because they do not have to. So I always had the feeling that this is no fair game and that is why I always smiled and blew kisses to all the black hatters who were able to simply upend the system, and I still do today. I think they are the coolest people in the world. What am I getting at? That I think now, as the nice English saying goes, this playing field is leveling.
Maciej: Yes.
Rafał: And we, having possibilities now of building tools, acquiring and processing larger amounts of data. For example, before, when you processed five, ten competitors while doing discovery for a client, even how to do reverse engineering to see what this competitor did, where, when they started, where they get their links, mentions, some history of their site, today you are able to query all one hundred, really, or two hundred, and do it faster.
Maciej: Or fetch data from some API.
Rafał: Exactly. So from the perspective of our work I think we live in the most interesting period. And what do you think?
Maciej: You know, I agree with you, because for me the most interesting period was 2012, 2013.
Rafał: I had top positions then too, so no, I am joking. I mean I had them, but I am joking, mentioning it.
Maciej: That is the most interesting moment, because the forces were equal, but also there is Black Hat still, and we can talk a bit about Black Hat too. The crawling of these chats is simply not looked after in this respect and you can do all kinds of tricks again, because Google has armed itself and it is hard. This cloaking has not existed for a good few years, and here you can. In a very easy way. I am not encouraging, listen, it is at your own responsibility. I am just playing around.
Rafał: I am not encouraging, but I will send you how to do it. We will not go off on this.
Maciej: No, but coming back to this leveling of forces.
Rafał: Yes, yes. So in my opinion it is a leveling of forces. And I only wanted to tell you, by the way, that we are able to influence models by injecting information. I do not know if you read about what they did with DuckDuckGo.
Maciej: I did not read it, but I have seen various effects of it.
Rafał: They killed Trump. Okay, fine. They injected it, and if you asked whether Trump is alive, no. But listen to this, because I just fell over laughing reading this article. I mean that much was injected. Now for our listeners, because we have already gone into an insider conversation of two madmen: you can, by publishing certain false information in sufficient quantity and making sure that this information is found by language models, influence what those language models think, in quotation marks of course, because they do not think, meaning how they process information and how they give information to the user for a particular question. And into DuckDuckGo, I think, 240 documents, articles, were injected. They killed Trump, but how did Trump die? Because according to DuckDuckGo, Trump died of rabies, which the vice president infected him with.
Maciej: Pure gold, really.
Rafał: Black hat, but there is a black hat. Okay.
Maciej: Let us break this down into prime factors, because I know a bit about this too. First, it is called retrieval memory, and it is not the training memory of the model, on which the model learned, and models learn mainly on Common Crawl, a data set from the internet. We can talk about that for a moment too, because it is important. And this retrieval memory can change. But what matters here is that they did it well. It is not about producing a hundred articles, mentions of your company, and thereby becoming visible, it has to be done in different intents and in different contexts, because it also has to be connected to each other. Here you see, you said yourself that they killed him with rabies, from the vice president. So there had to be some stories, which already means that this death appeared in various contexts in those articles. So the model grabs that quickly. Because if you appear only in one context, only in one intent, it is hard.
Maciej: And one more important thing, that Trump is an entity. A very strong, hard entity. The harder the entity, the harder it is to change in the model's perception. So respect for what they did, in general. Because a new entity you can invent: someone did it, made up a new Olympic sport and introduced it into the models, pony riding in a swimming pool, something like that. But that is a new entity. And he introduced it, made a hundred subdomains, AI-generated content and introduced it, because it is a new entity. These models swallow it like pelicans and there is no layer of protection yet. That is why this cloaking will work there for now. But it is very hard to do with an entity that is strong. There you really need big forces and resources or some good cloaking.
Rafał: We live in interesting times and many things can be done, but as we mentioned, we do not encourage it. Here we could also get into a conversation about the intentions of different people, because as a joke is a joke, but here I think that...
Maciej: That can end badly legally too.
Where does a model learn about a brand: training, retrieval, search?
Rafał: Okay, let us move on a bit so that we do not get stuck in this area, although I could really talk about it. Let us vary this podcast a bit. Let us just go a bit further, because I would like people who are not necessarily so hooked, like old light bulbs like us, but are a bit fresh to the topic, to take something from this, or at least to be able to go to their, I do not know, SEO freelancer or marketing team or agency and ask them good questions. Because as you said you are poking a stick into an anthill, well I am supposedly known for that too, although it is not particularly intentional that I want to knock someone over, but it terribly annoys me how many companies, how many agencies simply fool their clients, so let us just ask them good questions.
Rafał: Let us go on. Let us organize what determines whom AI recommends. So let us try to sort out a bit about those entities. How does a model even know a brand exists? We started a bit with Common Crawl, with retrieval. What is the source of knowledge? Training, search, one, two.
Maciej: Let us start with the basics. A model is trained for many months and later it is further trained on data from the internet. Most of this data comes from Common Crawl, whether it is 98 percent of this data or 95 percent, I do not remember these statistics, but roughly that volume comes from Common Crawl. It is a huge archive of the internet, in which there are, I do not know, five hundred, six hundred billion pages, and it is an enormous library on which models are trained. It is above all the base for these models. That is why these models, listen, are sometimes so stupid, because they are also trained on spam, which we also used to produce. Well, there is everything on the internet, and these are not scientific entries, these are not only articles from some decent newspapers, there is simply everything there. So these models learn from this junk. Later it has to be regulated. But okay, that is Common Crawl. For example, Wikipedia is only about one percent of the training data, plus some other sources. And that is the training data.
Maciej: So if your brand is in the training data, then it is recommended straight from the model's memory. It will recite it right away, without having to search for the answer on the internet. And you will recognize it by that. If you have, for example, web search turned off in ChatGPT, there is a snippet in the model where you see what the model is doing, you can expand it: am I searching the web now, or something, I did not find it, and it will keep searching. But if it does not do that and recommends your brand, that is very good, because you are in the training data.
Maciej: One remark, because I also overheard a conversation of the owner of Common Crawl, now in Zakopane. He said that from Common Crawl everyone takes only the English language, they have the page in English. So if you want to be in the training data, it is good to provide an English version of your site. A good tip. I was surprised, because I was actually sure it was the whole internet. No, these are English versions and it would be worth even checking it, but it seems to me that it matches. Only, for example, large Polish brands that do not have English-speaking markets in e-commerce, I saw they have fewer mentions than other brands. So maybe this is true. It would be nice to run some tests on that, but it would be good to know when exactly they are current. But you can probably figure out how to bring those Common Crawl robots to your site. So here is also that layer that you have to get a handle on. And that is the foundation, let us agree, the training data.
Maciej: Then there is the so-called retrieval, meaning that temporary memory. Let us say it is RAM. Information is read in there, and the more of the same information the model encounters, the faster it fixes it and holds it in that retrieval. It can change, it can drop out of memory. Like RAM: you reset the computer and you have a clean one. And that is how it looks. And what I said about the manipulations we talked about, that is possible. And finally there is the search layer, meaning the model has its own search algorithm. Everyone wrongly thinks that ChatGPT is based only on Bing. Absolutely not. At this point you have your own algorithm, which is some mix of Bing and Google, and it searches for information there, and it looks for pages in a way known to itself, which we can also talk about for a moment, because I also have some data on that. That is the third layer. It is best to be everywhere.
Rafał: What you said shows a bit that those companies or brands that earlier invested in the visibility of their brand in Google, in the sense that there were articles about that brand, it was mentioned, there was information and they also had their Wikipedia, then really, when LLMs were introduced, they landed in that memory right away.
Maciej: Yes, yes, yes. So that confirms the thesis that everyone states, which I do not quite agree with, because it is some correlation, that if you did good SEO, you will also be visible, but it seems to me it is based more on intent, on context, on how your brand is described, and not on whether you are in the TOP10 in Google. Because we know well that pages from the top fifty, the top hundred of Google are cited. I also did a study. In some AI models, cited pages are chosen completely randomly, and in others the dependence is exponential: the higher you are in Google, the sooner you are cited. As for ChatGPT and other models... remember this one thing.
Maciej: The same with Google. Here the cost of processing one page matters, and I think they pay attention to it. That is why it is important for them to first check whether the page is really about what they are looking for, for example through the title or other initial selection methods, and then possibly download the page. Because you know, a billion downloaded pages a day is a very large cost for everyone. They optimize it very hard. So your page has to be above all fast, and this information has to be available on that page in a very simple way for those robots. That is why SEO yes, but not only.
Rafał: Exactly, and this brings us a bit to the place where you said that pages have to be fast, efficient, accessible, have structured data, ideally in JSON-LD format. We said something about that Markdown, although I also read that Google released that it would be nice to have Markdown, and then released that no, because after all...
Maciej: But that is Google, you know. I have not seen Markdown hurt, but in the studies I did and showed at the SEO Festival, in OpenAI there was no correlation at all with having structured data.
Rafał: In OpenAI itself, we are talking. Right, we have to clarify that.
Maciej: Do not think it is the same in other models. We are talking exclusively about OpenAI. So there you could have structured data sprinkled all over the page, and other pages were cited anyway.
Which pages do AI models cite: Reddit, YouTube and transcripts?
Rafał: Okay, because a bit of this shows there are a lot of these things. You said it is best to be everywhere, but if you were to answer the question: which types of pages does AI actually most readily cite? We talked about Wikipedia, that it is nice to be in Wikipedia. We talked a bit about brand mentions, that it is good to have media, but now which media, how many, in what form.
Maciej: I will summarize it all for you in a moment, very cool. Everyone is on Reddit.
Rafał: I know you have research on all these things. That is exactly why I am heading there. Because that Reddit shows up somewhere, other forums and so on. Very funny, I started noticing that in the States very often now, to some large entities they add some forums and typically user-generated content. What do we do here?
Maciej: Okay, I will tell you right now. Let us start with ChatGPT, because every platform, we studied it, has its own ecosystem of citations and recommendations. It is not true that you do actions for ChatGPT and you will be visible in Gemini and further on. Unfortunately no, I also showed that in my presentation. You mentioned UGC, user-generated content. That is probably the most important source for ChatGPT itself. Reddit reigns, YouTube reigns, those forums. ChatGPT really pays a lot of attention to that, and if you want to be present in ChatGPT, you also have to provide that layer. So how people write about a brand. I am not urging anyone to buy...
Rafał: ...on Reddit, because there are probably already offers of warmed-up accounts and so on. Just in case, I have a promotion on that, so get in touch.
Maciej: But it is very visible in all our studies. Really, Reddit reigns and forums reign. I also thought it is worth having your own YouTube channel, for example. Podcasts are very important. A transcript is very important, because it is the transcript, not the video, that gets into these models. So if you record a video, make sure you publish the entire transcript of that conversation. And that can really matter, because as I said, in different intents and in different contexts how your brand is perceived, and not just that you commission a hundred publications saying that your brand is great and that is it. This is very important in ChatGPT.
Rafał: Okay, so do it very strategically, and not mindlessly spend, for example, a hundred thousand on a hundred publications. I fully agree, but you touched on one more thing by mentioning YouTube. YouTube and transcripts are something one of the first GPTs was fed with, which ran out of data. They scraped the whole of YouTube from A to Z. That is why at some point it was slightly fascist and they had to put very strong handcuffs on it. I read a great book on this, simply very impressed, one, by the scale, but two, by how safety matters were loosened there. But that is a topic for a completely different conversation.
Maciej: So what, a podcast?
Rafał: Yes. Or we will talk for seven hours and please listen to it and do not interrupt. I am joking. But the point is that you mentioned multimodality, right? Basically. Because you surely remember when Google introduced this MUM, 2018, 2019, everyone said: oh dear, the MUM algorithm. Exactly. And today we all talk about this quite officially, that content has many modalities and text alone will not do the job for you. And now if you were the owner of a company, a store. I know it is a hard question. Let us say hypothetically that you had to make the decision how you divide this, how you arrange it, what would you do with it?
Maciej: I will explain right now. It is not a hard question, because we do it. I will not talk product-wise here today, I will just tell you about the ways in which we do it. I study the competition of a given client, for example, and I look at where above all my client does not show up, where they are not cited, in which intents they are not cited, in which categories, on which topics, at which stage of the sales funnel. And thanks to that, reversing again, because I like going back, reverse engineering, we can stop, damn it, firing blanks, and aim exactly where we want to be, and exactly whether you should publish a guide, or mention your brand more in definitional terms. What is more, by tracking the competition, we have the places where they are cited. We connected it with the database of one of the intermediaries. I will not name it. And we know where you can buy such a publication. These are real actions that I would recommend to everyone who wants to increase their visibility in LLMs.
Rafał: That is a subtle but very good tip. And does the industry matter?
Maciej: The industry matters in general. That was also in my presentation. There are industries that are cemented, meaning electronics. It is very hard to enter with a new brand there, because, if I remember this statistic correctly, twelve brands account for eighty percent of answers. It is cemented, so if you wanted to come in with a new name, you produce electronic equipment, to be in LLM answers, it will be very hard. Somewhere in fashion, well, for example eighty percent of answers contained seven hundred brands, those that we monitored, and that only shows the scale, what size it is. And those brands in fashion rotate non-stop, because we have not yet talked about how much the places change.
How much do answers change and how do you measure it?
Rafał: Let us talk about how that happens.
Maciej: Remember, we still have to come back from this branch, because we are in ChatGPT, and we still have to talk about ecosystems, Gemini and so on. But we will remember. Okay, I studied it, I call it a short-term study, I did it, I put in the same prompts, I think it was a hundred prompts in different categories and so on. I put them in every half hour and collected the answers, and compared simply how the brand citations change and so on. Well, the first place of a brand held in thirty percent of answers.
Rafał: That is frighteningly good.
Maciej: That is very bad. Generally you have first place in an answer, but really over a month you were mentioned once, and later that first place changed to another brand. Top 3 holds the longest, but that middle, meaning let us say top 6, from top 3 to top 7, is replaced completely with every answer. And the same with the tail, there it is a lottery altogether. And interestingly, your brand can drop from first place completely out of that answer, or from outside presence altogether fly into first place. We have statistics on that. So see how much we stretched the same prompts, every half hour, how it changes. So I am giving you something to think about. That is the crux of all this, what we started with, that this is not the new SEO, that everything changes here.
Maciej: And I will go straight on with the statistics: we did a long-term study, meaning a month, week by week, and there first place holds in fifty percent. A bit more already, because it is not determined either. That study shows only that we can really take a report and swap it and say that it was today, yesterday and so on.
Rafał: It is a kind of snapshot in time, right?
Maciej: Yes, but to really monitor your presence, you have to do it long-term. So watch not even the position but the change of that position, when you were there, when you were not, in what context and so on. When we started doing it long-term, started making that kind of historical data, such charts, it turned out that it stays more or less in the same place. But if we wanted to query it once, we can be in first place or not be there at all. And that is the most important thing. It is a trap, it is really even a bit of a fraud, if you make a report once. You would have to make four such reports a month and only then draw conclusions. And best of all every day. I do not know if that is needed, you know? Because look, with the short-term one it changes anyway. So if I do four, or I do not know, seven studies now in an hour, and I do it daily, I suspect it will be the same lottery.
How much does monitoring cost and which prompts should you study?
Rafał: Gosh, that is also a bit strange, in that these studies are expensive. I mean when you look at those companies that offer such a service, then de facto, to actually study, let us say, a hundred prompts, which seems to me is not much at all for a medium-sized company, because people ask...
Maciej: We study in the thousands for one brand, right? Tens of thousands of new ones.
Rafał: Exactly, and I have seen prices of 75 to 1,500 euros for a hundred, two hundred prompts studied once, twice, three times.
Maciej: You know what, for example we have it at the price of 0.02 dollars per answer. Those are not such prices. I know that these are also big applications, I know those prices and so on. It is a terribly large markup, because collecting this answer in this whole machine ecosystem is not that expensive. I think it is fleecing people. I would like to be perceived as the one who tells the truth, because nobody wants to tell the truth. No, I am joking. There are a few people who say it loudly. I know that it is probably because it is a whole application. Sure. We do not have to nitpick that. Although it does not have to be expensive at all.
Maciej: And the more prompts you monitor, the better, but I would, following Michał Suski and his studies, monitor prompts in different intents. If you monitor a hundred prompts in the same intent, there is no such need. Because Michał did a study on that and that really visibility can change only between intents. So I would reduce the number of prompts per intent and monitor different intents. And not such SEO-type intents, because that in my opinion... God, I forgot the name of the author of the book. I will come back to you, or maybe send it to Rafał later.
Rafał: Cool, I will give the link or whatever.
Maciej: And in that book there were nice intents, because there were those intents that determine how an LLM answer is built. So there is definition, there is ranking, there is comparison, recommendation. And those intents are very important, because they are strictly correlated with the type of answer. So you have to be in rankings, you have to be compared, you have to be in definitions, you have to be recommended. UGC comes in, and presence in all those intents in different contexts only then makes sense, the same with monitoring.
How do the Perplexity, Gemini and Copilot ecosystems differ?
Rafał: Okay, let us put a full stop here and come back, because we talked about ChatGPT, but you wanted to say a few words about other platforms. In Poland I do not know what is the second most popular, really. I assume Gemini.
Maciej: Perplexity in my opinion could be. Perplexity is more of an AI search engine than a model itself, but it is very e-commerce. E-commerce brands dominate there and they have that very developed. So if you wanted to search for something to buy, Perplexity has e-commerce very strongly developed. The question is how many users there are and whether for store owners it would not be a good direction. And it is also a completely separate ecosystem. There are many stores in the answers there too.
Rafał: There is the Comet brand, which is connected with it, and I use it, because there are many models there. It works for me very well. Because there is Claude there and so on. So I have everything in one, and completely separately I use Chinese models. I generally prefer Chinese ones for many repetitive things, for production and so on, because it is cheaper and often better.
Maciej: And comparable. For content I have lately been using Chinese ones.
Rafał: Yes, yes. I am a big fan of DeepSeek.
Maciej: Me too. It actually won in my content generation benchmarks. And not only because it is the cheapest, it was simply the best. So you know, it is ten times cheaper and the quality is the same.
Rafał: Exactly. We could talk about all these models here, how Qwen and GLM are catching up and so on, because a lot is really happening, but let us leave that for another meeting too. Good. You were talking about Perplexity for e-commerce. I confirm in this respect that Perplexity has a nicely connected browser and it is an interesting solution for someone who does not want to buy chat, Claude and so on separately, because you can have access to several models at once there.
Maciej: Right, and there is also that shopping. For a while it did not work, I have not checked it now, I did not prepare for this conversation, but shopping was very strongly developed. Shopping, meaning the snippet for purchase where you open, a sidebar with products comes in, you can quickly see the price and so on.
Maciej: Okay, Gemini, canned food. Gemini is canned food. I call it canned food because in Gemini you cannot really force web search. They have their own name for it. The model itself decides whether to search the web, but in most answers it does not search. That is why it is conservative, because it answers only from its own data. So there, for example, citations of government pages, EDU dominate. Or if it is, for example, health, then only health pages verified by it dominate. It is rather hard to reach those citations. So it is canned food, here you can get in through the training data. Or there is probably some retrieval, but it is very rare and very small. So here I would not insist on Gemini at all. And that is it.
Maciej: And there is Copilot, which is probably the weakest of these models, few citations, although you can trigger that too. And each of them, look, each of these models has its own ecosystem. I call it an ecosystem of recommendations and citations. So if you wanted to be in each, there is no one good strategy. You would have to split it up, like with categories.
Where does ChatGPT get its results: Bing, Google or elsewhere?
Rafał: The truth is that, at least I think so, where you really have to be today, you have to be in Google. To a large extent you have to be in Bing and in ChatGPT. The rest... Especially in Poland, because I think if we looked at it geographically, how it is laid out, probably a bit differently. And in the context of Gemini, I heard such stories which, as you say, are untrue. Confirm also that it has direct access to the Google index and pulls information from the index.
Maciej: It has to. Probably yes, but you know, I tested these answers, and these answers do not suggest it. Rather I see that the training data... For me Bing is still a closed ecosystem. It is hard to say anything, but it has to have access. I would not be surprised if they had direct access to saved data, cached pages and so on.
Rafał: Okay, we are talking about Bing, because I use Bing, as I mentioned, so I think it is there to some extent for sure, because I meant Gemini, that it has access to the index. And Bing? The same question basically, but here I am convinced, because even when you search, I think you can see it exactly, although I have no statistical data as a user.
Maciej: What we talked about at the very beginning: for quite a long time ChatGPT had access to Bing, it searched only there, and very many people started indexing their content in Bing, suddenly started creating content for Bing, there is that IndexNow that lets you index quickly. I started doing the same, and at some point it does not enter the index, even though it uses IndexNow, even though the content is well written. They restricted it because the whole world threw itself at it. But de facto at the end, now when I tested it, as I already said, they stopped using that Bing and I do not know what they use. They have their own search mechanism based probably a bit on Google. We also talked with colleagues and suddenly it turned out that many pages that are not visible in Bing are cited in ChatGPT and searched. So there is still a zone of knowledge that we do not know, and can only guess at. Again a bit of guesswork, like with Google. A bit of a game of cat and mouse.
Rafał: The deeper we get into this conversation, the more I have the impression that we do not know what is going on here. And if anyone knows, it is probably you and me. No, I am joking. It is a very interesting world, but indeed there are a lot of unknowns.
What to do on Monday morning?
Rafał: I wanted to slowly head toward the end, so that we also stop at the hour. One thing that a listener should do on Monday morning after this episode. Say it is a listener who has an online store, has an agency that does something, has some dashboard, some KPIs. What in your opinion should they do? Because they want to be in that product discovery, brand discovery, let us call it nicely in Old Polish, in language models. What would you recommend?
Maciej: Right, I would ask from the start for a long-term analysis of the answers, meaning study it week after week, at least two, three times, and only on that basis draw conclusions from the analysis. They can measure the places they occupy, but it has to be an averaged position from all those studies. But that averaged position also says little. The most important thing they should ask for is information about in which places they are, and in which thematic, intent-related context they are invisible. And to ask for the campaign actions to be carried out solely and exclusively for that, so that it is not like I said, that we mention ourselves blindly, but we aim exactly at those places where we are not. And such an agency should even run a campaign for them and may collect in a very easy way information about the pages that are cited in the answers where they are not.
Rafał: That is very good advice. You see, there is a consistency here, because I also like strategy first, before you start spending money. It all fits together well. It is a good conversation.
What will change in a year: protocols and agentic commerce?
Rafał: Maciek, a year ahead. Not further, because a year these days is, oh dear, a few decades not long ago. What will change most in this area? Will models change, user behavior, measurement tools? What do you think changes? I know you do not have a crystal ball, but you have a lot of experience, so on that basis.
Maciej: In my opinion, as we talked about, the share of searches through LLMs will grow, mainly through ChatGPT at the moment, and that will have quite a big impact. The area of internet infrastructure will change in my opinion. As I said, people will also start paying attention to making pages accessible to language models, and they will modify them. It will go further and further. And at some point the website will stop being needed visually, and more structurally. I think that will happen. Look at what a craze there is for llms.txt, even though it is a myth or not a myth. Nobody knows. It seems to me that within this year there will finally be some unified protocol. What should the background of this page look like? That is, to be structurally unified for such models, and if not, then a simple Markdown with this information, and it will gain more importance. And certainly shopping will be bigger and bigger, those purchases through language models.
Rafał: And are you talking about shopping in the context of agentic commerce, meaning agent-based purchasing?
Maciej: Yes, I probably have not had time to get acquainted with it either. Something is always coming out and we cannot imagine it, but from what I read, Google has already released its protocol for it.
Rafał: Yes, the Universal Commerce Protocol. And they worked there also with large marketplaces like Zalando, Shopify, Etsy took part. So they are certainly preparing something. I mean it is released, you can generally test it, but what I mean more is whether you seriously think that users will attach a card, a payment option, to, let us say, their assistant. Let it be this ChatGPT that will choose things for them and have some memory. Do you think that is realistic? I do not know, it seems to me that it is also the mentality of a given country: in Poland probably less, in the Czech Republic least, because we have such high bars when it comes to trust, and also generationally. I cannot look at it, because we are, as you said, already dinosaurs. I do not know how the new generation will react.
Maciej: I have two children, who react to this time very well. It is the generation of children.
Rafał: My daughter actually goes to school, fifth grade. So practically from the moment of commercial use of LLMs, so they will not remember another world either.
Maciej: And already now both use ChatGPT or other models very heavily, because of course dad forces some other things, but they handle it great, so maybe for them the entry threshold, that kind of switching over, can be painless, much easier than for us. That is what I still think about it.
Rafał: I am basically convinced you are right. I asked this question in a tricky way, because I am on the side that this change will happen faster than it seems to us. Very often it is like that: we look at something from the perspective of our experience, our own habits, and it seems to us that this wall will surely not fall. A few years pass and there is no trace of that wall, and so it was with many things, and it will be the same with this.
Maciej: Recently, months even, not years, you know, so it would be three times faster.
Rafał: You said yourself that it simply accelerates. Yes, according to those laws of acceleration it will probably be even faster. We have to finish, although I could talk with you, like on Huberman, for another four hours, and it would be pleasant and probably interesting, but maybe we leave that for another occasion.
Recommendations and closing
Rafał: As standard at the end of every episode I have this request, that if you have some recommended material, a recommendation, whether it is a book, a blog, a YouTube channel, or maybe it is just some theater in Olsztyn and you like something there, whatever it is that is interesting, that inspires you, that intrigues you, that you say it and invite our listeners.
Maciej: Then I will surprise you a bit. In mid-August the most beautiful festival in Poland takes place in Olsztyn. It is the Olsztyn Green Festival, I think already the eleventh edition. Only Polish bands. It lasts three days, it is on the shore of a lake. Against the setting sun various groups play. It is very nice, it is very big, there are three stages. You can sit on a deck chair and listen to music. You can paddle up in a boat and listen to the music. It is beautiful, colorful, in mid-August. So far the weather has held up. An amazing place, amazing people and a lot of music. I invite you to Olsztyn.
Rafał: Great, you have old tickets. Of course the three-day ones are best.
Maciej: I am deep in LLMs, so from these recommendations maybe I would like to step out of that, maybe we stay with the music one. That book I mentioned, we would also recommend it, we will definitely send it to you, because it changed a lot. His name is Sergey Lucktinov. "Semantic SEO, SRO & AI".
Rafał: Okay, I will find it and link it. By the way, since you mentioned it, I recommend a great book, "Superintelligence". I will link it too. It is a book that really grounds what superintelligence might look like. I do not know if it has been published in Polish, honestly, but if I find it, I will link it in one language or another, because I think it is super interesting, and we also have LLMs, so if we buy a book in English...
Maciej: You can also summarize it quickly.
Rafał: We can translate it, only that is illegal. Of course I do not encourage that.
Maciej: Right.
Rafał: But if someone wanted to break the law, there is no problem. And if someone wanted to find you, Maciek, and did not feel like going to that festival, which they will regret for the rest of their life, if they wanted to find you on the internet, read, or just ask you a question, where is the best place to do it?
Maciej: Listen, there is no problem, write to me on LinkedIn. I am open, you are welcome if you want to talk. In less than a week there is one of the SEO events where I will probably meet some people. And in October, it is not announced yet, but at I Love Marketing, on the GEO track, there are only six speakers, forty minutes each, in Poznań.
Rafał: Oh, great, I did not even know.
Maciej: And there I will also have a presentation exactly related to what we talked about today. So you are welcome there too, and along the way probably at some smaller SEO events.
Rafał: Great, we will probably see each other in Olsztyn at the festival. You have encouraged me so much that I probably will not skip it.
Maciej: Just check accommodation, because there was a year when there was not a single bed left. Seventy percent of people come to this festival from outside Olsztyn.
Rafał: Okay, I will be here in the trunk of your car, just so.
Maciej: Oh, then you will stay at my place. You are welcome.
Rafał: Oh you see, I specifically wanted to say that about the car so that you would offer. It is a game. Great, they fall for it. Okay, you see, one against the other. All the best, thank you very much for your time and for all this information. It was really refreshing, I like the way you think. It is such constructive criticism of the status quo, and I think that such people are needed, especially if they rely on data and not only on intuition. Thank you again very much.
Maciej: Thank you too.
Rafał: Until next time. Bye!
Closing of the episode
Rafał: And that is all for today's Globalne Horyzonty. I really care about your opinion, so let me know in a comment which thread from today's conversation was the most interesting to you. Your activity is the simplest and best way to appease the algorithms and help me develop this project. Do you want to talk about this episode more broadly? Then add me to your network on LinkedIn or write to me directly by email: [email protected]. I love getting feedback and information from you. For now, thanks for today and see you soon!
Listen and watch
The same conversation in your favourite app.
The full edited transcript as a Markdown file, with the guest’s details and quotes.
Back to top



