If you’re working in a company that uses web data, you know for sure that it’s just the first layer of a more complex product. In fact, web-scraped data has traditionally been stored in files or databases (cloud storage buckets, data lakes, or data warehouses) and then analyzed using Business Intelligence (BI) tools, both commercial and proprietary. For example, a team might scrape product prices or customer reviews from the web, save the raw data as CSV/JSON files, load it into an SQL data warehouse, and then use BI platforms like Tableau or Power BI to create dashboards and reports. Modern web scraping solutions even output data in structured formats (CSV, JSON, Excel, etc.) specifically to facilitate this pipeline, making it easy to feed the results into existing analytics systems. This process allows analysts to slice and dice the data for insights, but it requires manual effort to formulate queries or build visualizations. Today, Large Language Models (LLMs) are changing this paradigm. Instead of relying solely on static dashboards or SQL queries, organizations can use AI assistants to derive insights from scraped data via natural language prompts. In other words, rather than a human writing a query or interpreting a chart, an AI assistant can directly answer questions about the data. Think about having your ChatGPT-like interface and write some prompts to get insights, bypassing the dashboard creation. I’ve seen this kind of approach in several products before LLMs came out but with no great success. The evolving AI landscape and the weekly improvements we see in LLMs could change the landscape, shifting data exploration from a BI-centric approach to an AI-centric one. Imagine asking, “Which competitor had the highest price last quarter?” instead of finding the right dashboard or Excel report by yourself to get the response. This approach promises faster, more intuitive access to information for non-technical users without the overhead of building charts or writing code. In the past companies where I worked, I’ve seen too many times a proliferation of different dashboards per group of users (if not per user), sometimes extracting different numbers from each other. “Which competitor had the highest price last quarter?” Of course, a new set of challenges arise: what about hallucinations? If we don’t see the underlying number of an answer, can we be 100% sure that the answer is correct? In this post (and the next one in the Lab series), we'll build an end-to-end project where we’ll start by scraping the articles in this newsletter, putting them in a database suitable for AI use, retrieving this knowledge, and then publishing a web app that can use an enriched version of a GPT model. Improving LLM Knowledge Integrating custom data (like your scraped dataset) into an LLM can be done primarily in two ways: by fine-tuning the model or by using Retrieval-Augmented Generation (RAG), and each of these methods has pros and cons. Let’s see how they differ and which approach could be the best for our use case. fine-tuning Retrieval-Augmented Generation (RAG) Fine-Tuning vs. Retrieval-Augmented Generation Fine-tuning means training the base LLM on additional data so it absorbs new knowledge. Essentially, you take a pre-trained model and continue training it on your domain-specific dataset, adjusting its weights to embed that knowledge. For example, if you scraped a collection of technical articles, you could fine-tune an LLM on those articles. After fine-tuning, the model intrinsically knows the information (to the extent it was represented in the training data). Fine-tuning is often done by providing the model with a large set of question-answer pairs or text passages from your data so it learns to respond with that information when relevant. In effect, the model is augmented from the inside – the knowledge becomes part of its parameters. The next time you query the model, it can draw on this baked-in information without needing external help. Fine-tuning intrinsically knows augmented from the inside Retrieval-Augmented Generation (RAG) takes a different approach: the model remains unchanged, but we give it access to an external knowledge base (usually via a vector search). When a query comes in, the system will retrieve relevant documents from your data and feed those into the model along with the question. The LLM then generates its answer with the help of that supplemental context. For boomers like myself, consider it like inserting a CD in the Neo’s head in Matrix to learn new skills and moves. In our case, the knowledge base could be a collection of scraped web articles stored in a specialized database. RAG is like an open-book exam for the LLM – at query time it looks up the relevant “page” of data and uses it to formulate a response, instead of relying solely on its internal memory. Retrieval-Augmented Generation (RAG) retrieve relevant documents As you can imagine, one key difference is where the additional knowledge stays: with fine-tuning, the knowledge is embedded in the model itself (the model’s weights are updated). With RAG, the knowledge stays external, in a database or index, and is fetched as needed. Fine-tuning is akin to teaching the model new facts permanently, whereas RAG is like equipping the model with a dynamic library it can reference on the fly. in the model itself external The two approaches have different pros and cons: Fine-Tuning: Pros: Once fine-tuned, the model can respond faster and more integratedly to new knowledge. It doesn't need lengthy prompts with documents each time. A well-fine-tuned model typically outperforms the base model on domain-specific questions because it has a deeper understanding of that niche terminology and content. Cons: Fine-tuning can be resource-intensive and time-consuming – you need sufficient training data and computing power (or budget if using a service). It also makes the model static concerning that training snapshot. If your scraped data changes or new information comes in, you’d have to fine-tune again to update the model. There’s also a risk of the model forgetting or overriding some of its original knowledge if not carefully managed. Importantly, fine-tuning means your data becomes part of the model’s parameters, which could be a privacy concern if the model weights are exposed or if using a third-party service to fine-tune (your data is uploaded for training). Last but not least, once the knowledge is embedded in the model, you cannot cite any article used to improve it. Retrieval-Augmented Generation (RAG): Pros: No need to modify the LLM itself – you leave the base model as-is and simply provide relevant context at query time. This makes updating or expanding the knowledge base easy: add or remove documents in your external index, and the model will use the latest data. It’s very flexible and keeps your proprietary data external (which can be more secure). RAG can reduce hallucinations by grounding answers in real sources – essentially, the model has the “receipts” to back up its answer. It also typically requires less upfront work than full fine-tuning; most effort goes into setting up the retrieval system. Cons: RAG introduces more moving parts – you need a system for embedding and indexing documents and retrieving them at runtime. At query time, you pay the cost in latency and token length for feeding documents into the model’s prompt. If the retrieved documents aren’t relevant (due to a bad query or vector mismatch), the answer will suffer. The LLM is also limited by its input size; if the documents plus question exceeds the model’s context window, you might have to truncate or select fewer documents. Additionally, the raw text of documents might influence the model's style, which could lead to less coherent or conversational answers unless you prompt it to refine the wording. Fine-Tuning: Pros: Once fine-tuned, the model can respond faster and more integratedly to new knowledge. It doesn't need lengthy prompts with documents each time. A well-fine-tuned model typically outperforms the base model on domain-specific questions because it has a deeper understanding of that niche terminology and content. Cons: Fine-tuning can be resource-intensive and time-consuming – you need sufficient training data and computing power (or budget if using a service). It also makes the model static concerning that training snapshot. If your scraped data changes or new information comes in, you’d have to fine-tune again to update the model. There’s also a risk of the model forgetting or overriding some of its original knowledge if not carefully managed. Importantly, fine-tuning means your data becomes part of the model’s parameters, which could be a privacy concern if the model weights are exposed or if using a third-party service to fine-tune (your data is uploaded for training). Last but not least, once the knowledge is embedded in the model, you cannot cite any article used to improve it. Fine-Tuning: Fine-Tuning: Pros: Once fine-tuned, the model can respond faster and more integratedly to new knowledge. It doesn't need lengthy prompts with documents each time. A well-fine-tuned model typically outperforms the base model on domain-specific questions because it has a deeper understanding of that niche terminology and content. Cons: Fine-tuning can be resource-intensive and time-consuming – you need sufficient training data and computing power (or budget if using a service). It also makes the model static concerning that training snapshot. If your scraped data changes or new information comes in, you’d have to fine-tune again to update the model. There’s also a risk of the model forgetting or overriding some of its original knowledge if not carefully managed. Importantly, fine-tuning means your data becomes part of the model’s parameters, which could be a privacy concern if the model weights are exposed or if using a third-party service to fine-tune (your data is uploaded for training). Last but not least, once the knowledge is embedded in the model, you cannot cite any article used to improve it. Pros: Once fine-tuned, the model can respond faster and more integratedly to new knowledge. It doesn't need lengthy prompts with documents each time. A well-fine-tuned model typically outperforms the base model on domain-specific questions because it has a deeper understanding of that niche terminology and content. Pros: Cons: Fine-tuning can be resource-intensive and time-consuming – you need sufficient training data and computing power (or budget if using a service). It also makes the model static concerning that training snapshot. If your scraped data changes or new information comes in, you’d have to fine-tune again to update the model. There’s also a risk of the model forgetting or overriding some of its original knowledge if not carefully managed. Importantly, fine-tuning means your data becomes part of the model’s parameters, which could be a privacy concern if the model weights are exposed or if using a third-party service to fine-tune (your data is uploaded for training). Last but not least, once the knowledge is embedded in the model, you cannot cite any article used to improve it. Cons: forgetting Retrieval-Augmented Generation (RAG): Pros: No need to modify the LLM itself – you leave the base model as-is and simply provide relevant context at query time. This makes updating or expanding the knowledge base easy: add or remove documents in your external index, and the model will use the latest data. It’s very flexible and keeps your proprietary data external (which can be more secure). RAG can reduce hallucinations by grounding answers in real sources – essentially, the model has the “receipts” to back up its answer. It also typically requires less upfront work than full fine-tuning; most effort goes into setting up the retrieval system. Cons: RAG introduces more moving parts – you need a system for embedding and indexing documents and retrieving them at runtime. At query time, you pay the cost in latency and token length for feeding documents into the model’s prompt. If the retrieved documents aren’t relevant (due to a bad query or vector mismatch), the answer will suffer. The LLM is also limited by its input size; if the documents plus question exceeds the model’s context window, you might have to truncate or select fewer documents. Additionally, the raw text of documents might influence the model's style, which could lead to less coherent or conversational answers unless you prompt it to refine the wording. Retrieval-Augmented Generation (RAG): Retrieval-Augmented Generation (RAG): Pros: No need to modify the LLM itself – you leave the base model as-is and simply provide relevant context at query time. This makes updating or expanding the knowledge base easy: add or remove documents in your external index, and the model will use the latest data. It’s very flexible and keeps your proprietary data external (which can be more secure). RAG can reduce hallucinations by grounding answers in real sources – essentially, the model has the “receipts” to back up its answer. It also typically requires less upfront work than full fine-tuning; most effort goes into setting up the retrieval system. Cons: RAG introduces more moving parts – you need a system for embedding and indexing documents and retrieving them at runtime. At query time, you pay the cost in latency and token length for feeding documents into the model’s prompt. If the retrieved documents aren’t relevant (due to a bad query or vector mismatch), the answer will suffer. The LLM is also limited by its input size; if the documents plus question exceeds the model’s context window, you might have to truncate or select fewer documents. Additionally, the raw text of documents might influence the model's style, which could lead to less coherent or conversational answers unless you prompt it to refine the wording. Pros: No need to modify the LLM itself – you leave the base model as-is and simply provide relevant context at query time. This makes updating or expanding the knowledge base easy: add or remove documents in your external index, and the model will use the latest data. It’s very flexible and keeps your proprietary data external (which can be more secure). RAG can reduce hallucinations by grounding answers in real sources – essentially, the model has the “receipts” to back up its answer. It also typically requires less upfront work than full fine-tuning; most effort goes into setting up the retrieval system. Pros: keeps your proprietary data external Cons: RAG introduces more moving parts – you need a system for embedding and indexing documents and retrieving them at runtime. At query time, you pay the cost in latency and token length for feeding documents into the model’s prompt. If the retrieved documents aren’t relevant (due to a bad query or vector mismatch), the answer will suffer. The LLM is also limited by its input size; if the documents plus question exceeds the model’s context window, you might have to truncate or select fewer documents. Additionally, the raw text of documents might influence the model's style, which could lead to less coherent or conversational answers unless you prompt it to refine the wording. Cons: In short, fine-tuning bakes in a focused understanding, whereas RAG allows real-time access to knowledge. For our use case of integrating constantly updated scraped data, RAG seems a better approach: you can continuously pull in new web data and have your assistant use it immediately rather than periodically retraining the whole model. Before moving on, it’s worth noting that fine-tuning and RAG aren’t mutually exclusive; they can complement each other. For instance, you might fine-tune a model to adjust its tone or ability to follow instructions (or to add knowledge that is small and static), and still use RAG to give it access to a larger knowledge base that updates frequently. In practice, though, RAG alone often provides a simpler and more scalable path for enabling an AI assistant to handle custom knowledge, which is the path we’ll focus on in our implementation. Using a Local Model vs. an External API Another consideration is what LLM to use for your AI assistant: a local (open-source) model you run yourself or a hosted model via an API (like OpenAI’s GPT-3.5/GPT-4 or others). Both fine-tuning and RAG can be done with either, but there are trade-offs: what LLM to use Local Open-Source LLMs – Models such as LLaMA 2, Mistral, or Falcon can be run on your own servers. The big benefit here is control and privacy. Your scraped data never leaves your environment, which is important if it contains sensitive information. You can fine-tune them freely on your data or modify how they work. Cost-wise, running a local model might be cheaper for large volumes of queries (no API fees), but you have to invest in hardware or cloud infrastructure to host it. The downside is that many open models could not match the performances of the newest GPTs. You might need to use larger or more specialized models to get comparable performance, which can be complex to manage. Additionally, maintaining and updating the model (and possibly the vector database if on-prem) is on you. In case you do have a highly domain-specific dataset and the expertise, a local model can be fine-tuned to excel in that domain, making it a strong solution for a “private GPT.” Local Open-Source LLMs control and privacy do External API LLMs (e.g. OpenAI’s) – Using an API like OpenAI’s GPT-4 means you don’t have to worry about running the model; you just send your prompts to their service and get the completion. This is very convenient and typically gives you access to cutting-edge model quality with no infrastructure hassle. For our scenario, you could use RAG by simply prepending the retrieved documents to your prompt and asking the API to answer. The drawbacks here relate to customization and privacy. Not every model is available for fine-tuning (OpenAI, for example, allows you to do it on GPT4-o and GPT4-mini). You’re also subject to the provider’s rules (for example, they might have content filters that could prevent some scraping-related queries). From a privacy standpoint, you are sending your data (query and retrieved context) to a third party, so I won’t suggest this approach for sensitive or copyrighted data. Another limitation is cost – each API call costs tokens, which can add up when you’re including a lot of context (scraped text) in prompts. External API LLMs (e.g. OpenAI’s) OpenAI, for example, allows you to do it on GPT4-o and GPT4-mini OpenAI, for example, allows you to do it on GPT4-o and GPT4-mini In general, if your use case involves highly sensitive data or requires full control, a local LLM is recommended despite the extra effort. If your priority is the best possible language ability and quick setup, a hosted model like OpenAI’s might be the better choice. In this post’s implementation, we’ll illustrate using OpenAI’s GPT API for simplicity and quality, but the retrieval system we build could just as easily feed into an open-source model like Llama2 via HuggingFace or LangChain libraries. The retrieval mechanism (vector database + similarity search) remains the same; only the final answer-generating model would differ. Considering these decisions, let’s integrate my scraped articles into an AI assistant. We’ll use the RAG approach with an OpenAI model, which aligns well with continuously updated web data and avoids the need for costly fine-tuning jobs. Scraping TWSC with Firecrawl Firecrawl is a web scraping engine exposed as a REST API and SDK. It is specifically designed to turn websites into LLM-ready data (in formats like clean text or markdown), handling all the heavy lifting, such as crawling links, rendering JavaScript, and so on. The great benefit of Firecrawl is that with one API call, you can scrape an entire site. This makes it perfect for quickly grabbing all the content from my newsletter to feed into an AI. Firecrawl Firecrawl Firecrawl LLM-ready data For The Web Scraping Club blog, we’ll use its sitemap to discover all article URLs. (The blog is hosted on Substack, which provides an XML sitemap listing all posts.) Firecrawl could crawl the site without a sitemap, but using it as a starting point can be more efficient and ensure we don't miss any pages. First, we set up Firecrawl by installing its Python SDK and authenticating with an API key (assuming you have signed up and obtained a key): from firecrawl import FirecrawlApp import os os.environ["FIRECRAWL_API_KEY"] = "YOURKEY" # or load from .env app = FirecrawlApp() # Define the sitemap URL (we can loop this for multiple years if needed) map_result = app.map_url('https://substack.thewebscraping.club/sitemap.xml', params={ 'includeSubdomains': True }) print(map_result) for article in map_result['links']: if '/p/' in article: print(article) response = app.scrape_url(url=article, params={'formats': [ 'markdown' ]}) from firecrawl import FirecrawlApp import os os.environ["FIRECRAWL_API_KEY"] = "YOURKEY" # or load from .env app = FirecrawlApp() # Define the sitemap URL (we can loop this for multiple years if needed) map_result = app.map_url('https://substack.thewebscraping.club/sitemap.xml', params={ 'includeSubdomains': True }) print(map_result) for article in map_result['links']: if '/p/' in article: print(article) response = app.scrape_url(url=article, params={'formats': [ 'markdown' ]}) With just a few lines of code, our articles are already in Markdown format. Choosing a Vector Database for RAG A vector database is a key component of RAG implementation. It stores your documents' embeddings (vector representations) and allows fast similarity search to fetch relevant documents for a given query embedding. Many options are available, including open-source libraries and managed cloud services, but for our implementation, we will use Pinecone. vector database Pinecone Pinecone is a fully managed vector database built for fast, scalable similarity search. Unlike open-source databases that require self-hosting, Pinecone is a cloud-native solution, meaning we don’t need to worry about infrastructure management. It automatically handles indexing, search latency optimization, and scaling. Pinecone Pinecone fully managed vector database Setting up Pinecone The first steps are, of course, signing up to Pinecone and getting the API_KEY and environment from the dashboard. Then, we can install the Python SDK as usual pip install pinecone pip install pinecone Finally, then, we can connect to Pinecone in our script from pinecone import Pinecone, ServerlessSpec pc = pinecone.Pinecone( api_key="YOUR API KEY" ) from pinecone import Pinecone, ServerlessSpec pc = pinecone.Pinecone( api_key="YOUR API KEY" ) The environment value comes from the Pinecone web console when you create your API key. Creating a Pinecone Index An index is where your data is getting stored for a later retrivial from the LLM. Instead of being in plain text, it’s in vectorial format (basically a string of number), that allows the LLM to understand which entry in the index is more likely a good response for the query the LLM makes. In a vector database like Pinecone, ChromaDB, or Weaviate, apart from vector data, we have also metadata: metadata is extra information stored alongside each vector (embedding) to make retrieval more meaningful. vector database Pinecone, ChromaDB, or Weaviate ChromaDB ChromaDB Weaviate Weaviate extra information While vectors represent numerical embeddings used for similarity search, metadata provides structured information about what the vector represents. This allows filtering, categorization, and interpretability. While in the vector data we’ll insert the markdown of the articles, in the metadata we’ll use some information we want the LLM to share with us, like the author of the blog used for the answer, the title and the link to the post. vectors numerical embeddings metadata filtering, categorization, and interpretability index_name = "article-index" if not pc.has_index(index_name): index_model = pc.create_index_for_model( name=index_name, cloud="aws", region="us-east-1", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) #pc.describe_index(index_name) #to get the host index=pc.Index(host='YOURINDEXENDPOINT') ..... article_data = [ {"id": f"article-{i}", "text": page["markdown"], "url": page["metadata"]["url"], "title": page["metadata"]["title"], "author": page["metadata"]["author"][0]} for i, page in enumerate(scrape_results['data']) ] #print(article_data) # Generate embeddings using LLaMA v2 for article in article_data: # Estrai il testo dell'articolo text = article["text"] #print(text) # Single article insert to avoid failures embedding = pc.inference.embed( model="llama-text-embed-v2", inputs=[text], parameters={"input_type": "passage"} ) # Prepare data for Pinecone upsert vector_data = { "id": article["id"], # Unique article ID "values": embedding[0]["values"], # Embedding vector "metadata": { "url": article["url"], # Store article URL "content": text[:300], # Store first 500 chars as a preview/snippet "title": article["title"][:100], "author": article["author"][:50] } } #print(vector_data) # Upsert the single article into Pinecone index.upsert(vectors=[vector_data], namespace="articles") print(f"✅ Upserted: {article['id']} ({article['title']})") # Optional: Add a short delay to prevent API rate limits (adjust as needed) time.sleep(1) index_name = "article-index" if not pc.has_index(index_name): index_model = pc.create_index_for_model( name=index_name, cloud="aws", region="us-east-1", embed={ "model":"llama-text-embed-v2", "field_map":{"text": "chunk_text"} } ) #pc.describe_index(index_name) #to get the host index=pc.Index(host='YOURINDEXENDPOINT') ..... article_data = [ {"id": f"article-{i}", "text": page["markdown"], "url": page["metadata"]["url"], "title": page["metadata"]["title"], "author": page["metadata"]["author"][0]} for i, page in enumerate(scrape_results['data']) ] #print(article_data) # Generate embeddings using LLaMA v2 for article in article_data: # Estrai il testo dell'articolo text = article["text"] #print(text) # Single article insert to avoid failures embedding = pc.inference.embed( model="llama-text-embed-v2", inputs=[text], parameters={"input_type": "passage"} ) # Prepare data for Pinecone upsert vector_data = { "id": article["id"], # Unique article ID "values": embedding[0]["values"], # Embedding vector "metadata": { "url": article["url"], # Store article URL "content": text[:300], # Store first 500 chars as a preview/snippet "title": article["title"][:100], "author": article["author"][:50] } } #print(vector_data) # Upsert the single article into Pinecone index.upsert(vectors=[vector_data], namespace="articles") print(f"✅ Upserted: {article['id']} ({article['title']})") # Optional: Add a short delay to prevent API rate limits (adjust as needed) time.sleep(1) As you can see from the code, we basically iterate every article scraped before and add it to a newly created index called article-index. article-index If you want to play more with Pinecone there’s an extensive documentation on their website. an extensive documentation on their website. an extensive documentation on their website. But now I’ve inserted all the articles on the index, can we extract the information we need? I’ve created a basic script called query.py to test the results of the searches on the index. When asked “Can you please list some articles about bypassing Kasada?”, the query returns the following articles: {'matches': [{'id': 'article-0', 'metadata': {'author': 'Pierluigi Vinciguerra', ..., 'title': 'THE LAB #76: Bypassing Kasada With Open ' 'Source Tools In 2025', 'url': 'https://substack.thewebscraping.club/p/bypassing-kasada-2025-open-source'}, 'score': 0.419812053, 'values': []}, {'id': 'article-129', 'metadata': {'author': 'Pierluigi Vinciguerra', ..., 'title': 'How to by-pass Kasada bot mitigation?', 'url': 'https://substack.thewebscraping.club/p/how-to-by-pass-kasada-bot-mitigation'}, 'score': 0.418432325, 'values': []}, {'id': 'article-227', 'metadata': {'author': 'Pierluigi Vinciguerra', ..., 'title': 'Scraping Kasada protected websites', 'url': 'https://substack.thewebscraping.club/p/scraping-kasada-protected-websites'}, 'score': 0.378159761, 'values': []}], 'namespace': 'articles', 'usage': {'read_units': 6}} {'matches': [{'id': 'article-0', 'metadata': {'author': 'Pierluigi Vinciguerra', ..., 'title': 'THE LAB #76: Bypassing Kasada With Open ' 'Source Tools In 2025', 'url': 'https://substack.thewebscraping.club/p/bypassing-kasada-2025-open-source'}, 'score': 0.419812053, 'values': []}, {'id': 'article-129', 'metadata': {'author': 'Pierluigi Vinciguerra', ..., 'title': 'How to by-pass Kasada bot mitigation?', 'url': 'https://substack.thewebscraping.club/p/how-to-by-pass-kasada-bot-mitigation'}, 'score': 0.418432325, 'values': []}, {'id': 'article-227', 'metadata': {'author': 'Pierluigi Vinciguerra', ..., 'title': 'Scraping Kasada protected websites', 'url': 'https://substack.thewebscraping.club/p/scraping-kasada-protected-websites'}, 'score': 0.378159761, 'values': []}], 'namespace': 'articles', 'usage': {'read_units': 6}} Not bad! All three articles were exactly about the topic! For today it’s enough, in the next episode we’ll see how to connect this DB to GPT4 and then creating a simple user interface to write prompts and get the data you need. The article is part of “The Lab” series by Pierluigi Vinciguerra. Check out his Substack page for more knowledge on Web Scraping. The article is part of “The Lab” series by Pierluigi Vinciguerra. Check out his Substack page for more knowledge on Web Scraping. “The Lab” “The Lab” Pierluigi Vinciguerra Substack
