Web Scraping Without Getting Blocked | ScrapeOps (2023)

The goal of every web scraper is to not standout, instead doing everything you can to blend into a websites normal traffic.

However, not being flagged as scraper is getting harder and harder as anti-bot technologies get ever more sophisticated and more widely used.

Today, you can be detected by:

  • IP Address
  • TLS or TCP/IP fingerprint
  • HTTP headers (values, order and cases used)
  • Browser fingerprints
  • Cookies/Sessions

In this guide we're going to share with you some of the common ways detect you as a scraper, and how you optimise your scrapers so that you can blend into a websites normal traffic and not get blocked.

  • Header Optimisation
  • Browser Fingerprinting
  • TLS Fingerprinting
  • Request Profiling

Need help scraping the web?

Then check out ScrapeOps, the complete toolkit for web scraping.

Web Scraping Without Getting Blocked | ScrapeOps (1)

Proxy Manager

Web Scraping Without Getting Blocked | ScrapeOps (2)

(Video) How to Scrape Websites Without Getting Blacklisted or Blocked

Scraper Monitoring

Header Optimization

The first step every developer should take when making their scrapers production ready so they don't get blocked is optimising the headers they use with their requests.

The headers you use is one of the easiest ways for a website to detect that you are a scraper and not a real user.

You need to make sure the headers you use are like the headers a real web browser would send, and that they are consistent with who you are trying to pretend to be.

In our header optimisation guide, we go through in detail how you should optimise your headers when scraping, however, here are the main points:

1. Use Real Web Browser Headers

By default, most HTTP libraries (Python Requests, Scrapy, NodeJs Axios, etc.) either don't attach real browser headers to your requests or include headers that identify the library that is being used. Both of which immediately tell the website you are trying to scrape that you are scraper, not a real user.

So with every request you should use a real set of headers, and vary them with each request. For example, here are example headers for using Chrome on a MacOS machine:

Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8

(Video) 5 Ways to Scrape Websites Without Getting Blocked

2. Pay Attention To Header Order

Most web browsers attach headers in a certain order that doesn't change, however, a lot of HTTP clients either use their own header ordering or randomise their order making the identification of web scrapers very easy.

Some HTTP clients, like the popular Python Requests library does not respect the header order you define in your request (see issue 5814). Making it easier for websites to detect requests using unnatural headers.

To combat this you should use HTTP clients that respect the header orders you define, so you can match them exactly to how a browser would send them. In the case of Python, the httpx library does respect header orders so it is a good alternative to Python requests.

3. Optimise Headers For Specific Websites

Depending on what website you are scraping, using specific header combinations may increase your scraping performance.

Sometimes websites require you to include specific headers when accessing lower level pages on the site, or certain header combinations may increase your success rates (for example, setting the referer header as facebook.com versus google.com).

A big part of the reason why proxy services like ScraperAPI, Scrapingbee, or ScrapingAnt can get much higher performances out of data center proxies than you can, is because they are much better at managing headers and have systems in place to constantly split test various header combinations to maximise performance.


  • Use real browser header and user-agent combinations with every request.
  • Make sure you use a HTTP client that respects header ordering so you can make the header order seem natural.
  • Optimise your headers for specific websites.

IP Addresses & Proxies

If you want to scrape a website quickly or at scale (more than a couple thousand requests per day) then next bottleneck you will run into is websites determining you are a scraper from the IP address you are using.

A real human user will rarely request more than 5 pages per second from the same website.

However, a scraper making concurrent request certainly can do this and it is pretty obvious to the website that the requests aren't coming from a real user.

This means that if you want to disquise your scraper then you will have to start using proxies.

You have numerous options when it comes to picking a proxy solution for your scrapers, from using:

  • Free proxy lists
  • Data center proxies
  • Residential or mobile proxies
  • Proxy APIs
  • Building your own proxy network

All of which have their own pros and cons, from better performance to lower/higher costs, however, you will likely need a proxy solution at some point if you want to prevent your scrapers from getting blocked.

(Video) The Various Ways of Getting Blocked in Web Scraping | OxyCast

Depending on which option you go with, you will also need to build the system to manage your proxy infrastructure. Including proxy selection, rotation, blacklisting, unblocking, etc.

Picking the best proxy provider for your particular use case is quite a big topic, so if you want more info then check out our proxy guides and tools:

  • Proxy Pool Optimization Guide
  • How To Pick The Best Proxy Solution For Your Use Case
  • Proxy Provider Comparison Tool


Check out of Proxy Pool Optimization Guide for more information, but here are some of the key things to remember.

  • If scraping protected websites, or scraping at scale then it is very likely you will need to use proxies to disquise your scraper.
  • The type of proxy you use (datacenter, ISP, residential, mobile, etc.) can have a big affect on your performance.
  • You need a diversified pool of proxies (proxies from different subnets, not just different IP addresses).
  • You need to put in place a system to manage your proxies, otherwise you will get poor performance and utlimately burn out the proxy pool.

Request Profiling

A common mistake we see developers make, is making requests that are very obvious to the target website that they are a web scraper and not a real user.

Sometimes two users can be scraping the same website, and use the exact same header/IP systems and one is getting consistently blocked and the other is scraping without any issues solely because of how they structure their requests.

One users requests are believable as real users, whereas the others is obviously a scraper.

Here are some of the common mistakes that can quickly give you away as a scraper:

1. Unrealistic URLs

The URL you use to make the request can often give you away as a scraper. The question you should be asking yourself is, are you sending requests to URLs that a real user would never use?

A common example of this is when scraping e-commerce sites. To keep their scrapers as simple as possible, a lot of developers will design their scrapers to request URLs that use a products ASIN number. For example: https://www.shop.com/product/[product_id]

These URLs can work, however, if you are browsing the website as a real user you will rarely see the website format the URL like this. Making it much easier for the website to detect you as a scraper.

Instead, you should make your URLs look like the URL a real user would request. For example: https://www.walmart.com/ip/Surface-Bassu-Moisture-Conditioner-2-oz-Pack-of-2/643905888 will have much higher success rate than https://www.walmart.com/ip/643905888 both of which are URLs for the same product.

2. Request Patterns

The pattern you make the requests can also be a easy give away that you are a scraper and not a real user.

For example, if I want to scrape all the products in a category from a e-commerce store, and I started with page 1 and product 1, and scraped the entire category in sequential order and at a constant rate then it is highly likely the requests are automated.

(Video) The Biggest Mistake Beginners Make When Web Scraping

Instead, of scraping every page and product in order, you should randomly scrape different pages/products and vary the interval between your requests (from seconds to minutes).

3. Location

A big giveaway for websites that have a very specific geographic focus is making requests from a very weird location that normal users would never use.

If you are scraping a South American real estate platform, but you are making all your requests using Russion proxies then it will show up very quickly in their website analytics that there is suspisous traffic coming from Russia. So they might decide to show more CAPTCHAs to Russion traffic in future or block it completely.


  • Use believeable URLs that normal users will request when scraping a website, not the shortest/simplest to code into your scrapers.
  • Randomise how you scrape a site and the intervals between your requests to make it less obvious the requests are automated.
  • Make requests from locations that the websites real users actually live, not a country on the other side of the world.

Browser Fingerprinting

Increasingly, to combat websites using antibot technologies, lots of developers are turning to using headless browsers like Puppeteer, Playwright, or Selenium to avoid getting blocked when scraping a website.

Using a headless browser does make your requests seem more like a real user than using a HTTP client, however, they aren't a magic bullet and open up a pandoras box of ways for websites to test if you are a scraper or a real user.

Modern antibot technologies use browser fingerprinting, can detect browser automation leaks and integrate honeytraps and other challenges into the page that your scrapers could fail.

Here are some of the major issues:

1. Fixing Browser Leaks

Browsers provide information about themselves in the Javascript execution context, which the client (i.e. website) can access to verify that the browser is in fact a real user and not a bot.

By default, most headless browsers leak information that tells the website that the browser is automated and not a real user. To avoid being blocked you need to patch these leaks and fortify your browser fingerprint so that your scraper isn't detected.

In our guide to fortifying your headless browser we go through in detail what some of these fingerprint leaks are, and how to patch them.

However, when you are using a headless browser for web scraping you should always use the stealth versions, as they often have the most common leaks fixed.

2. Consistent Identity

A common issue that many developers overlook is making sure the identity you are displaying in your headers, user-agent, browser, server and proxy are all consistent and match each other.

For example, if you are using a headless browser you need to make sure the user-agent string you defined matches the browser version you are using. Or if you are running your scraper on a Linux machine, but your user-agent says it is a Windows machine.

(Video) How to avoid getting blocked while web scraping ?

Inconsistencies like this won't happen for real users visiting a website, so if you don't make sure you use a consistent identity with every request then you are likely to get blocked.


When you start using headless browsers for web scraping you can get a performance boost, however, websites can still detect you. And the number of ways they can detect your is truely massive. Checkout our guide to fortifying your headless browser which goes into much more detail on the topic.

  • Always use the stealth version of your automation library, be it puppeteer-stealth, playwright-stealth or selenium-stealth.
  • Make sure the identity you present with the request is consistent amongst the headers, user-agent, browser, server and proxies you use.


How do you not get blocked while scraping? ›

10 Tips For Web Scraping Without Getting Blocked/Blacklisted
  1. IP Rotation. ...
  2. Set a Real User Agent. ...
  3. Set Other Request Headers. ...
  4. Set Random Intervals In Between Your Requests. ...
  5. Set a Referrer. ...
  6. Use a Headless Browser. ...
  7. Avoid Honeypot Traps. ...
  8. Detect Website Changes.

Can you get blocked for web scraping? ›

Web pages detect web crawlers and web scraping tools by checking their IP addresses, user agents, browser parameters, and general behavior. If the website finds it suspicious, you receive CAPTCHAs and then eventually your requests get blocked since your crawler is detected.

How do I hide my IP when scraping a website? ›

Use IP Rotation

To avoid that, use proxy servers or a virtual private network to send your requests through a series of different IP addresses. Your real IP will be hidden. Accordingly, you will be able to scrape most of the sites without an issue.

Should I use a VPN when web scraping? ›

Where proxies provide a layer of protection by masking the IP address of your web scraper, a VPN also masks the data that flows between your scraper and the target site through an encrypted tunnel. This will make the content that you are scraping invisible to ISPs and anyone else with access to your network.

How do I bypass blocked sites on Chrome? ›

To unblock a site on Chrome, assuming you have administrator privileges:
  1. Open the Chrome app.
  2. Select More on the right (the three vertical dots).
  3. Click on Settings.
  4. Go to Privacy and Security.
  5. Select Site Settings.
  6. Unblock your desired website.

How do I bypass blocked Chrome? ›

How to Unblock Websites on Chrome (5 Working Tips)
  1. Bypass Chrome Warning.
  2. Disable Windows Firewall.
  3. Remove sites from the Restricted List.
  4. Make Changes to the Host File.
  5. Use a VPN Service.
  6. Bottom Line.
  7. FAQs.
May 13, 2023

Does Google ban scraping? ›

IP blocks. When you scrape data with a bot, the Google site will block your IP address from any further scraping. This is because when you send multiple requests from the same IP address, the target website will recognize your activity and ban you.

Does Google search allow web scraping? ›

Can you scrape Google search results? Yes. You can scrape Google SERP by using Google Search Scraper tool.

Does a VPN hide your IP from websites? ›

VPNs can hide your search history and other browsing activity, like search terms, links clicked, and websites visited, as well as masking your IP address.

Does VPN hide source IP? ›

A VPN hides your IP address and encrypts your online activity for maximum privacy and security. It does this by connecting you to an encrypted, private VPN server, instead of the ones owned by your ISP. This means your activity can't be tracked, stored, or mishandled by third-parties.

Will private browsing hide my IP? ›

But Incognito mode doesn't hide your info from websites, advertisers, your Internet service provider (ISP), or Big Tech companies. Even in Incognito mode, Google and others can still track you. Incognito does not hide your IP address.

Is it ethical to web scrape a website? ›

Ethics of Web Scraping

All your data scraping efforts must be ethical. Here are few approaches to ensure the Web Scraping process is completely transparent and ethical: Use a Public API when available and avoid scraping all together if the data you're looking for is available through the API.

What happens if you click an IP grabber with a VPN? ›

The IP address never remains the same once you connect to a VPN server. Your external (public IP) is now that of the server and all sites you intend to visit will see the IP of the VPN server instead of your own connection.

Is proxy or VPN better for scraping? ›

Cost: Proxies can be free or low-cost, while VPNs can be a bit more expensive. This makes proxies a better option for tasks like web scraping where you might want to source thousands or millions of different IPs for making automated requests.

How to bypass blocked website without VPN? ›

Method 1: Use a proxy
  1. Method 1: Use a proxy.
  2. Method 2: Use the Google cache.
  3. Method 3: Try a URL shortener.
  4. Method 4: Try the IP address.
  5. Method 5: Unblock websites in Chrome and Safari.
  6. Method 6: Switch between HTTP and HTTPS.
  7. Method 7: Use Tor Browser.
  8. Method 8: Use a VPN.
May 26, 2022

Which browser can open all blocked sites? ›

The Tor browser is a free web browser that is used to keep you anonymous on the web by routing your web traffic through a series of proxy servers. Tor is often used to access websites that are blocked by the country or region you live in. You can install it on your computer directly or use it as a portable browser.

Why does Google Chrome keep blocking websites? ›

If Google Chrome blocks a site automatically, it may be because Google deems that site unsafe, or because your employer or school has chosen to prevent access to that site, so you should proceed with caution.

How do I bypass blocked websites at school? ›

Use a VPN: The most effective way to unblock websites at school. Use a Web Proxy: The fastest way to bypass school restrictions. Use Mobile Data: A free way to access websites that are blocked on school WiFi. Use a URL Shortener: A simple, free tool for beating URL-based website blocks.

Does Bing block scraping? ›

However, you may find it difficult if you are looking to constantly scrape Bing search results to gather data for your research, analysis, or SEO campaigns, as you will get blocked immediately due to their bot detection algorithm.

Does Google block illegal content? ›

We do not allow content that: is illegal, promotes illegal activity or infringes on the legal rights of others.

Is scraping reddit legal? ›

The scraping itself is legal, sure. All Wikipedia text is available under the Creative Commons Attribution-ShareAlike License (CC-BY-SA). So long as any reuse follows the terms of that license, that reuse is also legal.

How do you get around blocks in discord? ›

How To Unblock Discord?
  1. Use VPN. The first and most popular way to unlock Discord is to use a VPN Service. ...
  2. Use Proxy. Web Proxy is one of the simplest ways to access restricted websites. ...
  3. Change DNS. ...
  4. Use Web Version. ...
  5. Access Discord With IP Address. ...
  6. Use Discord On Smartphone. ...
  7. Discord Unblocked App.
Nov 23, 2022

What is IP rotation for scraping? ›

IP Rotation will let your web scraper use a different IP every time it makes a request from a website. This way, even if the website is blocking some of the IPs your web scraping is using, your web scraper will be able to rotate to new IPs and avoid the blocks.

How do you know if you can scrape a website? ›

In order to check whether the website supports web scraping, you should append “/robots. txt” to the end of the URL of the website you are targeting. In such a case, you have to check on that special site dedicated to web scraping. Always be aware of copyright and read up on fair use.

How many requests per second web scraping? ›

A safe figure would be to send 1 request per second when scraping data from websites. How does this affect your overall crawling speed? Assuming you're performing 10 million requests per second each IP with 1000 data center IPs, your procedure might take approximately 3 hours.

How to bypass school block Discord? ›

How To Get Discord to Work at Your School or College
  1. Use a VPN to Access Discord. ...
  2. Try the Browser App. ...
  3. Copy the IP Address (Windows Only) ...
  4. Use a Web Proxy. ...
  5. Install Discord from External Storage. ...
  6. Use Discord App from Another Device. ...
  7. Utilize the Alpha Testing Version of Discord. ...
  8. Use a TOR Mirror.
Apr 2, 2023

Can you get an IP address from Discord? ›

No, Discord does not provide users with the IP address of others due to privacy and security reasons.

How do you get a GREY box around text in Discord? ›

To create boxed text in Discord, you will need to write in code blocks. To do that you will incorporate the backtick key. This is on the same key as the tilde, not a parenthesis. If you are only creating a single lined block, you will need to write your text between two backticks.

How to do IP masking? ›

5 ways to hide your IP address
  1. Use a proxy. Proxy or a proxy server has its own IP address and acts as an intermediary between you and the internet. ...
  2. Use a VPN. VPN stands for Virtual Private Network, and this is the most common way to hide your IP address. ...
  3. Use TOR. ...
  4. Use mobile network. ...
  5. Connect to public Wi-Fi.

What is IP sweeping? ›

An IP Sweep attack occurs when an attacker sends ICMP echo requests (pings) to multiple destination addresses. If a target host replies, the reply reveals the target's IP address to the attacker.

How to manipulate IP? ›

How to change IP on Android
  1. Find settings.
  2. Select Connections, then choose Wi-Fi.
  3. Find your connected network and choose the settings gear to the right of it.
  4. Select IP settings.
  5. Choose Static.
  6. Add the new IP address in the provided box.
  7. Select Save to save your changes.

Can screen scraping be detected? ›

Using fingerprinting to detect web scraping

Application Security Manager (ASM) can identify web scraping attacks on web sites that ASM protects by using information gathered about clients through fingerprinting or persistent identification.

What websites are easy to scrape? ›

Best Websites to Practice Web Scraping
  • Toscrape. Toscrape is a web scraping sandbox, ideal for both beginners and advanced scrapers. ...
  • Scrapethissite. Another great sandbox for learning web scraping, Scrapethissite, strongly resembles Toscrape. ...
  • 3. Yahoo! Finance. ...
  • Wikipedia. ...
  • Reddit.
May 4, 2022


1. How to Scrape Difficult Targets Without Getting Blocked
2. How to avoid getting blocked while web scraping BeautifulSoup Python
(Yes I Am Geeking )
3. Avoid Blocked Requests When Web Scraping
(David Teather Codes)
4. Always Check for the Hidden API when Web Scraping
(John Watson Rooney)
5. Scrape Instagram Data WITHOUT Getting Blocked or Banned
(Stevesie Data)
6. How do YOU Solve this common Web Scraping issue?
(John Watson Rooney)


Top Articles
Latest Posts
Article information

Author: Sen. Ignacio Ratke

Last Updated: 29/07/2023

Views: 6215

Rating: 4.6 / 5 (76 voted)

Reviews: 91% of readers found this page helpful

Author information

Name: Sen. Ignacio Ratke

Birthday: 1999-05-27

Address: Apt. 171 8116 Bailey Via, Roberthaven, GA 58289

Phone: +2585395768220

Job: Lead Liaison

Hobby: Lockpicking, LARPing, Lego building, Lapidary, Macrame, Book restoration, Bodybuilding

Introduction: My name is Sen. Ignacio Ratke, I am a adventurous, zealous, outstanding, agreeable, precious, excited, gifted person who loves writing and wants to share my knowledge and understanding with you.