Error: Unable to crawl your landing page on iOS devices. HTTP error 403
Step 1: Go to robots.txt and add the followng to the top of the txt
User-agent: AdsBot-Google
Disallow:
*leave the Disallow: section blank"
Keep checking your logs for 403 errors.
Although robots.txt is the first line of fixing bot related errors, the error could also be from rules set in the Web Application Firewall or the .htaccess file
Step 2: Allowing Google IP's
This is a bit of a challenge as Google updates and maintains a list of their IPs. You have to whitelist them in your firewall or your .htaccess file
Write a program or update the list of allowed IPs by visiting https://www.gstatic.com/ipranges/goog.json
As of 2022, the list includes
8.8.4.0/24
8.8.8.0/24
8.34.208.0/20
8.35.192.0/20
23.236.48.0/20
23.251.128.0/19
34.3.3.0/24
34.4.4.0/24
34.64.0.0/10
34.128.0.0/10
35.184.0.0/13
35.192.0.0/14
35.196.0.0/15
35.198.0.0/16
35.199.0.0/17
35.199.128.0/18
35.200.0.0/13
35.208.0.0/12
35.224.0.0/12
35.240.0.0/13
64.15.112.0/20
64.233.160.0/19
66.102.0.0/20
66.249.64.0/19
70.32.128.0/19
72.14.192.0/18
74.114.24.0/21
74.125.0.0/16
104.154.0.0/15
104.196.0.0/14
104.237.160.0/19
107.167.160.0/19
107.178.192.0/18
108.59.80.0/20
108.170.192.0/18
108.177.0.0/17
130.211.0.0/16
136.112.0.0/12
142.250.0.0/15
146.148.0.0/17
162.216.148.0/22
162.222.176.0/21
172.110.32.0/21
172.217.0.0/16
172.253.0.0/16
173.194.0.0/16
173.255.112.0/20
192.158.28.0/22
192.178.0.0/15"
193.186.4.0/24"
199.36.154.0/23"
199.36.156.0/24"
199.192.112.0/22"
199.223.232.0/21"
207.223.160.0/20"
208.65.152.0/22"
208.68.108.0/22"
208.81.188.0/22"
208.117.224.0/19"
209.85.128.0/17"
216.58.192.0/19"
216.73.80.0/20"
216.239.32.0/19"
2001:4860::/32
2404:6800::/32
2404:f340::/32
2600:1900::/28
2606:73c0::/32
2607:f8b0::/32
2620:11a:a000::/40
2620:120:e000::/40
2800:3f0::/32
2a00:1450::/32
2c0f:fb50::/32
Step 3: Allowing Google Proxy hostname
The Google Ad Bot has a proxy name that should also be allowed
Ad this to you .htaccess file - above all redirect rules and just after the host redirect.
Keep checking the logs after you add a campaign.
#allow google adbots and crawlers
RewriteEngine On
RewriteCond %{HTTP_HOST} rate-limited-proxy-[0-9]-[0-9]-[0-9]-[0-9].google.com [OR]
RewriteCond %{REMOTE_ADDR} =8.8.4.0/24 [OR]
RewriteCond %{REMOTE_ADDR} =8.8.8.0/24 [OR]
RewriteCond %{REMOTE_ADDR} =8.34.208.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =8.35.192.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =23.236.48.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =23.251.128.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =34.3.3.0/24 [OR]
RewriteCond %{REMOTE_ADDR} =34.4.4.0/24 [OR]
RewriteCond %{REMOTE_ADDR} =34.64.0.0/10 [OR]
RewriteCond %{REMOTE_ADDR} =34.128.0.0/10 [OR]
RewriteCond %{REMOTE_ADDR} =35.184.0.0/13 [OR]
RewriteCond %{REMOTE_ADDR} =35.192.0.0/14 [OR]
RewriteCond %{REMOTE_ADDR} =35.196.0.0/15 [OR]
RewriteCond %{REMOTE_ADDR} =35.198.0.0/16 [OR]
RewriteCond %{REMOTE_ADDR} =35.199.0.0/17 [OR]
RewriteCond %{REMOTE_ADDR} =35.199.128.0/18 [OR]
RewriteCond %{REMOTE_ADDR} =35.200.0.0/13 [OR]
RewriteCond %{REMOTE_ADDR} =35.208.0.0/12 [OR]
RewriteCond %{REMOTE_ADDR} =35.224.0.0/12 [OR]
RewriteCond %{REMOTE_ADDR} =35.240.0.0/13 [OR]
RewriteCond %{REMOTE_ADDR} =64.15.112.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =64.233.160.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =66.102.0.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =66.249.64.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =70.32.128.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =72.14.192.0/18 [OR]
RewriteCond %{REMOTE_ADDR} =74.114.24.0/21 [OR]
RewriteCond %{REMOTE_ADDR} =74.125.0.0/16 [OR]
RewriteCond %{REMOTE_ADDR} =104.154.0.0/15 [OR]
RewriteCond %{REMOTE_ADDR} =104.196.0.0/14 [OR]
RewriteCond %{REMOTE_ADDR} =104.237.160.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =107.167.160.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =107.178.192.0/18 [OR]
RewriteCond %{REMOTE_ADDR} =108.59.80.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =108.170.192.0/18 [OR]
RewriteCond %{REMOTE_ADDR} =108.177.0.0/17 [OR]
RewriteCond %{REMOTE_ADDR} =130.211.0.0/16 [OR]
RewriteCond %{REMOTE_ADDR} =136.112.0.0/12 [OR]
RewriteCond %{REMOTE_ADDR} =142.250.0.0/15 [OR]
RewriteCond %{REMOTE_ADDR} =146.148.0.0/17 [OR]
RewriteCond %{REMOTE_ADDR} =162.216.148.0/22 [OR]
RewriteCond %{REMOTE_ADDR} =162.222.176.0/21 [OR]
RewriteCond %{REMOTE_ADDR} =172.110.32.0/21 [OR]
RewriteCond %{REMOTE_ADDR} =172.217.0.0/16 [OR]
RewriteCond %{REMOTE_ADDR} =172.253.0.0/16 [OR]
RewriteCond %{REMOTE_ADDR} =173.194.0.0/16 [OR]
RewriteCond %{REMOTE_ADDR} =173.255.112.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =192.158.28.0/22 [OR]
RewriteCond %{REMOTE_ADDR} =192.178.0.0/15 [OR]
RewriteCond %{REMOTE_ADDR} =193.186.4.0/24 [OR]
RewriteCond %{REMOTE_ADDR} =199.36.154.0/23 [OR]
RewriteCond %{REMOTE_ADDR} =199.36.156.0/24 [OR]
RewriteCond %{REMOTE_ADDR} =199.192.112.0/22 [OR]
RewriteCond %{REMOTE_ADDR} =199.223.232.0/21 [OR]
RewriteCond %{REMOTE_ADDR} =207.223.160.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =208.65.152.0/22 [OR]
RewriteCond %{REMOTE_ADDR} =208.68.108.0/22 [OR]
RewriteCond %{REMOTE_ADDR} =208.81.188.0/22 [OR]
RewriteCond %{REMOTE_ADDR} =208.117.224.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =209.85.128.0/17 [OR]
RewriteCond %{REMOTE_ADDR} =216.58.192.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =216.73.80.0/20 [OR]
RewriteCond %{REMOTE_ADDR} =216.239.32.0/19 [OR]
RewriteCond %{REMOTE_ADDR} =2001:4860::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2404:6800::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2404:f340::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2600:1900::/28 [OR]
RewriteCond %{REMOTE_ADDR} =2606:73c0::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2607:f8b0::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2620:11a:a000::/40 [OR]
RewriteCond %{REMOTE_ADDR} =2620:120:e000::/40 [OR]
RewriteCond %{REMOTE_ADDR} =2800:3f0::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2a00:1450::/32 [OR]
RewriteCond %{REMOTE_ADDR} =2c0f:fb50::/32 [OR]
RewriteCond %{HTTP_USER_AGENT} ^AdsBot-Google
RewriteRule ^.* - [R,L]