Python过cloudflare盾爬取数据

在对于绝大多数网站都有防爬的机制，比如人机验证已经cloud flare盾等等，其中最比较难受的就是过cloudflare的盾，下面介绍一种方法过cloudflare的盾的python小技巧

怎么过盾

1.对于win端在性能可以的情况下可以使用，pyautogui的方法点击过盾，通过自动化库比如DrissionPage库来获取需要点击的坐标输入到pyautogui让其自动点击过盾，这种比较简单就不贴代码了

2.对于Linux或者无头使用的场景下，无法使用pyautogui来点击过盾，使用只能使用下面的方法来过盾了
先需要对页面注入js代码，因为会有些问题如果不注入的话会被检测出来

保存为manifest.json

json

{
    "manifest_version": 3,
    "name": "Turnstile Patcher",
    "version": "2.1",
    "content_scripts": [
        {
            "js": [
                "./script.js"
            ],
            "matches": [
                "<all_urls>"
            ],
            "run_at": "document_start",
            "all_frames": true,
            "world": "MAIN"
        }
    ]
}

保存为script.js

javascript

function getRandomInt(min, max) {
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

// old method wouldn't work on 4k screens

let screenX = getRandomInt(800, 1200);
let screenY = getRandomInt(400, 600);

Object.defineProperty(MouseEvent.prototype, 'screenX', { value: screenX });

Object.defineProperty(MouseEvent.prototype, 'screenY', { value: screenY });

将上面两个文件保存在同一个文件夹，然后将其注册为插件使用

各种自动化库都有对应的方法，这里对应DP单独写出

python

page_new = browser.new_tab("你的url")
t = page.run_js("try { return turnstile.getResponse() } catch(e) { return null }")

大概的内容为

python

from DrissionPage import Chromium, ChromiumOptions
import time
import os

co = ChromiumOptions()
co.auto_port()

co.set_timeouts(base=1)

EXTENSION_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), "turnstilePatch"))
co.add_extension(EXTENSION_PATH)

co.headless()
from sys import platform

if platform == "linux" or platform == "linux2":
    platformIdentifier = "X11; Linux x86_64"
elif platform == "darwin":
    platformIdentifier = "Macintosh; Intel Mac OS X 10_15_7"
elif platform == "win32":
    platformIdentifier = "Windows NT 10.0; Win64; x64"

co.set_user_agent(
    f"Mozilla/5.0 ({platformIdentifier}) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36")
co.set_argument('--incognito')
co.set_argument('--no-sandbox')
co.set_argument('--headless=new')
co.set_argument('--disable-gpu')
browser = Chromium(co)
page = browser.latest_tab()

page.get("https://example.com/")
page.wait(30)
page.get_screenshot(f'1111_t.png')

def getTurnstileToken():
    page.run_js("try { turnstile.reset() } catch(e) { }")

    turnstileResponse = None

    for i in range(0, 15):
        try:
            turnstileResponse = page.run_js("try { return turnstile.getResponse() } catch(e) { return null }")
            if turnstileResponse:
                page.get_screenshot(f'{i}_t.png')
                return turnstileResponse

            challengeSolution = page.ele("@name=cf-turnstile-response")
            challengeWrapper = challengeSolution.parent()
            challengeIframe = challengeWrapper.shadow_root.ele("tag:iframe")
            challengeIframeBody = challengeIframe.ele("tag:body").shadow_root
            challengeButton = challengeIframeBody.ele("tag:input")
            challengeButton.click()
            page.get_screenshot(f'{i}-1.png')
        except:
            pass
        time.sleep(1)
    page.refresh()
    raise Exception("failed to solve turnstile")

while True:
    print(getTurnstileToken())

Python过cloudflare盾爬取数据

怎么过盾

你可能也喜欢

评论