Documentation
README
Download bot-gated PDFs via Wayback id_
Problem
Many think-tank and publisher sites (taxpolicycenter.org, urban.org, SSRN delivery) serve an HTML bot-challenge page instead of the PDF to non-browser clients. A browser User-Agent header does not help. The downloaded "PDF" is actually HTML.
Context / Trigger Conditions
curl -o file.pdf <url>succeeds but the file starts with<!DOC- pypdf raises
invalid pdf header: b'<!DOC'orPdfStreamError: Stream has ended unexpectedly - Firecrawl
scrapereturns clean markdown for the same URL (its proxies get through), but Firecrawl does not return the binary — only parsed content
This is the opening of the README. Read the full README on GitHub.