Web crawlers that crawl HTML pages to discover more resources don’t work these days. At least, not always. Thanks to the dynamic web technologies.
This blog post by PortSwigger explains the problem and then describes how they solved it in Burp Suite. Instead of embedding a JavaScript engine, they embedded the open-source Chromium browser. Embedding Chromium browser allows Burp’s crawler to behave like a human user.