includes/Integrations/class-litespeed-crawler.php
LiteSpeed Crawler — curl_multi variant-matrix preloader.
Class LiteSpeed_Crawler
Class LiteSpeed_Crawler
final class LiteSpeed_CrawlerConstants
| Constant | Visibility | Value | Line |
|---|---|---|---|
BLACKLIST_THRESHOLD | public | 3 | 38 |
DEFAULT_CONCURRENCY | public | 2 | 46 |
BLACKLIST_PREFIX | private | \'wppo_crawler_blacklist_\' | 54 |
SERVER_IP_KEY | private | \'wppo_crawler_server_ip\' | 62 |
DEFERRED_BATCH_MAX_URLS | public | 50 | 70 |
DEFERRED_BATCH_TTL | private | 2 * HOUR_IN_SECONDS | 78 |
MAX_REDIRECT_HOPS | private | 5 | 93 |
publicstatic get_concurrency()
public static function get_concurrency(): intGet concurrency (2-4) filtered via wppo_crawler_concurrency.
Return: int.
publicstatic get_blacklist_threshold()
public static function get_blacklist_threshold(): intGet blacklist threshold filtered via wppo_crawler_blacklist_threshold.
Return: int.
publicstatic get_load_limit()
public static function get_load_limit(): floatGet load limit for throttling.
Return: float.
publicstatic is_overloaded()
public static function is_overloaded(): boolWhether system load is over limit (throttler).
Return: bool.
publicstatic get_server_ip()
public static function get_server_ip(): stringGet cached server IP for CURLOPT_RESOLVE.
Return: string — IP or empty string.
publicstatic is_blacklisted()
public static function is_blacklisted(string $url): boolWhether URL is blacklisted (failures >= threshold). Parameter Type Default Description $urlstring— URL.
Return: bool.
publicstatic record_failure()
public static function record_failure(string $url): voidRecord failure for URL; increment blacklist counter. Parameter Type Default Description $urlstring— URL.
Return: void.
publicstatic clear_blacklist()
public static function clear_blacklist(string $url): voidClear blacklist for URL on success. Parameter Type Default Description $urlstring— URL.
Return: void.
publicstatic build_variant_matrix()
public static function build_variant_matrix(string $url): arrayBuild variant matrix for a URL. Parameter Type Default Description $urlstring— Base URL.
Return: array<int, — array{url:string,headers:array<string,string>}>.
publicstatic get_variants_to_warm()
public static function get_variants_to_warm(string $url): arrayGet variants to warm respecting varyGroups. Parameter Type Default Description $urlstring— Base URL.
Return: array<int, — array{url:string,headers:array<string,string>}>.
publicstatic get_urls_to_crawl()
public static function get_urls_to_crawl(int=500 $cap): arrayGet URLs to crawl (posts + sitemap merge, 500 cap, 15s budget). Parameter Type Default Description $capint=500— Cap for sitemap + total URLs.
Return: string[].
privatestatic is_url_same_host_allowed()
private static function is_url_same_host_allowed(string $url): boolWhether a URL is allowed as a crawler target or redirect landing. Parameter Type Default Description $urlstring— URL to check.
Return: bool — True when the URL is allowed.
privatestatic resolve_validated_redirect()
private static function resolve_validated_redirect(string $location, string $current_url): string|falseResolve a redirect Location against the current URL and validate it. Parameter Type Default Description $locationstring— Raw Location header value. $current_urlstring— URL of the response that sent the Location.
Return: string|false — Absolute validated URL, or false when the hop is not allowed.
publicstatic crawl_batch()
public static function crawl_batch(array $urls, ?int=null $concurrency): arrayExecute curl_multi batch for URLs (variant matrix). Parameter Type Default Description $urlsarray— URLs to crawl. $concurrency?int=null— Concurrency override or null for setting.
Return: array{success:int,failed:int,skipped:int}.
privatestatic defer_crawl_batch()
private static function defer_crawl_batch(array $urls): voidDefer a crawler batch when the server is overloaded. Parameter Type Default Description $urlsarray— URLs to defer.
Return: void.
publicstatic crawl_single()
public static function crawl_single(string $url): voidLane wrapper for single URL preload (wraps process_url). Parameter Type Default Description $urlstring— URL.
Return: void.
publicstatic reset_cache()
public static function reset_cache(): voidReset crawler runtime state for testing isolation.
Return: void.
publicstatic get_status()
public static function get_status(): arrayGet crawler queue status.
Return: array{concurrency:int,load_limit:float,overloaded:bool,server_ip:string}.
Hooks
Hooks referenced in includes/Integrations/class-litespeed-crawler.php: Hook Type Line Notes wppo_crawler_concurrencyfilter 114 @param ×1 wppo_crawler_blacklist_thresholdfilter 147 @param ×1 wppo_crawler_use_nprocfilter 169 — wppo_crawler_load_limitfilter 196 @param ×1 wppo_crawler_is_overloadedfilter 238 @param ×3 wppo_crawler_variantsfilter 353 @param ×2 wppo_crawler_full_matrixfilter 374 — wppo_crawler_variantsfilter 426 @param ×2 wppo_crawler_sitemap_urlsfilter 469 @param ×2 wppo_crawler_urlsfilter 499 @param ×1 wppo_invalidation_urlsfilter 500 @param ×1 wppo_crawler_urlsfilter 611 @param ×1 wppo_invalidation_urlsfilter 618 @param ×1 wppo_crawler_disable_curlfilter 749 @param ×1