Rob Brackett Mr0grog

Comparing Browserstrix-crawler v1.8.1 vs. v1.9.0 on EPA URLs

These are some quick and simple tools for comparing performance of different versions of browsertrix-crawler (using the webrecorder/browsertrix-crawler Docker images). You'll need Docker and uv to run this.

First, adjust epatest.yaml for whatever settings you’d like.
Then, run run-many.sh <name-of-scenario>. This will run 3 crawls each with v1.8.1 and v1.9.0, where the resulting collections are named epatest--<version>--<scenario>--<index>. For example:
```
# Try it with default settings:
```

Fastest way to check for a BOM in Python

Results:

Testing nesting...
             Baseline (A):: Min: 0.29188, mean: 0.29287
              Nesting (B):: Min: 0.25730, mean: 0.25863
 Nesting w/ Substring (C):: Min: 0.25488, mean: 0.25534
 Nesting w/ each byte (D):: Min: 0.24767, mean: 0.24830

EPA `/climate*` Crawl (2021-09-27)

This is a list of URLs found by crawling the EPA’s website starting from https://www.epa.gov/climate-change and including any page with a URL like https://*epa.gov/climate*.

It also includes pages that can be reached by going through https://www.epa.gov/node/<number>, since many of those are shortcuts for more nicely named pages that live at more canonical URLs like https://*epa.gov/climate*.

URLs are sorted alphabetically for nice, visual grouping.

	/**
	* Delete e-mails older than N days based on their label(s).
	* Starred/flagged e-mails are never deleted.
	*
	* Set this up as a project at https://script.google.com/
	* And create a trigger that runs `cleanUpMailers` every hour.
	*/

	var DAY = 24 * 60 * 60 * 1000;

	import log from 'loglevel';
	import logPrefix from 'loglevel-plugin-prefix';
	import { logFromA } from './module-a.js';
	import { logFromB } from './module-b.js';

	logPrefix.reg(log);
	logPrefix.apply(log, { template: '[%t] %l %n:' });
	log.enableAll();

	// Comment out the imports at the top of the file and uncomment these to make

	"""
	Quickie script for comparing legacy single-byte character encoding definitions
	from the Unicode Consortium (found at https://unicode.org/Public/MAPPINGS/) and
	the WHATWG (at https://encoding.spec.whatwg.org/#legacy-single-byte-encodings or
	https://github.com/whatwg/encoding), since they differ slightly.

	Typically, you'll want to download a copy of the Unicode mapping files via FTP:

	ncftpget -R ftp.unicode.org . Public/MAPPINGS
	mv MAPPINGS unicode

	{
	"services": [
	{
	"type": "Vaccine",
	"name": "OTC COVID TEST KITS 1",
	"label": "OTC COVID TEST KITS 1",
	"scientificname": "",
	"epsDisplayName": "OTC COVID TEST KITS 1",
	"eligible": true,
	"description": "No Age Restrictions Present.",

	import EventEmitter from "node:events";
	import * as Sentry from "@sentry/node";
	import { Transaction } from "@sentry/tracing";
	import type { Span, SpanStatusType } from "@sentry/tracing";
	import type { SpanContext } from "@sentry/types";
	export type { Span } from "@sentry/tracing";

	interface SpanOptions extends SpanContext {
	parentSpan?: Span;
	timeout?: number;

	/**
	* Parse Airtable's "ConstantPooledData" format. They recently started using
	* this format to compress some API responses, and it appears to be a
	* home-grown format.
	*
	* Call `parseData()` if you have an object with data (e.g. a JSON-parsed API
	* response body).
	*
	* Call `parseString()` if you have a raw string of data (e.g. an API response
	* body).

	/**
	* Test out the performance and memory usage of various ways to compose a gzip
	* of streaming JSON log lines.
	*
	* One fun property of gzip is that a stream with multiple gzip chunks is
	* itself valid as a single gzip stream/file. That means there are lots of
	* interesting ways to break down how you build and stream out gzip data.
	*/

	const fs = require("fs");

Rob Brackett Mr0grog

Comparing Browserstrix-crawler v1.8.1 vs. v1.9.0 on EPA URLs

Fastest way to check for a BOM in Python

EPA /climate* Crawl (2021-09-27)

Missing URLs

EPA `/climate*` Crawl (2021-09-27)