Skip to content

Instantly share code, notes, and snippets.

View rrbutani's full-sized avatar
🐢
slowly but surely

Rahul Butani rrbutani

🐢
slowly but surely
  • 15:45 (UTC -08:00)
View GitHub Profile
❯ pytest --chrome-trace parallel-baseline.json -n auto --dist=worksteal
664 passed, 24 warnings in 9.08s

chrome trace of running pytest w/xdist's workstealing strategy

❯ pytest --chrome-trace parallel-reordered.json -n auto --dist=worksteal --expensive-tests-first
We couldn’t find that file to show.

observations:

  1. path mapping does not really[^1] "mask" out the repo name from non-main repo paths: source files OR generated artifacts
    • this means that whether you build a particular target (or really produce a particular action for the target) when that target is in the main (root) repo vs. when its in an "external" repo effects the paths used in the action
    • i.e. you will not get cache hits if you build //:your_target and then pull your repo into another bazel workspace as @foo and build @foo//:your_target
    • TODO: would be nice to repo names as well...
  2. path mapping silently falls back to using unmapped paths if the set of mapped paths has any conflicts (see :example_both)
  • note that ctx.actions.run(_shell)'s inputs does influence path mapping and does appear to factor into how conflicts are detected but... if you try to add a path to the command line (i.e. via arguments) that you haven't listed in inputs it isn't an error — some path mapping heuristi

Enforcement of allowed dep edges

the idea is to have testonly-esque checks

consider a project where we want to partition content (for an existing rule) into three categories:

  • "open-source"; a.k.a. oss
  • "proprietary"; a.k.a. prp
  • "development-only"; a.k.a. dev
//! for a given `N`, how many unique shapes of `N` contiguous minos (all connected via at least one
//! of the four cardinal directions) are possible?
//!
//! a shape is unique if no translation or rotation of a shape makes it the same as another shape
use std::{collections::HashSet, fmt};
#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
struct Coord<const N: usize> {
x: u8, // 0 <= x < N

see:

semantics of order-only deps:

  • when deciding whether to rebuild a target, the mtimes (or existence) of order-only deps are not considered
  • if the target is built (i.e. due to a normal prereq that was newer), it will wait on order-only deps
  • even if the target is not built (i.e. up to date), order-only deps will be scheduled for building (if out of date)

what

tests the we can "project" particular files out of a TreeArtifact for consumption in downstream rules

the intent is that by doing this projection:

  • downstream rules that operate on files (not directories — i.e. not TreeArtifact aware) can consume our artifact
  • sensitivity in downstream targets is narrowed to only the files in the TreeArtifact that are projected out