Docs to HTML Converter

This repository contains convert_tex_to_html.py, a self-contained Python script that scans a source folder of LaTeX documents and converts the main documents to HTML using tex4ht tooling. It prefers make4ht and falls back to htlatex when needed. It also generates a simple HTML viewer and index to browse the converted docs.

See SETUP.md for installation requirements.

What It Does

• Finds .tex files under a source directory (default tmp_Docs).
• Heuristically selects “main” documents that contain a \documentclass (unless --all-tex is set).
• Attempts HTML conversion with make4ht (HTML5 output with either MathML+MathJax or TeX+MathJax) and falls back to htlatex if needed.
• Handles minted by ensuring -shell-escape is used, preferring cached output, and surfacing errors clearly.
• Copies generated HTML and any local assets to a destination folder (default tmp_Docs_html), adds small CSS overrides, and optionally injects MathJax into HTML.
• Produces docs_index.json and an index.html viewer to navigate all converted documents.

Command-Line Usage

python convert_tex_to_html.py [options]

Options:
  --src SRC            Source directory with .tex files (default: tmp_Docs)
  --dst DST            Output directory for HTML (default: tmp_Docs_html)
  --all-tex            Convert every .tex file, not just main docs
  --clean              Clean each output subfolder before converting
  --math-output {mathml,mathjax}
                       Math format for make4ht output:
                       - mathml  : write MathML; rendered via MathJax SVG
                       - mathjax : keep TeX; rendered via MathJax SVG
  --shell-escape       Enable -shell-escape for LaTeX runs (default)
  --no-shell-escape    Disable -shell-escape for LaTeX runs
  --engine {pdflatex,lualatex,xelatex}
                       LaTeX engine to use (default: pdflatex)
  --postbuild-fix-images
                       Run the post-build image fixer to copy/normalize
                       equation PNGs referenced by *_htmlwrap.html

Typical runs:

• Convert all main documents: python convert_tex_to_html.py
• Force convert every .tex: python convert_tex_to_html.py --all-tex
• Clean outputs first: python convert_tex_to_html.py --clean
• Use MathML flavor: python convert_tex_to_html.py --math-output mathml
• Convert and auto-fix image references: python convert_tex_to_html.py --postbuild-fix-images

Outputs for each .tex go under DST/<subdir>/<stem>/. Each conversion also writes a build.log in the document output folder with the detailed tool logs.

Post-Build Image Fixes

Some tex4ht outputs reference equation images as <stem>_htmlwrapNx.png while the LaTeX run produces <stem>Nx.png in the source folder. To prevent 404s, run:

make postbuild-fix-images

This runs scripts/post_build_fix_images.py, which:

• Copies math/equation PNGs from tmp_Docs/<Module>/ to the corresponding tmp_Docs_html/.../ locations, renaming to the expected _htmlwrapNx.png pattern.
• Normalizes case-only file name mismatches (e.g., .PNG → .png) next to the HTML files when needed.

How It Works (Pipeline)

1. Detection: For each .tex, detect_main_tex reads at most the first 200 lines and flags files containing \documentclass as main docs (unless --all-tex is used).

2. Environment setup: The script populates:

   • TEXINPUTS to include the .tex parent and source roots, ensuring local includes are found.
   • PATH with ~/.local/bin (for user-installed pygmentize) and ./bin.

3. make4ht attempt: try_tex4ht_html tries make4ht first with:

   • Format: html5+mathjax or html5+mathml.
   • Shell-escape: Adds -s to make4ht and passes -interaction=nonstopmode and -shell-escape as the LaTeX argument when enabled.
   • Minted handling: If the source uses \usepackage{minted}, convert_one creates a tiny wrapper <stem>_htmlwrap.tex that does:
     • \PassOptionsToPackage{frozencache,cachedir=_minted-<stem>}{minted}
     • \input{<orig>.tex} This minimizes re-running pygmentize by reusing cached outputs when present.
   • Error detection: If make4ht logs contain common minted errors (missing -shell-escape or Pygments output), the script triggers the fallback even if an HTML file was produced.

4. Fallback to htlatex: If needed, htlatex is called with correct option ordering:

   • htlatex <file> "tex4ht opts" "t4ht opts" "latex opts"
   • Output dir is passed in the t4ht argument (e.g., -d<dir>), and LaTeX options include -interaction=nonstopmode and (if enabled) -shell-escape.
   • The environment variables LATEX and TEX are set to the chosen engine and -shell-escape when enabled, for wrappers that respect them.

5. Result handling: The produced HTML is copied into the destination folder (if generated elsewhere), together with any linked local assets (CSS/JS/images) and an assets directory named like the document stem.

6. Post-process: A small CSS file (zzz_overrides.css) is added and linked into each page. When make4ht succeeds, the script injects a MathJax <script> snippet for math rendering; for htlatex fallback we keep MathJax injection off, as tex4ht often converts math to images.

7. Catalog and Viewer: All successful conversions are gathered into docs_index.json and a simple index.html viewer is written to the destination root for browsing.

Function Reference

• detect_main_tex(tex_path: Path) -> bool

  • Heuristic: returns True if \documentclass is found in the first ~200 non-comment lines.

• which(cmd: str) -> Optional[str]

  • Wrapper for shutil.which.

• run(cmd: List[str], cwd: Path, env: dict, timeout: int) -> Tuple[int, str, str]

  • Executes a subprocess with given working directory and environment, returns (exit_code, stdout, stderr); kills on timeout.

• _collect_local_links(html: Path) -> Set[str]

  • Parses HTML and extracts local src/href references for asset copying.

• _inject_overrides(html_file: Path, css_name: str, mj_snippet: Optional[str]) -> None

  • Inserts a link to css_name and an optional MathJax snippet into a single HTML file.

• _inject_in_all_html(out_dir: Path, mj_snippet: Optional[str]) -> None

  • Writes a small zzz_overrides.css (if missing) and injects overrides/MathJax into all HTML files in out_dir.

• try_tex4ht_html(tex: Path, out_dir: Path, env: dict, timeout: int = 3600, math_output: str = 'mathjax', shell_escape: bool = True, engine: str = 'pdflatex', extra_latex_opts: str = '', jobname: Optional[str] = None) -> Tuple[bool, str, Optional[Path]]

  • Orchestrates the conversion using make4ht first, then htlatex as a fallback. Accepts extra LaTeX options and a jobname (used by the minted wrapper).

• convert_one(tex: Path, src_root: Path, dst_root: Path, clean: bool = False, math_output: str = 'mathjax', shell_escape: bool = True, engine: str = 'pdflatex') -> Tuple[bool, str, Path]

  • Prepares the output folder, environment, and minted wrapper (if needed), then calls try_tex4ht_html and returns a tuple (ok, summary, html_path_or_dir).

• _extract_title(html_file: Path) -> str

  • Retrieves an HTML page title from <title> or the first <h1>/<h2>; falls back to the filename stem.

• _write_catalog(dst_root: Path, items) -> int

  • Writes docs_index.json with the list of successfully converted docs and their titles; returns the count written.

• _write_viewer(dst_root: Path) -> None

  • Writes a simple, static index.html that loads the catalog and lets you browse documents.

• main()

  • CLI entry point. Parses arguments, discovers .tex files, converts them, and writes the catalog/viewer.

Minted, Pygments, and Caching

minted calls pygmentize via shell escape. The converter enables -shell-escape by default, and creates a wrapper that enables frozencache in minted to reuse previously generated .pyg outputs. If your environment disallows shell escape, run a one-time pdflatex -shell-escape on the source documents to populate _minted-<jobname> cache directories; subsequent HTML conversions can then reuse the cache.

Troubleshooting

• “Package minted Error: You must invoke LaTeX with the -shell-escape flag.”

  • Ensure your TeX run permits -shell-escape (the script requests it by default).
  • Ensure pygmentize is installed and on PATH.

• “Missing Pygments output; \inputminted was …”

  • Same as above; also verify the _minted-<stem> cache exists or prebuild it by running pdflatex -shell-escape once.

• Math not rendering in the browser

  • make4ht path injects MathJax from a CDN; ensure network access. The htlatex fallback typically uses images for math and should work offline.

• Broken relative images/links

  • The script copies assets listed in HTML src/href. If your document references assets generated by external tools during LaTeX runs, ensure those tools ran and outputs exist before conversion.

Development Notes

• The script uses only the Python standard library. No Python packages are required beyond core Python.
• Conversion logs are written per-document to help diagnose toolchain issues.
• Changes were made to ensure proper argument ordering for htlatex and to pass shell-escape reliably to both make4ht and htlatex paths.