Skip to content

Instantly share code, notes, and snippets.

@DouglasOrr

DouglasOrr/2023-04-tokenizers-byte-fallback.ipynb

Last active May 16, 2024 15:35
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save DouglasOrr/b5ce3bda0787260051a738739f30b283 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save DouglasOrr/b5ce3bda0787260051a738739f30b283 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
2023-04-tokenizers-byte-fallback.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@DouglasOrr
Copy link
Author

DouglasOrr commented Apr 21, 2023

Note: end_of_word_suffix="</w>" was not rendering properly on GitHub.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment