tokenizers Abner-Machado · PARTIAL
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
github.com/Abner-Machado/tokenizers · ★ 2 · Forks 0 · Size 19.0 MB
SUMMARY
Technologies 22
Scored 21
Observed 21
Practices 6
Evidence 27
Skips 1
COVERAGE
Analyzed 363 files · 2004 commits · 0 API calls
TECHNOLOGIES & DEPTH
TypeScript LANGUAGE Depth 70
12 files · PRODUCTION
Shell LANGUAGE Depth 70
1 files · PRODUCTION
Rust LANGUAGE Depth 70
130 files · PRODUCTION
JavaScript LANGUAGE Depth 70
8 files · PRODUCTION
JSON LANGUAGE Depth 70
18 files · PRODUCTION
TOML LANGUAGE Depth 70
10 files · PRODUCTION
Markdown LANGUAGE Depth 70
43 files · PRODUCTION
YAML LANGUAGE Depth 70
24 files · PRODUCTION
Python LANGUAGE Depth 70
64 files · PRODUCTION
Cargo BUILD_TOOL Depth 80
1 files · CONFIGURATION
npm BUILD_TOOL Depth 80
1 files · CONFIGURATION
Poetry BUILD_TOOL Depth 80
1 files · CONFIGURATION
Yarn BUILD_TOOL Depth 80
1 files · CONFIGURATION
ESLint QUALITY Depth 84
5 files · CONFIGURATION
Elasticsearch DATABASE Depth 65
1 files · CONFIGURATION
NumPy LIBRARY Depth 37
1 files · TEST
Prettier QUALITY Depth 73
3 files · CONFIGURATION
Requests LIBRARY Depth 58
1 files · PRODUCTION
Serde LIBRARY Depth 78
61 files · PRODUCTION
Tokio LIBRARY Depth 58
1 files · PRODUCTION
Tracing OBSERVABILITY Depth —
0 files · config only
pytest TESTING Depth 74
20 files · PRODUCTION, TEST
PRACTICES
documentation · observedautomated_tests · observedcontinuous_integration · observedcontainerization · absentlinting · observedformatting · observed
ACTIVITY & OWNERSHIP
First commit 2019-11-01
Last commit 2026-07-15
Active months 78
Commits 2004