https://find.data.gov.scot/datasets/34473
Gaelic POS tagger and tokeniser
Dtechtive discovers the datasets other search engines cannot reach. It also provides insights on dataset quality and usage, to help both data users and data...
gaelicpostaggertokeniser
https://www.freshports.org/math/py-mathics-scanner/
FreshPorts -- math/py-mathics-scanner: Tokeniser, char tables, & conversion routines for the...
This is a tokeniser or scanner portion for the Wolfram Language. As such, it also contains a full set of translation between Wolfram Language named characters,...
for thefreshportsmathpyscanner
https://lists.w3.org/Archives/Public/public-html-bugzilla/2010Apr/0277.html
[Bug 5863] Tokeniser: "Bogus comment" state unclear from bugzilla@jessica.w3.org on 2010-04-19...
bugtokeniserboguscommentstate