GwmngilfenOK, so running tokenizer code on half a gb of text data *and* running system updates in the background might *not* have been the smartest move.
I should tokenise this stuff in batches...
Jonkman Microblog is a social network, courtesy of SOBAC Microcomputer Services. It runs on GNU social, version 1.2.0-beta5, available under the GNU Affero General Public License.
All Jonkman Microblog content and data are available under the Creative Commons Attribution 3.0 license.