Notices by Allison Parrish (aparrish@mastodon.social), page 16

since the dst change I've been waking up in the morning right in the middle of a REM cycle (maybe my only REM cycle of the night because I am not good at getting enough sleep) and it's reeeeally starting to mess me up

@wrenpile characterizing semantics would be a different set of techniques altogether (topic modeling). I'm personally more interested in keywords as a way of analyzing distinctive word *choices* in a text even if those words have similar meanings to higher-frequency words.

@wrenpile "keyword" in this context meaning words that are particular to the text in question, as opposed to some other randomly chosen English text. the insight here is that many texts contain the word "beautiful," but shakespeare's sonnets are special in that the word "beauteous" occurs frequently there but not in English in general

(for comparison, the 7+ character adjectives deemed least keyword-like by this method are words like 'married', 'curious', 'necessary', 'private', 'religious', 'forward', 'beautiful', 'several', 'certain', 'different' etc. and of course it works with *all* kinds of words— limiting it to just 7+ character adjectives was just a fun experiment)

(this is from my latest attempt to do the thing that I was talking about here https://mastodon.social/@aparrish/99660550376065760, i.e., simple keyword extraction from small corpora using only stuff you get with spaCy, in particular spaCy's English-wide unigram log probabilities. current solution: use G² [see e.g. http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html], as implemented in scipy's chi2_contingency function, using a wild guess for what the actual token count is for spaCy's unigram frequencies. I guessed 1000000 and it seems to work fine)

@MightyPork sorry, I should have said "adjectives according to spaCy's POS tagger" and yes, any word is an adjective if you believe in yourself

top 25 keyword adjectives with length greater than seven characters from shakespeare's sonnets

@swizzard @pnathan yepppp

@pnathan it's not good enough for the onion tbqh

finally gave up on tapping "less like this" on the borowitz report in my phone news app and just muted the new yorker entirely

@enkiv2 this paper is super helpful btw, thanks! (I didn't know that this task also went under the name "automatic terminology recognition")

@enkiv2 yeah, that's it exactly. in the paper you linked it's called "Weirdness" (by doing log(p in doc) / log(p in ref) I'm sorta implicitly scaling the sizes of the corpora)

@js0000 unfortunately no ebook version at the moment :(