What we saw last time

• APIs
• The web as a gigantic API
• the REST protocol
• inspecting a website (hack 101)

• building datasets from APIs

• Guided practice on the wikipedia API

Today

• classic NLP
  • tokens
  • NER: Named Entity Recognition
  • POS: Part of Speech
  • topic modeling
  • text classification : sentiment, hate speech, categorisaton, spam etc

• some libraries : Spacy.io, NLTK

• Practice on the NYT API to build a dataset and apply classic NLP techniques

At the end of this class

You

• understand what NLP is
• can query the NYT API
• can apply classic NLP techniques to a dataset using Spacy.io

YouTube just dropped over 30 new creator tools at its Made on YouTube event, including AI-powered editing and clipping features, the addition of Veo 3 Fast in shorts, auto-dubbing, and more.

https://www.heygen.com/

The Rundown: Researchers at Stanford and the Arc Institute just created the first AI-generated, entirely new viruses from scratch that successfully infect and kill bacteria, marking a breakthrough in computational biology.

The details:

Scientists trained an AI model called Evo on 2M viruses, then asked it to design brand new ones — with 16 of 302 attempts proving functional in lab tests.

The AI viruses contained 392 mutations never seen in nature, including successful combos that scientists had previously tried and failed to engineer.

When bacteria developed resistance to natural viruses, AI-designed versions broke through defenses in days where the traditional viruses failed.

One synthetic version incorporated a component from a distantly related virus, something researchers had attempted unsuccessfully to design for years.

AI’s huge competitive coding win…

the neuron daily ai’s huge competitive coding win

OpenAI’s AI achieved a perfect score at the world’s top programming competition, beating all human teams.

OpenAI’s reasoning models achieved a perfect 12/12 score at the ICPC World Finals, the most prestigious programming competition in the world… and outperforming every human team.

To put this in perspective, the best human team solved 11 out of 12 problems. And OpenAI competed under the same 5-hour time limit as human teams. They used an ensemble of general-purpose models, including GPT-5, with no special training for competitive programming. In fact, 11 out of 12 problems were solved on the first try.

Google’s Gemini 2.5 Deep Think also won gold, solving 10 out of 12 problems. This means two different AI systems both outperformed every human team on the planet. And Google’s performance was equally jaw-dropping:

Gemini solved 8 problems in just 45 minutes and cracked one problem that stumped every single human team.

from Linguistic to NLP

Other classic refs

Speech and Language Processing

Speech and Language Processing (3rd ed. draft) Dan Jurafsky and James H. Martin

Latest release: August 24 2025!!

Timeline

NLP Timeline

NLP timeline : the early days

NLP timeline

Language is … complicated

Language is … complicated

• Variable length
• Wide variety of complexity across languages
  • German: Donaudampfschiffahrtsgesellschaftskapitän (5 “words”)
  • Chinese: 50,000 different characters (2-3k to read a newspaper)
  • Slavic: Different word forms depending on gender, case, tense
• Encoding: unicode vs Ascii
• Unstructured data
• code switching
• idioms, Generational lingo, slang

classic NLP problems

• Text mining
• NER: Named Entity Recognition: LOC, PER,
• POS: Part of speech Tagging : nouns, adjectives, verbs, …
• Classification: sentiment analysis, spam, hate speech, …
• Topic identification, topic modeling
• WSD: word sense disambiguation : bank, fly
• STT: speech to text, text to speech

and more difficult tasks such as

• Automated Translation, summarization, question answering

Classic NLP

• deterministic (not probabilistic - LLMs)
• based on the decomposition of text into identifiable elements: words, grammar roles, entities, etc
• applied to sentences, noun phrases, words
• includes pre processing methods of the raw text to facilitate processing
  • stop words: and, the, of, etc
  • stemming: universities, universal, universe -> univer (meaning is ofetn lost)
  • lemmatization: run, running, ran -> run (more efficient than stemming)
  • subword tokenization: “unhappiness” → [“un”, “happy”, “ness”]

Requires models, rules that are language specific. Russian or French need different lemmatizers than English.

What’s the unit of text ?

We could work with

• words, syllables, tokens, letters & punctuation
• Bigrams, n-grams: New York, cul-de-sac, pain au chocolat
• noun phrases: group of words that function as a noun: the big brown dog with spots
• sentences, paragraphes, tweets, articles, books, comments
• Corpus: a whole set of text

Need to deal with

• The vocabulary is large / infinite and fast changing
• Typos, multiple spellings
• word forms: plural, declension (home, house), conjugation, …,etc

Subword tokenization

What’s the most efficient unit of text ?

• Using words is problematic: very large vocabulary, multiple forms: plural, declension (home, house), conjugation, …,etc

• Using letters is too short

• Let’s split words into multiple tokens: subword tokenization
  • unhappiness -> un + happ + in + ess / un + hap + pin + ness
  • running -> runn + ing
  • universities -> uni + vers + iti + es

Benefits of subword tokenization

• no words are OOV: out of vocabulary (gpt 3.5 did not know about COVID)
• captures semantic and morphological meaning better
• Vocabulary Handling : we get infinite vocabulary from finite tokens

That’s also why LLms are very robust wrt to typos and misspellings.

Bag of words for binary classification

you want to build a model that can predict if an email is spam or not spam, or if a review is positive or negative (sentiment analysis)

To train the model you need to transform the corpus into numbers.

The main classic NLP method for that is tf-idf (term frequency-inverse document frequency)

For each sentence, we count the frequency of each word and we normalize it by the number of documents in the corpus that contain the word.

This gives us a matrix, that we can use to train a model.

but this appraoch has multiple problems

• OOV: Out of Vocabulary words are not taken into account
• large vocabulary => huge matrix
• full of zeros

etc

• works for easy tasks (spam, sentiment) but fails for more complex tasks

tf (term frequency) example

Let’s take an example. Consider the 3 following sentences from the well known Surfin’ Bird song and count the number of times each word appears in each sentence.

NER: named entities recognition

All types of predefined entities: location, groups, persons, companies, money, etc

NER uses

• Pattern matching: Looks for capital letters, titles (Mr., Dr.), known entity lists
• Context clues: Words like “works at” suggest an organization follows
• Statistical models: Trained on labeled data to recognize entity patterns

NER models and rules are language specific

POS: part of speech tagging

Identify the grammatical function of each word : ADJ, NOUN, VERBs, etc

POS uses:

• Word endings: “-ing” often = verb, “-ly” often = adverb
• Position rules: Determiners (“the”) come before nouns
• Context: Same word can be noun or verb (“run” vs “a run”)

POS models and rules are also language specific

Demo classic NLP : NER and POS

https://cloud.google.com/natural-language

Input some text: (text from FT on AI impact on traffic due to AI summaries in google search)

“Like everyone, we have definitely felt the impact of AI Overviews. There is only one direction of travel; not only are AIs getting better, but they’re getting better in an exponential fashion,” said Sean Cornwell, chief executive of Immediate Media, which owns the Radio Times and Good Food brands in the UK.

NER

Classification

• sentiment scoring
• categories
• Moderation

POS : part of speech and dependency tagging

Unfortunately this features is no longer available in the NLP google demo.

Tokens and LLMs

• Context Window = measured in tokens, not words : 1M token window, 200k tokens window, …
• Pricing Model = cost per token (input + output)
• Non-English Text = more tokens needed
• Token Limits = why responses cut off
• Character-Level Tasks = difficult (LLMs see tokens, not letters)
• Efficiency Varies = by language, domain, complexity

token based pricing

https://openai.com/api/pricing/

NER and POS with spacy.io

There are few important NLP python libraries : Spacy.io and NLTK

Spacy.io supports 75 languages,

• entity recognition, part-of-speech tagging, dependency parsing, sentence segmentation, text classification, lemmatization, morphological analysis, entity linking and more

=> follow Ines Montani (website) she’s super cool

Spacy code

To use Spacy.io we need to

• install and import the library
• then download a model associated to the language of the corpus.
  • Each language offers multiple models with varying sizes
  • Each model is trained to handle POS and NER and lemmatization
• Once the model is available we instanciate the spacy object doc on the text we want to analyze
• then we can easily extract NER like location or persons with these simple lines
• Similary we can identify all the ADJ and NOUNS in a text

# pip install -U spacy
# python -m spacy download en_core_web_sm
import spacy

# Load English tokenizer, tagger, parser and NER
nlp = spacy.load("en_core_web_sm")

# Process whole documents
text = ("When Sebastian Thrun started working on self-driving cars at "
        "Google in 2007, few people outside of the company took him "
        "seriously. “I can tell you very senior CEOs of major American "
        "car companies would shake my hand and turn away because I wasn’t "
        "worth talking to,” said Thrun, in an interview with Recode earlier "
        "this week.")

# instanciate the spacy object
doc = nlp(text)

# Analyze syntax
print("Noun phrases:", [chunk.text for chunk in doc.noun_chunks])
print("Verbs:", [token.lemma_ for token in doc if token.pos_ == "VERB"])

# Find named entities, phrases and concepts
for entity in doc.ents:
    print(entity.text, entity.label_)

NYT API

Offers free access: developer.nytimes.com/apis

follow instructions on developer.nytimes.com/get-started

• open an account on the NYT developer website
• create an app
• get an API key

Secret keys in google colab

Load key in colab

from google.colab import userdata
userdata.get('NYT_API_KEY')

NYT API - Practice

https://colab.research.google.com/drive/1PoFhONvZZxcpIG-_XMoN9wKTT1U7KS7X#scrollTo=aJVkXUfIFjqX

Goal :

• choose a topic, a set of articles
• build a dataset of articles
• extract entities, nouns, verbs, adjectives using spacy.io
• and also save the dataset on your laptop

Next time

• Modern NLP
• Embeddings
• RAG - context window

new data source: Andriy Burkov

aiweekly.substack.com/

Exit ticket

https://forms.gle/9eE9PUR6mFC2szq47