Hedge Fund Titan Peter Brown Built A Large Language Model 35 Years Ago: Read What It Wrote

In a recent episode of “Goldman Sachs Exchanges: Great Investors,” Raj Mahajan, who is the firm's global head of Systematic Client Franchise, Global Banking and Markets, had a conversation with Peter Brown, CEO of the legendary quantitative trading firm Renaissance Technologies.

The trading firm, co-founded by Jim Simons, manages about $160 billion in assets.

Before getting into the world of finance, Brown was in the world of language technology. Prior to joining Renaissance Technologies in 1993, he worked at IBM IBM as a language technology expert, a journey that began with an innate curiosity about speech recognition in high school.

Discussing his early career, Brown recalled a high school fascination with the 4a transformer, wondering if it could be used to recognize speech. "You just take the speech data, transform it into the frequency domain, match it up against patterns for words, and presto magic," he explained.

Read Also: Fed Keeps Rates Steady At 5.5%, Dot Plot Indicates 1 More Hike In 2023

Brown’s fascination was reignited in college after a linguistics course and a chance discussion about a company named Dialogue Systems. His passion led him to graduate studies at Carnegie Mellon under the mentorship of Jeff Hinton, often regarded as the "godfather of AI."

Brown's transition from linguistics to broader fields like machine learning saw him at the forefront of building large language models.

His models, reminiscent of the modern ChatGPT, aimed to mimic human knowledge of grammar and semantics through raw data.

The earlier models, due to the technological limitations of their time, relied on significantly less data and computing power.

Describing an early example, Brown shared a generative text produced by the model 35 years ago:

“What do you mean? I don't know. Said the man. Is it? he asked said the man, they are not not to be good idea. The first time I was a good idea, she was a good idea. Certainly, I said, what's the matter may I be able to get the money? said the man. Scott was a good idea. Mrs. King, Nick said, I don't know what I mean. Take a look at the door, he was a good idea. I don't know what I mean, didn't you? He said.”

Brown humorously commented on the repetitive nature of the text but acknowledged the progress in the field over the past 35 years.

Those earlier experiments were not without their skeptics, either. Brown recalled a two-sentence review of their first paper on machine translation that was dismissive of their data-driven approach.

Connecting his past work with present tech, Brown mentioned he got into translation using an idea of Google Translate. Brown’s venture into translation models was initiated when they received data from the Canadian parliament in both French and English. By treating translation as a statistical process, they were aiming to estimate parameters solely from that data.

His team also used their language model to create a spelling corrector that considered context, a feature missing from some spelling correctors.

Brown said the IBM team was in awe as the system could translate random keystrokes into comprehensible English.

It’s evident that the Renaissance Technologies CEO's early endeavors left a lasting imprint on machine learning and AI.

Despite starting in a time when the idea of large language models was in its infancy, his foundational work paved the way for giants such as ChatGPT.

Read Next: Wow, ChatGPT Can Pick Stocks: Benzinga's AI Portfolio Now In 5th Place