Meta Platforms, Inc.’s META AI chief Yann LeCun has proposed that Tesla and SpaceX CEO Elon Musk could leverage a new paper from Fundamenta AI Research or FAIR, a team at Meta, to enhance xAI’s Grok.
What Happened: On Wednesday, LeCun took to X, formerly Twitter, and shared a thread by Jason Weston a researcher at Meta AI, explaining the new FAIR paper titled “CoPE: Contextual Positional Encoding.”
LeCun said, “A new paper from FAIR that @elonmusk could use to improve Grok.”
In his thread, Weston explained that CoPE is a new positional encoding method for transformers that considers context. It can “count” distances per head based on need, such as the “i-th sentence or paragraph, words, verbs, etc. Not just tokens.” CoPE can also solve counting and copy tasks that standard transformers cannot.
Weston went on to say that CoPE can solve artificial counting, selective copy, and “Flip-Flop” tasks where popular position encodings fail. It performs well both in-distribution and out-of-distribution, for larger or smaller contexts compared to training.
He concluded by saying, “CoPE performs very well on standard language and code modeling tasks, giving improved PPL. Future work: how will it perform if we scale it up to SOTA LLMs? Going to need some GPUs.”
Subscribe to the Benzinga Tech Trends newsletter to get all the latest tech developments delivered to your inbox.
Why It Matters: This comment by LeCun came amidst his ongoing feud with Musk. While the two tech titans have a history of negative public interactions, the latest disagreement started on Monday after Musk invited people to apply for positions at xAI.
At the time, LeCun said that people should only join xAI if they wanted a boss “who claims that what you are working on will be solved next year (no pressure.”
Later, when asked about his issues with Musk, LeCun praised the tech billionaire’s innovation in the field of cars, rockets, solar panels, and satellite networks but strongly criticized his politics, conspiracy theories, and excessive hype.
Meanwhile, earlier this week, Musk’s xAI secured an impressive $6 billion in Series B funding, raising its valuation to $24 billion and making it one of the world’s most valuable AI startups.
Currently, xAI’s sole product is a chatbot named Grok, which was trained using data from X, including posts by Musk himself.
Check out more of Benzinga's Consumer Tech coverage by following this link.
Read Next: Elon Musk Shares Hack To Improve Google Search Quality: ‘It’s A Real Problem!’
Disclaimer: This content was partially produced with the help of Benzinga Neuro and was reviewed and published by Benzinga editors.
© 2024 Benzinga.com. Benzinga does not provide investment advice. All rights reserved.
Trade confidently with insights and alerts from analyst ratings, free reports and breaking news that affects the stocks you care about.
