A leading figure at OpenAI, the company behind groundbreaking AI like ChatGPT, has issued a stark warning: superintelligence, potentially arriving this decade, could pose an existential threat to humanity. Ilya Sutskever, OpenAI's co-founder and Chief Scientist, along with Jan Leike, head of alignment, emphasized the urgent need to control this powerful technology.

In a recent blog post, they highlighted the double-edged sword of superintelligence. While it holds the promise of solving complex global issues, its immense power could also lead to humanity's disempowerment or even extinction. The core challenge, they argue, lies in ensuring these advanced AI systems align with human intentions, a concept known as 'superintelligence alignment.'

Current methods for aligning AI, such as reinforcement learning from human feedback, rely on human supervision. However, Sutskever and Leike acknowledge these techniques won't be effective with systems far surpassing human intelligence. They stress the necessity of "new scientific and technical breakthroughs" to steer and control potentially rogue superintelligent AI.

To tackle this challenge, OpenAI is dedicating a significant portion of its resources – 20% of its secured computing power – and a new team to a four-year research initiative. Their objective is to develop an automated alignment researcher operating at roughly human-level, leveraging vast computational resources to scale it and iteratively align superintelligence. This involves creating a scalable training method, validating the resulting model, and rigorously stress-testing the alignment pipeline. While acknowledging the ambitious nature of this undertaking, they express optimism about achieving a solution through focused effort.

This new team's focus extends beyond refining existing models like ChatGPT and mitigating current risks. Their primary goal is to address the machine learning challenges inherent in aligning superintelligent AI with human intent. OpenAI's endeavors in this area are supported by Microsoft.