GPT-3(Generative Pre-trained Transformer 3)

#GPT-3(Generative Pre-trained Transformer 3)은 2020년 출시된 자동회귀언어모델로 딥러닝을 사용하여 사람과 유사한 텍스트를 생성합니다.

초기 텍스트를 프롬프트로 지정하면 프롬프트를 계속하는 텍스트를 생성합니다.

Generative Pre-trained Transformer 3 (GPT-3) is an autoregressive language model released in 2020 that uses deep learning to produce human-like text. Given an initial text as prompt, it will produce text that continues the prompt.

이 아키텍처는 2048개 토큰 길이 컨텍스트와 1,750억 개 매개변수라는 전례 없는 크기의 디코더전용변환기 네트워크로, 저장하는 데 800GB가 필요합니다.

모델은 생성사전훈련 generative pre-training을 사용하여 훈련되었습니다;

이전 토큰을 기반으로 다음 토큰이 무엇인지 예측하게 훈련됩니다.

이 모델은 많은 작업에서 강력한 zero-shot 및 few-shot 학습을 보여주었습니다.[2]

저자는 #자연어처리 (NLP)의 언어이해성능이 "레이블이 지정되지 않은 다양한 텍스트 corpus에 대한 언어모델의 생성사전훈련 후 각 특정항목에 대한 차별적 미세 조정" 프로세스를 통해 GPT-n에서 어떻게 개선되었는지 설명했습니다.

이것은 사람의 감독과 시간이 많이 걸리는 수동 라벨링의 필요성을 제거했습니다.[2]

The architecture is a decoder-only transformer network with a 2048-token-long context and then-unprecedented size of 175 billion parameters, requiring 800GB to store. The model was trained using generative pre-training; it is trained to predict what the next token is based on previous tokens. The model demonstrated strong zero-shot and few-shot learning on many tasks.[2] The authors described how language understanding performances in natural language processing (NLP) were improved in GPT-n through a process of "generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task." This eliminated the need for human supervision and for time-intensive hand-labeling.[2]

샌프란시스코 기반 인공지능연구소 OpenAI에서 만든 GPT-2의 후속 모델인 GPT 시리즈의 3세대 언어예측모델입니다.[3]

2020년 5월에 도입, 2020년 7월 현재 베타 테스트 중인 GPT-3[4]는 사전훈련된 언어표현의 자연어처리(NLP) 시스템 트렌드의 일부입니다.[1]

It is the third-generation language prediction model in the GPT series, successor to GPT-2 created by OpenAI, a San Francisco-based artificial intelligence research laboratory.[3] GPT-3, which was introduced in May 2020, and was in beta testing as of July 2020,[4] is part of a trend in natural language processing (NLP) systems of pre-trained language representations.[1]

GPT-3에 의해 생성된 텍스트품질은 매우 높아서 사람이 작성했는지 여부를 판단하기 어려울 수 있으며 이점과 위험이 모두 있습니다.[5]

31명 OpenAI 연구원과 엔지니어가 2020년 5월 28일 GPT-3를 소개하는 원본 논문을 발표했습니다.

그들 논문에서 그들은 GPT-3의 잠재적 위험에 대해 경고하고 위험을 완화하기 위한 연구를 촉구했습니다.[1]: 34

호주철학자 David Chalmers는 GPT-3를 "지금까지 생산된 가장 흥미롭고 중요한 AI 시스템 중 하나"라고 설명했습니다. "[6]

The New York Times 2022년 4월 리뷰에서는 GPT-3 기능이 인간과 동등한 유창함으로 독창적 산문을 작성할 수 있다고 설명했습니다.[7]

The quality of the text generated by GPT-3 is so high that it can be difficult to determine whether or not it was written by a human, which has both benefits and risks.[5] Thirty-one OpenAI researchers and engineers presented the original May 28, 2020 paper introducing GPT-3. In their paper, they warned of GPT-3's potential dangers and called for research to mitigate risk.[1]: 34 David Chalmers, an Australian philosopher, described GPT-3 as "one of the most interesting and important AI systems ever produced."[6] An April 2022 review in The New York Times described GPT-3's capabilities as being able to write original prose with fluency equivalent to that of a human.[7]

Microsoft는 2020년 9월 22일에 GPT-3의 "독점적" 사용을 허가받았다고 발표했습니다.

다른 사람들은 여전히 공개 API를 사용하여 출력을 받을 수 있지만 Microsoft만이 GPT-3 기본모델에 액세스할 수 있습니다.[8]

Microsoft announced on September 22, 2020, that it had licensed "exclusive" use of GPT-3; others can still use the public API to receive output, but only Microsoft has access to GPT-3's underlying model.[8]

Background[edit]

Further information: GPT-2 § Background

The Economist에 따르면 개선된 알고리즘, 강력한 컴퓨터 및 디지털화된 데이터 증가는 #기계학습 혁명을 불러일으켰으며 2010년대에는 새로운 기술이 #언어조작 등 "작업의 급속한 개선"을 가져왔습니다.[9]

소프트웨어 모델은 "뇌의 신경구조에 느슨하게 기반한 구조"에서 수천 또는 수백만 개의 예제를 사용하여 학습하도록 훈련됩니다.[9]

자연어처리(NLP)에 사용되는 아키텍처 중 하나는 2017년에 처음 소개된 #트랜스포머 [10]라는 #딥러닝 모델을 기반으로 하는 신경망입니다.

GPT-n 모델은 Transformer 기반 딥러닝 신경망 아키텍처입니다.

According to The Economist, improved algorithms, powerful computers, and an increase in digitized data have fueled a revolution in machine learning, with new techniques in the 2010s resulting in "rapid improvements in tasks" including manipulating language.[9] Software models are trained to learn by using thousands or millions of examples in a "structure ... loosely based on the neural architecture of the brain".[9] One architecture used in natural language processing (NLP) is a neural network based on a deep learning model that was first introduced in 2017—the Transformer.[10] GPT-n models are Transformer-based deep learning neural network architectures.

텍스트 입력을 처리, 마이닝, 구성, 연결, 대조하고, 질문에 올바르게 대답할 수 있는 NLP 시스템은 많이 있습니다.[11]

There are a number of NLP systems capable of processing, mining, organizing, connecting and contrasting textual input, as well as correctly answering questions.[11]

2018년 6월 11일, OpenAI 연구원과 엔지니어는 #생성사전훈련(GP)이라는 프로세스에서 데이터세트를 통해 방대하고 다양한 텍스트 코퍼스로 사전훈련될 수 있는 생성적 모델(언어모델, 인공지능시스템)에 대한 원본 논문을 게시했습니다.[2]

On June 11, 2018, OpenAI researchers and engineers posted their original paper on generative models—language models—artificial intelligence systems—that could be pre-trained with an enormous and diverse corpus of text via datasets, in a process they called generative pre-training (GP).[2]

저자는 자연어처리(NLP)의 언어이해성능이 "레이블이 지정되지 않은 다양한 텍스트 코퍼스에 대한 언어 모델의 생성사전훈련 후 각 특정항목에 대한 차별적 미세 조정" 프로세스를 통해 GPT-n에서 어떻게 개선되었는지 설명했습니다."

이것은 사람의 감독과 시간이 많이 걸리는 수동 라벨링의 필요성을 제거했습니다.[2]

The authors described how language understanding performances in natural language processing (NLP) were improved in GPT-n through a process of "generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task." This eliminated the need for human supervision and for time-intensive hand-labeling.[2]

2020년 2월 Microsoft는 T-NLG (Turing Natural Language Generation)를 도입했는데, 이는 "170억 개의 매개변수로 게시된 가장 큰 언어모델"이라고 주장했습니다.

텍스트 요약 및 질문 답변을 포함하는 다양한 작업에서 다른 언어 모델보다 더 나은 성능을 보였습니다.

In February 2020, Microsoft introduced its Turing Natural Language Generation (T-NLG), which was claimed to be the "largest language model ever published at 17 billion parameters."[12] It performed better than any other language model at a variety of tasks which included summarizing texts and answering questions.

Training and capabilities[edit]

GPT-3이 작성한 교육학에 대한 샘플 학생 에세이

A sample student essay about pedagogy written by GPT-3

"학습스타일"의 구성은 학습스타일이 형성되는 과정을 설명하지 못하기 때문에 문제가 있습니다.

어떤 학생들은 특정 경험을 했기 때문에 특정한 학습스타일을 개발할 수 있습니다.

다른 사람들은 자신의 학습요구에 적합하지 않은 학습환경에 적응하려고 노력함으로써 특정 학습 스타일을 개발할 수 있습니다.

궁극적으로 우리는 학습스타일과 환경 및 개인적 요인 사이의 상호작용을 이해하고 이러한 요소가 우리가 학습하는 방법과 우리가 경험하는 학습의 종류를 어떻게 형성하는지 이해해야 합니다.

The construct of “learning styles” is problematic because it fails to account for the processes through which learning styles are shaped. Some students might develop a particular learning style because they have had particular experiences. Others might develop a particular learning style by trying to accommodate to a learning environment that was not well suited to their learning needs. Ultimately, we need to understand the interactions among learning styles and environmental and personal factors, and how these shape how we learn and the kinds of learning we experience.

– Text generated by Mike Sharples[13]

2020년 5월 28일, OpenAI의 31명의 엔지니어 및 연구원그룹이 arXiv 프리프린트에서 3세대 "최첨단 언어모델" GPT-3 개발에 대해 설명했습니다.[1][5]

On May 28, 2020, an arXiv preprint by a group of 31 engineers and researchers at OpenAI described the development of GPT-3, a third-generation "state-of-the-art language model".[1][5]

팀은 GPT-3 용량을 이전 모델 GPT-2보다 2배 이상 증가시켜[14] GPT-3를 현재까지 가장 큰 희소하지 않은 언어모델로 만들었습니다. (희소모델에서는 많은 매개변수가 상수 값으로 설정되어 있기 때문에 총 매개변수가 더 많아도 의미있는 정보가 적음) [1]: 14 [3]

The team increased the capacity of GPT-3 by over two orders of magnitude from that of its predecessor, GPT-2,[14] making GPT-3 the largest non-sparse language model to date. (In a sparse model, many of its parameters are set to a constant value, so even if there are more total parameters, there is less meaningful information.)[1]: 14 [3]

GPT-3는 구조적으로 전임자[1] 더 큰 정확도는 증가된 용량과 더 많은 매개변수 수에 기인합니다.[15]

GPT-3 용량은 당시 알려진 차세대 NLP 모델인 Microsoft의 Turing NLG보다 10배 더 큽니다.[5]

Because GPT-3 is structurally similar to its predecessors,[1] its greater accuracy is attributed to its increased capacity and greater number of parameters.[15] GPT-3's capacity is ten times larger than that of Microsoft's Turing NLG, the next largest NLP model known at the time.[5]

GPT-3에 대한 가중 사전교육 데이터세트의 60%는 4,100억 바이트 쌍으로 인코딩된 토큰으로 구성된 Common Crawl의 필터링된 버전에서 가져옵니다.[1]: 9

다른 소스는 WebText2의 190억 토큰으로, 가중총계, Books1의 120억 토큰이 8%, Books2의 550억 토큰이 8%, Wikipedia의 30억 토큰이 3%를 나타냅니다.[1]: 9

GPT-3는 수천억 개 단어에 대해 훈련되었으며 또한 CSS, JSX, Python 등으로 코딩할 수 있습니다.[4]

Sixty percent of the weighted pre-training dataset for GPT-3 comes from a filtered version of Common Crawl consisting of 410 billion byte-pair-encoded tokens.[1]: 9 Other sources are 19 billion tokens from WebText2 representing 22% of the weighted total, 12 billion tokens from Books1 representing 8%, 55 billion tokens from Books2 representing 8%, and 3 billion tokens from Wikipedia representing 3%.[1]: 9 GPT-3 was trained on hundreds of billions of words and is also capable of coding in CSS, JSX, and Python, among others.[4]

GPT-3 training data

Dataset	# tokens	Proportion within training
Common Crawl	410 billion	60%
WebText2	19 billion	22%
Books1	12 billion	8%
Books2	55 billion	8%
Wikipedia	3 billion	3%

GPT-3 훈련데이터는 모든 것을 포괄하므로, 별도의 언어작업에 대한 추가훈련이 필요하지 않습니다.[4]

Since GPT-3's training data was all-encompassing, it does not require further training for distinct language tasks.[4]

훈련데이터에는 때때로 독성 toxic 언어가 포함되어 있으며, GPT-3는 훈련데이터를 모방한 결과 때때로 독성 언어를 생성합니다.

워싱턴대학 연구에 따르면 GPT-3는 GPT-2 및 CTRL의 유사한 자연어 처리모델과 비슷한 독성 수준의 독성 언어를 생성하는 것으로 나타났습니다.

The training data contains occasional toxic language and GPT-3 occasionally generates toxic language as a result of mimicking its training data. A study from the University of Washington found that GPT-3 produced toxic language at a toxicity level comparable to the similar natural language processing models of GPT-2 and CTRL.

OpenAI는 GPT-3에서 생성되는 독성 언어의 양을 제한하기 위해 여러 가지 전략을 구현했습니다.[16]

결과적으로 GPT-3는 이전 모델인 GPT-1에 비해 덜 독성이 있는 언어를 생성했지만, 전적으로 Wikipedia 데이터에 대해 훈련된 언어 모델인 CTRL Wiki에 비해 더 많은 세대와 더 독성이 높은 언어를 생성했습니다.[17]

OpenAI has implemented several strategies to limit the amount of toxic language generated by GPT-3.[16] As a result, GPT-3 produced less toxic language compared to its predecessor model, GPT-1, although it produced both more generations and a higher toxicity of toxic language compared to CTRL Wiki, a language model trained entirely on Wikipedia data.[17]

2020년 6월 11일, OpenAI는 OpenAI가 이 새로운 기술의 "강점과 한계를 탐색"하는 데 도움이 되는 사용자친화적 GPT-3 API("기계학습도구세트")에 대한 액세스를 요청할 수 있다고 발표했습니다.[18][19 ]

초청장은 이 API가 어떻게 일반적 단일 사용사례 대신 거의 "모든 영어작업"을 완료할 수 있는 범용 "텍스트 입력, 텍스트 출력" 인터페이스를 가지고 있는지 설명했습니다.[18]

OpenAI GPT-3 API의 비공개 초기 릴리스에 액세스할 수 있었던 한 사용자에 따르면 GPT-3는 몇 가지 간단한 프롬프트만으로 "놀라울 정도로 일관성 있는 텍스트"를 작성하는 데 "엄청나게 뛰어났습니다".[20]

초기 실험에서 80명 미국 피험자들에게 ~200단어의 짧은 기사가 인간에 의해 쓰여졌는지 또는 GPT-3에 의해 쓰여졌는지 판단하도록 요청받았습니다.

참가자들은 시간의 52%를 정확하게 판단했으며 무작위 추측보다 약간 더 나을 뿐입니다.[1]

On June 11, 2020, OpenAI announced that users could request access to its user-friendly GPT-3 API—a "machine learning toolset"—to help OpenAI "explore the strengths and limits" of this new technology.[18][19] The invitation described how this API had a general-purpose "text in, text out" interface that can complete almost "any English language task", instead of the usual single use-case.[18] According to one user, who had access to a private early release of the OpenAI GPT-3 API, GPT-3 was "eerily good" at writing "amazingly coherent text" with only a few simple prompts.[20] In an initial experiment 80 US subjects were asked to judge if short ~200 word articles were written by humans or GPT-3. The participants judged correctly 52% of the time, doing only slightly better than random guessing.[1]

2021년 11월 18일, OpenAI는 API에 대한 액세스가 제한되지 않도록 충분한 안전장치가 구현되었다고 발표했습니다.[21]

OpenAI는 개발자가 OpenAI의 콘텐츠정책을 준수하는 데 도움이 되는 콘텐츠 조정도구를 제공했습니다.[22]

2022년 1월 27일, OpenAI는 총칭하여 InstructGPT라고 하는 최신 GPT-3 언어모델이 이제 API에서 사용되는 기본 언어모델이라고 발표했습니다.

OpenAI에 따르면 InstructGPT는 지침을 더 잘 따르고, 더 적은 구성 사실을 생성하고, 다소 덜 유해한 콘텐츠를 생성하여 사용자 의도에 더 잘 부합하는 콘텐츠를 생성했습니다.[23]

On November 18, 2021, OpenAI announced that enough safeguards had been implemented that access to its API would be unrestricted.[21] OpenAI provided developers with a content moderation tool that helps them abide by OpenAI's content policy.[22] On January 27, 2022, OpenAI announced that its newest GPT-3 language models, collectively referred to as InstructGPT, was now the default language model used on their API. According to OpenAI, InstructGPT produced content that was better aligned to user intentions by following instructions better, generating fewer made-up facts, and producing somewhat less toxic content.[23]

GPT-3는 "인간평가자가 인간이 작성한 기사와 구별하기 어려운 뉴스 기사를 생성"할 수 있기 때문에 GPT-3는 "언어모델의 유익한 적용과 유해한 적용을 모두 발전시킬 수 있는 잠재력"이 있습니다.[1]: 34

2020년 5월 28일자 논문에서 연구원들은 "잘못된 정보, 스팸, 피싱, 법적 및 정부 프로세스의 남용, 사기성 학술에세이 작성 및 사회공학적 프리텍스팅을 포함하는" "GPT-3의 잠재적인 유해한 영향"[5]을 자세히 설명했습니다".[1]

저자는 위험완화에 대한 연구를 요구하기 위해 이러한 위험에 주의를 기울입니다.[1][24]: 34

Because GPT-3 can "generate news articles which human evaluators have difficulty distinguishing from articles written by humans,"[5] GPT-3 has the "potential to advance both the beneficial and harmful applications of language models."[1]: 34 In their May 28, 2020 paper, the researchers described in detail the potential "harmful effects of GPT-3"[5] which include "misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting".[1] The authors draw attention to these dangers to call for research on risk mitigation.[1][24]: 34

GPT-3는 제로샷, 퓨샷 및 원샷 학습을 수행할 수 있습니다.[1]

2022년 6월, Almira Osmanovic Thunström은 GPT-3가 자신에 대한 기사의 주요 저자이며 출판을 위해 제출했으며[25] 검토 완료를 기다리는 동안 사전 출판되었다고 썼습니다.[25] 26]

GPT-3 is capable of performing zero-shot, few-shot and one-shot learning.[1]

In June 2022, Almira Osmanovic Thunström wrote that GPT-3 was the primary author on an article on itself, that they had submitted it for publication,[25] and that it had been pre-published while waiting for completion of its review.[26]

GPT-3.5[edit]

2022년 3월 15일, OpenAI는 "text-davinci-003" 및 "code-davinci-002"라는 이름으로 편집 및 삽입 기능 있는 API에서 GPT-3 및 Codex의 새 버전을 사용할 수 있게 했습니다.[27]

이러한 모델은 이전 버전보다 더 뛰어난 것으로 설명되었으며 2021년 6월까지의 데이터에 대해 교육을 받았습니다.[28]

2022년 11월 30일, OpenAI는 이러한 모델을 "GPT-3.5" 시리즈에 속하는 것으로 언급하기 시작했고[29] GPT-3.5 시리즈의 모델에서 미세 조정된 ChatGPT를 출시했습니다.[30]

On March 15, 2022, OpenAI made available new versions of GPT-3 and Codex in its API with edit and insert capabilities under the names "text-davinci-003" and "code-davinci-002".[27] These models were described as more capable than previous versions and were trained on data up to June 2021.[28] On November 30, 2022, OpenAI began referring to these models as belonging to the "GPT-3.5" series,[29] and released ChatGPT, which was fine-tuned from a model in the GPT-3.5 series.[30]

건강전문 몰

www.dopza.com

돕자몰

www.dopza.com

Reception[edit]

Applications[edit]

GPT-3, specifically the Codex model, is the basis for GitHub Copilot, a code completion and generation software that can be used in various code editors and IDEs.[31][32]
GPT-3 is used in certain Microsoft products to translate conventional language into formal computer code.[33][34]
GPT-3 has been used in CodexDB[35] to generate query-specific code for SQL processing.
GPT-3 has been used by Jason Rohrer in a retro-themed chatbot project named "Project December", which is accessible online and allows users to converse with several AIs using GPT-3 technology.[36]
GPT-3 was used by The Guardian to write an article about AI being harmless to human beings. It was fed some ideas and produced eight different essays, which were ultimately merged into one article.[37]
GPT-3 was used in AI Dungeon, which generates text-based adventure games. Later it was replaced by a competing model after OpenAI changed their policy regarding generated content.[38][39]
GPT-3 is used in Copy.ai, an AI copywriting app for marketers and business owners.[40]
GPT-3 is used in Jasper.ai, a content generator designed to assist marketers and copyeditors.[41][42]
GPT-3 is used in Hypotenuse AI, a content creation app, and is combined with their own proprietary technology for writing factual content for marketers and businesses.[43]
A 2022 Drexel University study suggested that systems based on GPT-3 could be used to screen for early signs of Alzheimer's disease.[44][45]

Reviews[edit]

In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to generate computer code, poetry, and prose is not just "amazing", "spooky", and "humbling", but also "more than a little terrifying".[46]
Daily Nous presented a series of articles by nine philosophers on GPT-3.[47] Australian philosopher David Chalmers described GPT-3 as "one of the most interesting and important AI systems ever produced".[6]
A review in Wired said that GPT-3 was "provoking chills across Silicon Valley".[48]
The National Law Review said that GPT-3 is an "impressive step in the larger process", with OpenAI and others finding "useful applications for all of this power" while continuing to "work toward a more general intelligence".[49]
An article in the MIT Technology Review, cowritten by Deep Learning critic Gary Marcus,[50] stated that GPT-3's "comprehension of the world is often seriously off, which means you can never really trust what it says."[51] According to the authors, GPT-3 models relationships between words without having an understanding of the meaning behind each word.
Jerome Pesenti, head of the Facebook AI lab, said GPT-3 is "unsafe," pointing to the sexist, racist and other biased and negative language generated by the system when it was asked to discuss Jews, women, black people, and the Holocaust.[52]
Nabla, a French start-up specializing in healthcare technology, tested GPT-3 as a medical chatbot, though OpenAI itself warned against such use. As expected, GPT-3 showed several limitations. For example, while testing GPT-3 responses about mental health issues, the AI advised a simulated patient to commit suicide.[53]
Noam Chomsky expressed his skepticism about GPT-3's scientific value: "It's not a language model. It works just as well for impossible languages as for actual languages. It is therefore refuted, if intended as a language model, by normal scientific criteria. [...] Perhaps it's useful for some purpose, but it seems to tell us nothing about language or cognition generally."[54]
Luciano Floridi and Massimo Chiriatti highlighted the risk of "cheap production of good, semantic artefacts".[55]
OpenAI's Sam Altman himself criticized what he called "GPT-3 hype", acknowledging GPT-3 "has serious weakness and sometimes makes very silly mistakes... AI is going to change the world, but GPT-3 is just a very early glimpse."[56]

Criticism[edit]

GPT-3 빌더 OpenAI는 처음에 2015년에 비영리단체로 설립되었습니다.[57]

2019년에 OpenAI는 모델이 가짜 뉴스를 영속시킬 것이라는 우려를 이유로 OpenAI의 이전 오픈 소스 관행을 깨고 GPT-3의 전구체 모델을 공개적으로 공개하지 않았습니다.

OpenAI는 결국 원래 모델 크기의 8%인 GPT-2 버전을 출시했습니다.[58]

같은 해에 OpenAI는 영리 회사로 구조 조정되었습니다.[59]

2020년에 Microsoft는 OpenAI에 수십억 달러를 투자한 후 Microsoft의 제품 및 서비스에 대한 GPT-3의 독점 라이선스를 보유했다고 발표했습니다.

이 계약은 OpenAI가 공개 API를 제공하도록 허용하여 사용자가 GPT-3에 텍스트를 보내 모델의 출력을 수신할 수 있도록 하지만 Microsoft만이 GPT-3의 소스 코드에 액세스할 수 있습니다.[8]

GPT-3's builder, OpenAI, was initially founded as a non-profit in 2015.[57] In 2019, OpenAI did not publicly release GPT-3's precursor model, breaking from OpenAI's previous open-source practices, citing concerns that the model would perpetuate fake news. OpenAI eventually released a version of GPT-2 that was 8% of the original model's size.[58] In the same year, OpenAI restructured to be a for-profit company.[59] In 2020, Microsoft announced the company had exclusive licensing of GPT-3 for Microsoft's products and services following a multi-billion dollar investment in OpenAI. The agreement permits OpenAI to offer a public-facing API such that users can send text to GPT-3 to receive the model's output, but only Microsoft will have access to GPT-3's source code.[8]

GPT-3 같은 대형 언어모델은 2021년 Timnit Gebru 및 Emily M. Bender가 공동집필한 논문에 자세히 설명된 모델교육 및 저장이 환경에 미치는 영향에 대해 Google의 AI 윤리연구원 몇 명으로부터 비판을 받았습니다.[60]

GPT-3 및 기타 언어생성기를 기반으로 하는 자동화된 쓰기 기술의 사용 증가[언제?]는 학문적 무결성에 대한 우려를 제기했으며[61] 대학과 학교가 표절 같은 학문적 위법행위를 구성하는 요소를 측정하는 방법에 대한 이해 관계를 높였습니다.[ 62]

Large language models, such as GPT-3, have come under criticism from a few of Google's AI ethics researchers for the environmental impact of training and storing the models, detailed in a paper co-authored by Timnit Gebru and Emily M. Bender in 2021.[60]

The growing[when?] use of automated writing technologies based on GPT-3 and other language generators, has raised concerns regarding academic integrity[61] and raised the stakes of how universities and schools will gauge what constitutes academic misconduct such as plagiarism.[62]

GPT was built with data from the Common Crawl dataset, a conglomerate of copyrighted articles, internet posts, web pages, and books scraped from 60 million domains over a period of 12 years. TechCrunch reports this training data includes copyrighted material from the BBC, The New York Times, Reddit, the full text of online books, and more.[63]

2019년 USPTO(미국특허청)의 인공지능 혁신을 위한 지적재산권 보호에 대한 의견요청에 대한 응답에서 OpenAI는 "현재 법에 따라 [GPT 모델 같은] AI 시스템을 교육하는 것은 공정 사용에 해당하나", "적절한 판례법이 없기 때문에 OpenAI 및 우리 같은 다른 AI 개발자는 상당한 법적 불확실성과 규정준수 비용에 직면합니다."[64]

In its response to a 2019 Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation from the United States Patent and Trademark Office (USPTO), OpenAI argued that "Under current law, training AI systems [such as its GPT models] constitutes fair use," but that "given the lack of case law on point, OpenAI and other AI developers like us face substantial legal uncertainty and compliance costs."[64]

See also[edit]

References[edit]

^ Jump up to:a b c d e f g h i j k l m Brown, Tom B.; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini; Herbert-Voss, Ariel; Krueger, Gretchen; Henighan, Tom; Child, Rewon; Ramesh, Aditya; Ziegler, Daniel M.; Wu, Jeffrey; Winter, Clemens; Hesse, Christopher; Chen, Mark; Sigler, Eric; Litwin, Mateusz; Gray, Scott; Chess, Benjamin; Clark, Jack; Berner, Christopher; McCandlish, Sam; Radford, Alec; Sutskever, Ilya; Amodei, Dario (December 2020). Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.F.; Lin, H. (eds.). "Language Models are Few-Shot Learners" (PDF). Advances in Neural Information Processing Systems. Curran Associates, Inc. 33: 1877–1901.
^ Jump up to:a b c d Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (June 11, 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). p. 12. Retrieved July 31, 2020.
^ Jump up to:a b Shead, Sam (July 23, 2020). "Why everyone is talking about the A.I. text generator released by an Elon Musk-backed lab". CNBC. Retrieved July 31, 2020. Four preprints were released between May 28 and July 22, 2020.
^ Jump up to:a b c Bussler, Frederik (July 21, 2020). "Will GPT-3 Kill Coding?". Towards Data Science. Retrieved August 1, 2020.
^ Jump up to:a b c d e Sagar, Ram (June 3, 2020). "OpenAI Releases GPT-3, The Largest Model So Far". Analytics India Magazine. Retrieved July 31, 2020.
^ Jump up to:a b Chalmers, David (July 30, 2020). Weinberg, Justin (ed.). "GPT-3 and General Intelligence". Daily Nous. Philosophers On GPT-3 (updated with replies by GPT-3). Retrieved August 4, 2020.
^ Johnson, Steven; Iziev, Nikita (April 15, 2022). "A.I. Is Mastering Language. Should We Trust What It Says?". The New York Times.
^ Jump up to:a b Hao, Karen (September 23, 2020). "OpenAI is giving Microsoft exclusive access to its GPT-3 language model". MIT Technology Review. Retrieved September 25, 2020. The companies say OpenAI will continue to offer its public-facing API, which allows chosen users to send text to GPT-3 or OpenAI's other models and receive its output. Only Microsoft, however, will have access to GPT-3's underlying code, allowing it to embed, repurpose, and modify the model as it pleases.
^ Jump up to:a b "An understanding of AI's limitations is starting to sink in". The Economist. June 11, 2020. ISSN 0013-0613. Retrieved July 31, 2020.
^ Polosukhin, Illia; Kaiser, Lukasz; Gomez, Aidan N.; Jones, Llion; Uszkoreit, Jakob; Parmar, Niki; Shazeer, Noam; Vaswani, Ashish (June 12, 2017). "Attention Is All You Need". arXiv:1706.03762 [cs.CL].
^ "Natural Language Processing". Retrieved July 31, 2020.
^ Sterling, Bruce (February 13, 2020). "Web Semantics: Microsoft Project Turing introduces Turing Natural Language Generation (T-NLG)". Wired. ISSN 1059-1028. Retrieved July 31, 2020.
^ Marche, Stephen (December 6, 2022). "The College Essay Is Dead". The Atlantic. Retrieved December 8, 2022.
^ "Language Models are Unsupervised Multitask Learners" (PDF). openai.com. Retrieved December 4, 2019. GPT-2, is a 1.5B parameter Transformer
^ Ray, Tiernan (June 1, 2020). "OpenAI's gigantic GPT-3 hints at the limits of language models for AI". ZDNet. Retrieved July 31, 2020.
^ "OpenAI's Initiative to Curb the Spread of Toxic Language by Prompt Engineers Using GPT-3". Prompt Engineering Wiki. January 31, 2023.
^ Gehman, Samuel; Gururangan, Suchin; Sap, Maarten; Choi, Yejin; Smith, Noah A. (November 16–20, 2020), REALTOXICITYPROMPTS: Evaluating Neural Toxic Degeneration in Language Models, Association for Computational Linguistics, pp. 3356–3369, arXiv:2009.11462, retrieved June 2, 2021
^ Jump up to:a b "OpenAI API". OpenAI. June 11, 2020.
^ Coldewey, Devin (June 11, 2020). "OpenAI makes an all-purpose API for its text-based AI capabilities". TechCrunch. Archived from the original on October 27, 2021. Retrieved July 31, 2020. If you've ever wanted to try out OpenAI's vaunted machine learning toolset, it just got a lot easier. The company has released an API that lets developers call its AI tools in on "virtually any English language task."
^ Arram (July 9, 2020). "GPT-3: An AI that's eerily good at writing almost anything". Arram Sabeti. Retrieved July 31, 2020.
^ "OpenAI's API Now Available with No Waitlist". OpenAI. November 18, 2021. Retrieved November 5, 2022.
^ "OpenAI API". beta.openai.com. Retrieved November 5, 2022.
^ "Aligning Language Models to Follow Instructions". OpenAI. January 27, 2022. Retrieved November 5, 2022.
^ Brown, Tom B.; et al. (2020). "Language Models are Few-Shot Learners". arXiv:2005.14165 [cs.CL].
^ Thunström, Almira Osmanovic (June 30, 2022). "We Asked GPT-3 to Write an Academic Paper about Itself—Then We Tried to Get It Published". Scientific American. Retrieved June 30, 2022.
^ Transformer, Gpt Generative Pretrained; Thunström, Almira Osmanovic; Steingrimsson, Steinn (June 21, 2022). "Can GPT-3 write an academic paper on itself, with minimal human input?". Archive ouverte HAL (in French). Retrieved June 30, 2022.
^ "New GPT-3 Capabilities: Edit & Insert". OpenAI. March 15, 2022. Retrieved January 13, 2023.
^ "Models".
^ "Model index for researchers".
^ "ChatGPT: Optimizing Language Models for Dialogue". OpenAI. November 30, 2022. Retrieved January 13, 2023.
^ "OpenAI Codex". OpenAI. August 10, 2021. Retrieved December 23, 2022.
^ Thompson, Clive (March 15, 2022). "How an AI Became My Code-Writing Genie". Wired. Retrieved December 23, 2022.
^ "Microsoft announced its first customer product features powered by GPT-3 and @Azure". The AI Blog. May 25, 2021.
^ Vincent, James (May 25, 2021). "Microsoft has built an AI-powered autocomplete for code using GPT-3". The Verge. Retrieved December 23, 2022.
^ "CodexDB - SQL Processing Powered by GPT-3". CodexDB - SQL Processing Powered by GPT-3.
^ Fagone, Jason (July 23, 2021). "The Jessica Simulation: Love and loss in the age of A.I." San Francisco Chronicle. Retrieved July 29, 2021.
^ GPT-3 (September 8, 2020). "A robot wrote this entire article. Are you scared yet, human? | GPT-3". The Guardian. ISSN 0261-3077. Retrieved September 15, 2020.
^ "Update: Language Models and Dragon". Latitude blog. December 8, 2021.
^ "This Mystical Book Was Co-Authored by a Disturbingly Realistic AI". www.vice.com. 2022. Retrieved December 23, 2022.
^ "Writing helper Copy.ai raises $2.9M in a round led by Craft Ventures". copy.ai. March 17, 2021. Retrieved November 5, 2022.
^ "Jasper Announces $125M Series A Funding Round, Bringing Total Valuation to $1.5B and Launches New Browser Extension". www.jasper.ai. Retrieved November 5, 2022.
^ Dzieza, Josh (July 20, 2022). "Can AI write good novels?". The Verge. Retrieved December 23, 2022.
^ Lomas, Natasha (August 7, 2020). "Hypotenuse AI wants to take the strain out of copywriting for e-commerce". TechCrunch. Retrieved January 5, 2023.
^ "Can ChatGPT AI chatbot spot early stages of Alzheimer's? - study". The Jerusalem Post. 2022. Retrieved December 30, 2022.
^ Agbavor, Felix; Liang, Hualou (December 22, 2022). "Predicting dementia from spontaneous speech using large language models". PLOS Digital Health. 1 (12): e0000168. doi:10.1371/journal.pdig.0000168. S2CID 255029590.
^ Manjoo, Farhad (July 29, 2020). "How Do You Know a Human Wrote This?". The New York Times. ISSN 0362-4331. Retrieved August 4, 2020.
^ Weinberg, Justin, ed. (July 30, 2020). "Philosophers On GPT-3 (updated with replies by GPT-3)". Daily Nous. Retrieved July 31, 2020.
^ Simonite, Tom (July 22, 2020). "Did a Person Write This Headline, or a Machine?". Wired. ISSN 1059-1028. Retrieved July 31, 2020.
^ Claypoole, Theodore (July 30, 2020). "New AI Tool GPT-3 Ascends to New Peaks, But Proves How Far We Still Need to Travel". The National Law Review. Retrieved August 4, 2020.
^ Marcus, Gary (December 1, 2018). "The deepest problem with deep learning". Medium. Retrieved September 29, 2020.
^ Marcus, Gary; Davis, Ernest (August 22, 2020). "GPT-3, Bloviator: OpenAI's language generator has no idea what it's talking about". MIT Technology Review. Retrieved August 23, 2020.
^ Metz, Cade (November 24, 2020). "Meet GPT-3. It Has Learned to Code (and Blog and Argue)". The New York Times. ISSN 0362-4331. Retrieved November 24, 2020.
^ "Medical chatbot using OpenAI's GPT-3 told a fake patient to kill themselves". AI News. October 28, 2020. Retrieved January 8, 2021.
^ Chomsky on Terence McKenna, Sam Harris, GPT3, Cryptocurrencies, Kierkegaard, Neuralink, & Hofstadter. March 24, 2021. Event occurs at 1:11:44.
^ Floridi, Luciano; Chiriatti, Massimo (November 1, 2020). "GPT‑3: Its Nature, Scope, Limits, and Consequences". Minds and Machines. 30 (4): 681–694. doi:10.1007/s11023-020-09548-1. S2CID 228954221.
^ Vincent, James (July 30, 2020). "OpenAI's latest breakthrough is astonishingly powerful, but still fighting its flaws". The Verge. Retrieved November 9, 2022.
^ Olanoff, Drew (December 11, 2015). "Artificial Intelligence Nonprofit OpenAI Launches With Backing From Elon Musk And Sam Altman". Tech Crunch. Retrieved May 31, 2021.
^ Hao, Karen (August 29, 2019). "OpenAI has released the largest version yet of its fake-news-spewing AI". MIT Technology Review. Retrieved May 31, 2021.
^ Coldewey, Devin (March 11, 2019). "OpenAI shifts from nonprofit to 'capped-profit' to attract capital". Tech Crunch. Retrieved May 31, 2021.
^ Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Shmitchell, Shmargaret (March 3, 2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. pp. 610–623. doi:10.1145/3442188.3445922.
^ Mindzak, Michael; Eaton, Sarah Elaine. "Artificial intelligence is getting better at writing, and universities should worry about plagiarism". The Conversation. Retrieved November 6, 2021.
^ Rogerson, Ann M.; McCarthy, Grace (December 2017). "Using Internet based paraphrasing tools: Original work, patchwriting or facilitated plagiarism?". International Journal for Educational Integrity. 13 (1): 1–15. doi:10.1007/s40979-016-0013-y. ISSN 1833-2595. S2CID 9473217.
^ Here are a few ways GPT-3 can go wrong. TechCrunch.
^ Comment Regarding Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation (PDF). USPTO.

show

Differentiable computing

show

Existential risk from artificial intelligence

Categories: Language modelingDeep learning software applicationsUnsupervised learningOpenAI

This page was last edited on 1 February 2023, at 14:10 (UTC).
Text is available under the Creative Commons Attribution-ShareAlike License 3.0; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.

저작자 명시 필수 영리적 사용 불가 내용 변경 불가

태그#GPT #자연어처리 #기계학습 #언어조작 #트랜스포머 #딥러닝 #생성사전훈련

머신러닝 GPT

GPT-3(Generative Pre-trained Transformer 3)

댓글

티스토리툴바