«   2022/07   »
์ผ ์›” ํ™” ์ˆ˜ ๋ชฉ ๊ธˆ ํ† 
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31            
Today
703
Yesterday
1,170

Total
140,151
07-05 15:21
๊ด€๋ฆฌ ๋ฉ”๋‰ด

Hey Tech

[ํŒŒ์ด์ฌ/NLP] ๋น ๋ฅด๊ฒŒ ํ•œ๊ตญ์–ด ํ˜•ํƒœ์†Œ ๋ถ„์„ํ•˜๋Š” ๋ฐฉ๋ฒ• (feat. KoNLPy) ๋ณธ๋ฌธ

AI & ๋น…๋ฐ์ดํ„ฐ/์ž์—ฐ์–ด์ฒ˜๋ฆฌ(NLP)

[ํŒŒ์ด์ฌ/NLP] ๋น ๋ฅด๊ฒŒ ํ•œ๊ตญ์–ด ํ˜•ํƒœ์†Œ ๋ถ„์„ํ•˜๋Š” ๋ฐฉ๋ฒ• (feat. KoNLPy)

Tony Park 2021. 1. 18. 19:33
728x90
๋ฐ˜์‘ํ˜•

๐Ÿ“š ๋ชฉ์ฐจ

1. KoNLPy setup
    1.1. KoNLPy ์„ค์น˜
    1.2. KoNLPy import
    1.3. ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ ๋น„๊ต
2. ํ˜•ํƒœ์†Œ๋ณ„ ํ† ํฐํ™”(tokenization)ํ•˜๊ธฐ
    2.1. ํ† ํฐํ™” without ํ’ˆ์‚ฌ ํƒœ๊น…
    2.2. ํ† ํฐํ™” with ํ’ˆ์‚ฌ ํƒœ๊น…
    2.3. ๋ช…์‚ฌ๋งŒ ์ถ”์ถœํ•˜๊ธฐ

1. KoNLPy setup

1.1. KoNLPy ์„ค์น˜

KoNLPy ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๊ฐ€ ์„ค์น˜๋˜์–ด ์žˆ์ง€ ์•Š๋Š” ๋ถ„๋“ค์€ ์•„๋ž˜ ๋งํฌ๋ฅผ ์ฐธ๊ณ ํ•˜์…”์„œ ์„ค์น˜ํ•ด ์ฃผ์‹œ๊ธธ ๋ฐ”๋ž๋‹ˆ๋‹ค.
heytech.tistory.com/3

 

[Python/NLP] MacOS์—์„œ KoNLPy ์„ค์น˜ํ•˜๊ธฐ

์˜ค๋Š˜์€ ํ•œ๊ตญ์–ด ์ž์—ฐ์–ด์ฒ˜๋ฆฌ(NLP)๋ฅผ ์œ„ํ•œ ํŒŒ์ด์ฌ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ KoNLPy๋ฅผ MacOS์—์„œ ์„ค์น˜ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๊ณต์œ ํ•ฉ๋‹ˆ๋‹ค. 1. ์‚ฌ์ „ ์„ค์น˜ํ•ญ๋ชฉ 1.1. MacOS/Linux ํŒจํ‚ค์ง€ ๋งค๋‹ˆ์ € https://brew.sh/index_ko Homebrew The Missing P..

heytech.tistory.com

1.2. KoNLPy import

from konlpy.tag import Mecab
mecab = Mecab()

KonNLPy ์„ค์น˜๊ฐ€ ์™„๋ฃŒ๋˜์—ˆ๋‹ค๋ฉด ์ด์ œ ํŒŒ์ด์ฌ์—์„œ KoNLPy ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ import ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฒˆ ํฌ์ŠคํŒ…์—์„œ๋Š” KoNLPy์—์„œ ์ œ๊ฐ€ ์ฃผ๋กœ ์‚ฌ์šฉ์€ Mecab ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ๋ฅผ ์‚ฌ์šฉํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ๋งŒ์•ฝ Mecab ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ๊ฐ€ ์ •์ƒ์ ์œผ๋กœ ๋™์ž‘ํ•˜์ง€ ์•Š๋Š”๋‹ค๋ฉด, ์•„๋ž˜ ํฌ์ŠคํŒ…์„ ์ฐธ๊ณ ํ•ด ์ฃผ์„ธ์š”!

https://heytech.tistory.com/395

 

Mecab ์„ค์น˜ ์—๋Ÿฌ ํ•ด๊ฒฐํ•˜๊ธฐ: Exception: Install MeCab in order to use it: http://konlpy.org/en/latest/install/

๐Ÿ‘จ‍๐Ÿ’ป ๋“ค์–ด๊ฐ€๋ฉฐ KoNLPy์™€ Mecab ํŒจํ‚ค์ง€๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ์„ค์น˜๋˜์–ด ์žˆ๋‹ค๊ณ  ๊ฐ€์ •ํ•ฉ๋‹ˆ๋‹ค. ํ˜น์‹œ๋‚˜ ์„ค์น˜๋˜์–ด ์žˆ์ง€ ์•Š๋‹ค๋ฉด ์•„๋ž˜ ํฌ์ŠคํŒ…์„ ์ฐธ๊ณ ํ•ด ์ฃผ์„ธ์š”. https://heytech.tistory.com/3 [Python/NLP] KoNLPy ์„ค์น˜ํ•˜๋Š”

heytech.tistory.com

1.3 ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ ๋น„๊ต

ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ๋Š” ์–ด๋–ค ๊ฒƒ์ด ๊ฐ€์žฅ ์ข‹์„๊นŒ? KoNLPy ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋Š” Kkma, Okt, Mecab, Komoran ๋“ฑ ์—ฌ๋Ÿฌ ํ•œ๊ตญ์–ด ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ๋ฅผ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ์ค‘์—์„œ๋„ ์ €๋Š” ์ฃผ๋กœ ๋น ๋ฅธ ์—ฐ์‚ฐ์†๋„๋ฟ๋งŒ ์•„๋‹ˆ๋ผ ์šฐ์ˆ˜ํ•œ ํ† ํฐํ™” ํ’ˆ์งˆ์„ ๊ฐ–์ถ˜  Mecab ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ๋ฅผ ์• ์šฉํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ ๋ชจ๋“  ์ƒํ™ฉ์—์„œ Mecab ๋ถ„์„๊ธฐ๊ฐ€ ๋‹จ์—ฐ ์ข‹์€ ๊ฒƒ๋„ ์•„๋‹™๋‹ˆ๋‹ค. ์ด์™€ ๊ด€๋ จํ•˜์—ฌ ๋‹ค๋ฅธ ๋ธ”๋กœ๊ฑฐ์˜ ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ ์„ฑ๋Šฅ ๋น„๊ต ๊ฒŒ์‹œ๊ธ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•ด ๋“œ๋ฆฝ๋‹ˆ๋‹ค. ์ด๋ฅผ ์ฐธ๊ณ ํ•˜์‹œ์–ด ๋ถ„์„ ๋ชฉ์ ์— ๋งž๊ฒŒ ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ๋ฅผ ์„ ํƒํ•˜์‹œ๊ธธ ๋ฐ”๋ž๋‹ˆ๋‹ค.
iostream.tistory.com/144#header-n14

 

ํ•œ๊ตญ์–ด ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ ์„ฑ๋Šฅ ๋น„๊ต

korean-tokenizer-experiments ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ ๋น„๊ต์‹คํ—˜ ํ™˜๊ฒฝํ•˜๋“œ์›จ์–ด (MacBook Pro Mid 2015)์†Œํ”„ํŠธ์›จ์–ด๋ฐ์ดํ„ฐ์‹คํ—˜ ๋‚ด์šฉ์‹คํ–‰ ์‹œ๊ฐ„ ๋น„๊ต๋กœ๋”ฉ ์‹œ๊ฐ„ํ˜•ํƒœ์†Œ ๋ถ„์„ ์‹œ๊ฐ„๋ฌธ์žฅ ๋ถ„์„ ํ’ˆ์งˆ ๋น„๊ต๋„์–ด์“ฐ๊ธฐ๊ฐ€ ์—†๋Š” ๋ฌธ์žฅ์ž

iostream.tistory.com

2. ํ˜•ํƒœ์†Œ๋ณ„ ํ† ํฐํ™”(tokenization)ํ•˜๊ธฐ

Mecab ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ์˜ ์‚ฌ์šฉ๋ฐฉ๋ฒ•์„ ์˜ˆ์‹œ๋กœ ์†Œ๊ฐœํ•ด ๋“œ๋ฆฝ๋‹ˆ๋‹ค.

2.1. ํ† ํฐํ™” without ํ’ˆ์‚ฌ ํƒœ๊น…

# To tokenize text data based on morphemes without the tagging
mecab.morphs("์•ˆ๋…•ํ•˜์„ธ์š”, ํ—ค์ด ํ…Œํฌ ๋ธ”๋กœ๊ทธ์˜ ์ž์—ฐ์–ด์ฒ˜๋ฆฌ ๊ด€๋ จ ํฌ์ŠคํŒ…์ž…๋‹ˆ๋‹ค. 1234-!5123 @!@!@@$")

ํ† ํฐํ™” ๊ฒฐ๊ณผ(๊ทธ๋ฆผ 1)

๊ทธ๋ฆผ 1. ํ˜•ํƒœ์†Œ ๋ถ„์„ without ํ’ˆ์‚ฌ ํƒœ๊น…

Mecab ํ˜•ํƒœ์†Œ ๋ถ„์„๊ธฐ ๋‚ด morphs ํ•จ์ˆ˜๋ฅผ ํ†ตํ•ด ๋ณ„๋„์˜ ํ˜•ํƒœ์†Œ ํƒœ๊น… ์—†์ด ์˜ˆ์‹œ ๋ฌธ์žฅ์„ ํ˜•ํƒœ์†Œ๋ณ„๋กœ ํ† ํฐํ™” ํ•œ ๊ฒฐ๊ณผ์ž…๋‹ˆ๋‹ค. ํ•œ๊ธ€๊ณผ ๊ตฌ๋‘์ , ํŠน์ˆ˜๋ฌธ์ž ๊ฐ„์—๋„ ํ† ํฐํ™”๋ฅผ ์ •์ƒ์ ์œผ๋กœ ์ง€์›ํ•˜๋Š” ๊ฒƒ์„ ์•Œ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, mecab morphs ํ•จ์ˆ˜๋Š” string ๋ฌธ์ž์—ด์„ ์ž…๋ ฅ๋ฐ›์•„ 1์ฐจ์› list ํ˜•ํƒœ๋กœ return ํ•ฉ๋‹ˆ๋‹ค.

2.2. ํ† ํฐํ™” with ํ’ˆ์‚ฌ ํƒœ๊น…

# To tokenize text data based on morphemes wit the tagging
mecab.pos("์•ˆ๋…•ํ•˜์„ธ์š”, ํ—ค์ด ํ…Œํฌ ๋ธ”๋กœ๊ทธ์˜ ์ž์—ฐ์–ด์ฒ˜๋ฆฌ ๊ด€๋ จ ํฌ์ŠคํŒ…์ž…๋‹ˆ๋‹ค. 1234-!5123 @!@!@@$")

ํ† ํฐํ™” ๊ฒฐ๊ณผ(๊ทธ๋ฆผ 2)

๊ทธ๋ฆผ 2. ํ˜•ํƒœ์†Œ ๋ถ„์„ with ํ’ˆ์‚ฌ ํƒœ๊น…

์ด๋ฒˆ์—๋Š” Mecab ๋‚ด pos ํ•จ์ˆ˜๋ฅผ ํ†ตํ•ด ์˜ˆ์‹œ ๋ฌธ์žฅ์„ ํ’ˆ์‚ฌ ํƒœ๊น…๊ณผ ํ•จ๊ป˜ ํ† ํฐํ™” ํ•˜์˜€์Šต๋‹ˆ๋‹ค. ๊ฒฐ๊ณผ ์Šคํฌ๋ฆฐ์ƒท์—์„œ ๋ณด์‹ค ์ˆ˜ ์žˆ๋“ฏ์ด, string์„ ์ž…๋ ฅ๋ฐ›์•„ ๋ฆฌ์ŠคํŠธ ์•ˆ์— ํŠœํ”Œ ํ˜•ํƒœ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ๋ฐ˜ํ™˜ํ•ด ์ค๋‹ˆ๋‹ค. ํƒœ๊น…์— ๋Œ€ํ•œ ํ’ˆ์‚ฌ ์ •๋ณด๋Š” ์•„๋ž˜์— ์ž˜ ์ •๋ฆฌ๋˜์–ด ์žˆ์œผ๋‹ˆ ์ฐธ๊ณ ํ•˜์‹œ๊ธธ ๋ฐ”๋ž๋‹ˆ๋‹ค.
docs.google.com/spreadsheets/d/1OGAjUvalBuX-oZvZ_-9tEfYD2gQe7hTGsgUpiiBSXI8/edit#gid=0

 

Korean POS tags comparison chart

chart Not provided in KoNLPy,Provided in KoNLPy Sejong project (ntags=42),Sim Gwangsub project (ntags=26),Twitter Korean Text (ntags=19),Komoran (ntags=42),Mecab-ko (ntags=43),Kkma (ntags=10),Kkma (ntags=30),Kkma (ntags=56),Hannanum (ntags=9),Hannanum (nta

docs.google.com

2.3. ๋ช…์‚ฌ๋งŒ ์ถ”์ถœํ•˜๊ธฐ

# To extract only noun in text data
mecab.nouns("์•ˆ๋…•ํ•˜์„ธ์š”, ํ—ค์ด ํ…Œํฌ ๋ธ”๋กœ๊ทธ์˜ ์ž์—ฐ์–ด์ฒ˜๋ฆฌ ๊ด€๋ จ ํฌ์ŠคํŒ…์ž…๋‹ˆ๋‹ค. 1234-!5123 @!@!@@$")

ํ† ํฐํ™” ๊ฒฐ๊ณผ(๊ทธ๋ฆผ 3)

๊ทธ๋ฆผ 3. ๋ช…์‚ฌ๋งŒ ์ถ”์ถœํ•œ ๊ฒฝ์šฐ

๐Ÿ“š ์ฐธ๊ณ ํ•  ๋งŒํ•œ ํฌ์ŠคํŒ…

 

Mecab ์„ค์น˜ ์—๋Ÿฌ ํ•ด๊ฒฐํ•˜๊ธฐ: Exception: Install MeCab in order to use it: http://konlpy.org/en/latest/install/

๐Ÿ‘จ‍๐Ÿ’ป ๋“ค์–ด๊ฐ€๋ฉฐ KoNLPy์™€ Mecab ํŒจํ‚ค์ง€๋Š” ๊ธฐ๋ณธ์ ์œผ๋กœ ์„ค์น˜๋˜์–ด ์žˆ๋‹ค๊ณ  ๊ฐ€์ •ํ•ฉ๋‹ˆ๋‹ค. ํ˜น์‹œ๋‚˜ ์„ค์น˜๋˜์–ด ์žˆ์ง€ ์•Š๋‹ค๋ฉด ์•„๋ž˜ ํฌ์ŠคํŒ…์„ ์ฐธ๊ณ ํ•ด ์ฃผ์„ธ์š”. https://heytech.tistory.com/3 [Python/NLP] KoNLPy ์„ค์น˜ํ•˜๋Š”

heytech.tistory.com


ํฌ์ŠคํŒ… ๋‚ด์šฉ์— ์˜ค๋ฅ˜๊ฐ€ ์žˆ๋‹ค๋ฉด ์•„๋ž˜์— ๋Œ“๊ธ€ ๋‚จ๊ฒจ์ฃผ์„ธ์š”!
๊ทธ๋Ÿผ ์˜ค๋Š˜๋„ ์ฆ๊ฒ๊ณ  ๊ฑด๊ฐ•ํ•œ ํ•˜๋ฃจ ๋ณด๋‚ด์‹œ๊ธธ ๋ฐ”๋ž๋‹ˆ๋‹ค.
๊ณ ๋ง™์Šต๋‹ˆ๋‹ค :)

728x90
๋ฐ˜์‘ํ˜•
0 Comments
๋Œ“๊ธ€์“ฐ๊ธฐ ํผ