Warning: Use of undefined constant memo - assumed 'memo' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 278
홍차넷
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 34
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 45
Warning: Use of undefined constant r - assumed 'r' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 47
Warning: Use of undefined constant r - assumed 'r' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 51
Warning: Use of undefined constant memo - assumed 'memo' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 237
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 34
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 45
Warning: Use of undefined constant r - assumed 'r' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 47
Warning: Use of undefined constant r - assumed 'r' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/outlogin_toby.php on line 51
Warning: Use of undefined constant memo - assumed 'memo' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 237
Warning: Use of undefined constant is_admin - assumed 'is_admin' (this will throw an Error in a future version of PHP) in /home/redtea/html/zero_header.php on line 269
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/zero_header.php on line 300
Warning: Use of undefined constant title - assumed 'title' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/bbs_title.php on line 16
질문
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/bbs_title.php on line 21
Warning: Use of undefined constant header - assumed 'header' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/bbs_title.php on line 31
- 질문 게시판입니다.
Warning: Use of undefined constant use_formmail - assumed 'use_formmail' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 519
Warning: Use of undefined constant use_category - assumed 'use_category' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/include/print_category.php on line 9
전체기타게임진로연애가정/육아체육/스포츠교육의료/건강철학/종교IT/컴퓨터여행과학문화/예술법률홍차넷경제
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/setup.php on line 29
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/setup.php on line 41
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 398
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant subject - assumed 'subject' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 402
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 403
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 405
Warning: Use of undefined constant total_comment - assumed 'total_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 406
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 2
Warning: Use of undefined constant HTTP_HOST - assumed 'HTTP_HOST' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/memo_replace.php on line 3
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 10
Warning: Use of undefined constant use_category - assumed 'use_category' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 40
| Date |
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 53 18/02/12 21:39:55 Warning: Use of undefined constant mod_date - assumed 'mod_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 56 |
| Name | 호라타래 |
| File #1 | 크롤링_코드_1.PNG (71.6 KB), Download : 11 |
| File #2 | 크롤링_코드_2.PNG (80.8 KB), Download : 7 |
| Subject | 웹사이트 분석 / 크롤링 관련 질문 있습니닷 |
  http://www.genie.co.kr/detail/songInfo?xgnm=87759645 파이썬을 이용해서 노래 관련된 정보들을 크롤링/분석하려고 해요. 이미지에 나와있듯이 for문을 이용해서 8~9천만개 정도 존재하는 songInfo 페이지를 하나씩 긁어들이게 만들었는데, 이 중 상당수는 빈페이지입니다. 지금과 같은 코드는 시간도 오래 걸리고, 비효율적이라 개선 방안을 고민하는 중이에요. 1) 셀레니움을 이용하지 않아도 Alert 창을 컨트롤 할 수 있나요? http://www.genie.co.kr/detail/songInfo?xgnm=1 페이지를 눌러보면 보이듯이 Alert창이 뜨기 때문에, 셀레니움을 이용해서 컨트롤을 해주고 있어요. 근데 속도가 상당히 느리더라고요. BeautifulSoup만 써서 정적 스크래핑을 하면 속도가 개선될 듯한데, Alert를 처리할 수 있는 방법이 있을까요? Ajax인지 아닌지도 모르겠네요 ㅋㅋㅋㅋㅋㅋ 2) 지금 코드를 효율화 할 수 있는 방법이 있을까요? 향후 전처리를 염두에 두고 코드를 짜기는 했는데, 이게 효율적인지 아닌지도 감이 안 잡히네요. 3) 구글 크롬 개발자 도구 외에 사이트 구조를 분석할 수 있는 방법이 있을까요? 비어있지 않은 songInfo 페이지를 사전에 파악한 후, 인덱스를 부여해서 for문을 돌리면 훨씬 효율적이지 않을까 싶은데, 구글 크롬 개발자 도구로는 파악을 못하겠습니당 조언 부탁드립니다ㅠㅠ 영어 사이트여도 활용 가능하니 힌트/키워드라도 얻을 수 있다면 흑흑... 맞다. 제 작업 환경은 램 4G, CPU 2.93Ghz, winddow7, Anaconda3(32bit) - Jupyter notebook - Python 3.6입니다. 설날 전까지는 이 환경에서 작업을 해야 해요. 코드 정리되고 나면 프롬프트 창에서 파이썬을 돌리려고요 8ㅅ8 Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 133 Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 146 Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view.php on line 161 0 |
|
Warning: Use of undefined constant memo - assumed 'memo' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 185
Warning: Use of undefined constant use_comment - assumed 'use_comment' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 540
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 542
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 543
Warning: Use of undefined constant memo - assumed 'memo' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 545
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 576
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 577
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 577
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 577
Warning: Use of undefined constant grant_delete - assumed 'grant_delete' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 577
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 429
Warning: Use of undefined constant use_formmail - assumed 'use_formmail' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 605
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 599
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 599
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 600
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 600
Warning: Use of undefined constant is_admin - assumed 'is_admin' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 605
Warning: Use of undefined constant is_admin - assumed 'is_admin' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 605
Warning: Use of undefined constant is_admin - assumed 'is_admin' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 609
Warning: Use of undefined constant is_admin - assumed 'is_admin' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 609
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 614
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 616
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 618
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/lib.php on line 618
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 612
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 612
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 620
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 628
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 628
Warning: Use of undefined constant HTTP_HOST - assumed 'HTTP_HOST' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 640
Warning: Use of undefined constant name - assumed 'name' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 644
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 646
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/view.php on line 646
Warning: Use of undefined constant HTTP_HOST - assumed 'HTTP_HOST' (this will throw an Error in a future version of PHP) in /home/redtea/html/pb/cmt_replace.php on line 3
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 6
Warning: Use of undefined constant file_name - assumed 'file_name' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 28
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 65
35703 class="iamcmt" style="display:none;"> Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 66
35703 class='vcm '>
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 70
Warning: Use of undefined constant x - assumed 'x' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 71
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 76
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 117
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 118
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 119
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 120
Warning: Use of undefined constant ismember - assumed 'ismember' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 121
35703'> Toby
Warning: Use of undefined constant mod_date - assumed 'mod_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 133
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 140
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 140
Warning: Use of undefined constant reg_date - assumed 'reg_date' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 142
18/02/12 21:49
35703>삭제 주소복사
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 150
결국에는 셀레니움을 사용하여 처리하는 방식과 유사한 방식이어야 할테니 속도문제의 비약적 개선은 어렵지 않을까 싶어요. 그래서 API와 같이 데이터 제공을 목적으로 한 별도의 장치가 필요한거죠. 사용자를 위해 열어놓은 수단을 봇이 사람인척 하며 접근해서 크롤링을 하고자하면 셀레니움의 방식이 최선이라 생각합니다.
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 158
Warning: Use of undefined constant password - assumed 'password' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 166
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 171
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 172
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 172
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 173
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 173
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 173
Warning: Use of undefined constant level - assumed 'level' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 158
Warning: Use of undefined constant password - assumed 'password' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 166
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 171
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 172
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 172
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 173
Warning: Use of undefined constant no - assumed 'no' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 173
Warning: Use of undefined constant vote - assumed 'vote' (this will throw an Error in a future version of PHP) in /home/redtea/html/a_skin/bbsd/view_comment.php on line 173
35703>
35703>
