Resolve Scrapy Crawler Errors: Python / Scrapy / Scrapyd / MySQL / Splash

已完成 已发布的 6 年前 货到付款
已完成 货到付款

Developer needed with Python / Scrapy / Scrapyd / MySQL / Splash.

The code repository is in bitbucket.

1. Task One: Video information not recording to page_video table. Our crawler is working but not saving this data to this table.

2. Task Two: Resolve Crawler Errors

We have added the ability to crawl javascript with Scrapy by using Splash. We have a few errors that we need resolving.

a. Prevent a request to [login to view URL] of splash server from happening.

b. Do not forward head requests to splash

c. There's a lot of errors like this

2018-01-29 07:56:07 [[login to view URL]] ERROR: Error downloading <POST [login to view URL]>

Traceback (most recent call last):

File "/usr/local/lib/python2.7/site-packages/twisted/internet/[login to view URL]", line 1299, in _inlineCallbacks

result = [login to view URL](result)

File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/[login to view URL]", line 53, in process_response

spider=spider)

File "/usr/src/app/InnoCrawler/downloadermiddleware/[login to view URL]", line 177, in process_response

request, response, spider

File "/usr/src/app/InnoCrawler/downloadermiddleware/[login to view URL]", line 156, in _convert_from_splash

for row in data['history']:

KeyError: 'history'

Linux PHP Python 软件构架 网页搜罗

项目ID: #16535481

关于项目

16个方案 远程项目 活跃的6 年前

授予:

chirgeo

Hi. I do have experience with scrapy and spash for large crawling projects. I can take a look at the project and see more details. First of all I would like to have access to the project and check it structure and h 更多

$833 USD 在5天内
(132条评论)
7.5

有16名威客正在参与此工作的竞标,均价$515/小时

mingxiao2008

Hello Dear! I am very interested in your project. I have high skills in Python. Specially, I have rich experience in Scrapy/Scrapyd. If you contact me, we can discuss on your project. Thansk

$400 USD 在5天内
(114条评论)
8.2
zekovicm

Hi there,I am Miljan,Python(Scrapy is my favourite library) expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immedi 更多

$388 USD 在3天内
(152条评论)
7.7
polarjin2017

Hello? How are you? My name is GuangJun I have read your brief so i can discuss the project with you. Hope to work with you. you can see my reviews. Thanks.

$555 USD 在10天内
(168条评论)
7.2
schoudhary1553

Hi there..... Warm Greetings We came along with your request for Resolve Scrapy Crawler Errors: Python / Scrapy / Scrapyd / MySQL / Splash and we reviewed your project description. We'd like to help you with conf 更多

$600 USD 在10天内
(110条评论)
6.9
hunmin888

hi, sir. i am a python expert. so if you give me this project, i can finish it in time. the price may be negotiable. thanks.

$444 USD 在10天内
(66条评论)
6.6
umg536

A proposal has not yet been provided

$555 USD 在5天内
(22条评论)
6.2
beautiXiao

I believe that my 9+ years of programming experience makes me highly qualified for this project. I believe that I would be an asset to this project. Linux, PHP, Python, Software Architecture, Web Scraping,

$555 USD 在4天内
(23条评论)
6.1
exansoft

Hello I'm experienced Python developer and I'm very interested with the task. Let's discuss it in detail over the chat. Regards, Alex.

$750 USD 在7天内
(50条评论)
6.2
abhilashrana01

Hello dear, I hope this proposal find you well, I saw your job posting online, I would like to request you please have a look into my previous works. I am a full-time freelancer with over 5 years of vast experience i 更多

$750 USD 在3天内
(24条评论)
4.3
alex2018cs

Hi Sir, I can do the project with ease, i just need the bitbucket repository access, and I am ready to fix those errors. I am proficient with splash, scrapy, scrapyd, and Mysql. I can easily fix Task1 but when it com 更多

$500 USD 在5天内
(5条评论)
2.9