requests.get 异常hang住
这两天优化端口扫描脚本，在获取HTTP页面内容时，发现requests.get hang住了（之前httplib不会）。
requests.get 提供了超时参数timeout，但在实际使用过程中，仍发现get请求可能被hang住。
原因在于非HTTP Server，response中一直没有出现换行符\n，导致_read_status()在获取状态码的时候就hang住了。
一个简单的解决方法，可以创建新的线程，示例代码参考如下，这部分代码是无法直接工作的：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
def do_get_thread(url, ret): try: headers = { 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 ' '(KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25', 'Range': 'bytes=0-10240', 'Connection': 'Close'} ret.append(requests.get(url, headers=headers, timeout=(4, 30))) except: pass def get_http_title(ip, port, service, log_queue): if service.lower().find('https') >= 0 or service.lower().find('ssl') >= 0: service = 'https' else: service = 'http' url = '%s://%s:%s' % (service, ip, port) msg = 'Get http title %s' % url log_queue.put(msg) try: start_time = time.time() ret = [] t = threading.Thread(target=do_get_thread, args=(url, ret)) t.daemon = True t.start() while t.isAlive(): if time.time() - start_time > 31: log_queue.put('[get_http_title timed out] %s' % url) return {}, '' else: time.sleep(1.0) if not ret: return {}, '' resp = ret[0] headers = resp.headers m = re.search('<title>(.*?)</title>', resp.text, re.IGNORECASE) http_title = m.group(1).strip() if m else '' if http_title: http_title = decode_response_text(http_title) else: http_title = resp.text.replace('\n', ' ').strip()[0:200] return headers, http_title except requests.exceptions.RequestException, e: pass except Exception, e: log_queue.put('[get_http_title.exception] %s' % str(e)) return {}, '' |
Struts2-045漏洞批量扫描工具
一个 Struts2-045漏洞批量扫描工具
https://github.com/lijiejie/struts2_045_scan
Requirements
|
1 2 |
pip install requests |
Usage
|
1 2 3 4 5 6 7 8 9 |
usage: scan.py [options] Struts2-045 Scanner. By LiJieJie (http://www.lijiejie.com) optional arguments: -h, --help show this help message and exit -f File New line delimited targets from File -t THREADS Num of scan threads, 100 by default |
Screenshot
iTunes 更改备份文件保存路径
建议使用iPhone的同学周期性地备份自己的手机，以免意外数据丢失。
使用iTunes备份手机到电脑上，win系统默认是保存在系统盘 C盘的。
我购置了240G固态硬盘，C盘只有120G，多备份几次就被耗尽了。
观察到iTunes无法配置备份文件路径，那么我们可以移动文件到一个更大的盘符下（例如我的G盘，是机械硬盘的分区），之后再创建一个目录连接到C盘对应目录即可。
移动文件夹
“C:\Users\Administrator\AppData\Roaming\Apple Computer\MobileSync\Backup”
到
“G:\iphone\Backup”
然后创建一个目录连接
mklink /J “C:\Users\Administrator\AppData\Roaming\Apple Computer\MobileSync\Backup” “G:\iphone\Backup”
Ref: https://discussions.apple.com/thread/4977359?tstart=0
.DS_Store文件泄漏利用工具: ds_store_exp
.DS_Store是Mac下Finder用来保存如何展示 文件/文件夹 的数据文件，每个文件夹下对应一个。
如果开发/设计人员将.DS_Store上传部署到线上环境，可能造成文件目录结构泄漏，特别是备份文件、源代码文件。
ds_store_exp 是一个.DS_Store 文件泄漏利用脚本，它解析.DS_Store文件并递归地下载文件到本地： https://github.com/lijiejie/ds_store_exp
DS_Store parser is based on ds_store 1.1.0 。
一个示例
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
python ds_store_exp.py http://hd.zj.qq.com/themes/galaxyw/.DS_Store hd.zj.qq.com/ └── themes └── galaxyw ├── app │ └── css │ └── style.min.css ├── cityData.min.js ├── images │ └── img │ ├── bg-hd.png │ ├── bg-item-activity.png │ ├── bg-masker-pop.png │ ├── btn-bm.png │ ├── btn-login-qq.png │ ├── btn-login-wx.png │ ├── ico-add-pic.png │ ├── ico-address.png │ ├── ico-bm.png │ ├── ico-duration-time.png │ ├── ico-pop-close.png │ ├── ico-right-top-delete.png │ ├── page-login-hd.png │ ├── pic-masker.png │ └── ticket-selected.png └── member ├── assets │ ├── css │ │ ├── ace-reset.css │ │ └── antd.css │ └── lib │ ├── cityData.min.js │ └── ueditor │ ├── index.html │ ├── lang │ │ └── zh-cn │ │ ├── images │ │ │ ├── copy.png │ │ │ ├── localimage.png │ │ │ ├── music.png │ │ │ └── upload.png │ │ └── zh-cn.js │ ├── php │ │ ├── action_crawler.php │ │ ├── action_list.php │ │ ├── action_upload.php │ │ ├── config.json │ │ ├── controller.php │ │ └── Uploader.class.php │ ├── ueditor.all.js │ ├── ueditor.all.min.js │ ├── ueditor.config.js │ ├── ueditor.parse.js │ └── ueditor.parse.min.js └── static ├── css │ └── page.css ├── img │ ├── bg-table-title.png │ ├── bg-tab-say.png │ ├── ico-black-disabled.png │ ├── ico-black-enabled.png │ ├── ico-coorption-person.png │ ├── ico-miss-person.png │ ├── ico-mr-person.png │ ├── ico-white-disabled.png │ └── ico-white-enabled.png └── scripts ├── js └── lib └── jquery.min.js 21 directories, 48 files |
QNetworkReply获取Content-Length的方法(PyQT)
使用pyqt webkit写headless 动态爬虫，需要捕获到所有的HTTP request和response(包括JS异步请求)。
QNetworkReply 没有返回content-length相关的方法，rawHeaderList实际上并不包括content-length。
一个可行的方法，是在readyRead信号触发时，size()方法获取到body的大小。示例代码如下，请注意self.reply_lst这个列表是必不可少的：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
import sys from PyQt5.Qt import QApplication from PyQt5.QtCore import QUrl from PyQt5.Qt import QWebView, QWebPage from PyQt5.QtNetwork import QNetworkAccessManager class Manager(QNetworkAccessManager): def __init__(self, parent=None): QNetworkAccessManager.__init__(self, parent) self.finished.connect(self._finished) self.reply_lst = [] def _finished(self, reply): print '[%s bytes] %s' % (reply.content_length, reply.url().toString()) self.reply_lst.remove(reply) def createRequest(self, operation, request, body=None): _url = request.url() reply = super(Manager, self).createRequest(operation, request, body) reply.readyRead.connect(self.read_read) self.reply_lst.append(reply) return reply def read_read(self): self.sender().content_length = self.sender().size() def app_quit(): app.quit() if __name__ == "__main__": app = QApplication(['-platform', 'minimal']) browser = QWebView() page = QWebPage() manager = Manager() page.setNetworkAccessManager(manager) browser.setPage(page) browser.load(QUrl('http://www.lijiejie.com/')) browser.loadFinished.connect(app_quit) sys.exit(app.exec_()) |
最后，拿到所有URL的content-length如下，可以把它写到爬虫的结果中：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[499 bytes] http://www.lijiejie.com/ [2082 bytes] https://s6.cnzz.com/stat.php?id=3804994&web_id=3804994&show=pic [11414 bytes] http://www.lijiejie.com/wp-includes/js/wp-emoji-release.min.js?ver=4.7.1 [20172 bytes] http://www.lijiejie.com/wp-content/plugins/crayon-syntax-highlighter/css/min/crayon.min.css?ver=_2.7.2_beta [2850 bytes] http://www.lijiejie.com/wp-content/plugins/crayon-syntax-highlighter/themes/github/github.css?ver=_2.7.2_beta [86 bytes] http://www.lijiejie.com/wp-content/plugins/crayon-syntax-highlighter/fonts/courier-new.css?ver=_2.7.2_beta [374 bytes] http://www.lijiejie.com/wp-content/plugins/wp-pagenavi/pagenavi-css.css?ver=2.70 [27597 bytes] http://www.lijiejie.com/wp-content/themes/retro-fitted/style.min.css?ver=0.4 [10056 bytes] http://www.lijiejie.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1 [3678 bytes] http://www.lijiejie.com/wp-content/plugins/dynamic-to-top/js/libs/jquery.easing.js?ver=1.3 [4214 bytes] http://www.lijiejie.com/wp-content/themes/retro-fitted/library/js/drop-downs.min.js?ver=20110920 [22337 bytes] http://www.lijiejie.com/wp-content/plugins/crayon-syntax-highlighter/js/min/crayon.min.js?ver=_2.7.2_beta [1192 bytes] http://www.lijiejie.com/wp-content/plugins/dynamic-to-top/js/dynamic.to.top.min.js?ver=3.5 [1398 bytes] http://www.lijiejie.com/wp-includes/js/wp-embed.min.js?ver=4.7.1 [6108 bytes] http://www.lijiejie.com/wp-includes/js/jquery/jquery.js?ver=1.12.4 [765 bytes] https://c.cnzz.com/core.php?web_id=3804994&show=pic&t=z [3197 bytes] http://www.lijiejie.com/wp-content/themes/retro-fitted/images/bg.jpg [2236 bytes] http://www.lijiejie.com/wp-content/plugins/crayon-syntax-highlighter/css/images/toolbar/buttons.png [576 bytes] http://www.lijiejie.com/wp-content/themes/retro-fitted/images/quotes.png [9766 bytes] http://www.lijiejie.com/wp-content/themes/retro-fitted/images/header.png [7210 bytes] http://www.lijiejie.com/wp-content/uploads/2014/10/stom.tencent.com_.png [719 bytes] https://icon.cnzz.com/img/pic.gif [43 bytes] https://hzs9.cnzz.com/stat.htm?id=3804994&r=&lg=zh-cn&ntime=none&cnzz_eid=1834513944-1484813777-&showp=1920x1080&t=李劼杰的博客&h=1&rnd=1792993998 [43 bytes] https://cnzz.mmstat.com/9.gif?abc=1&rnd=2124379566 [142 bytes] http://www.lijiejie.com/wp-content/themes/retro-fitted/images/bullet.png [2516 bytes] http://www.lijiejie.com/wp-content/uploads/2017/01/eping.png [3282 bytes] http://www.lijiejie.com/wp-content/uploads/2014/10/stom.tencent.com_upfile.png |
eping — 为ping增加urlparse
做渗透测试或提交漏洞单时，常常用到ping，来确认内外网IP，或者寻找资产的归属。
然而ping并不支持url parse。我们写个简单的extended ping脚本。
vim /usr/bin/eping
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
#!/usr/bin/env python # Extented Ping import sys import urlparse import platform import os if __name__ == '__main__': if len(sys.argv) == 1: print ' Usage: python %s URL' % sys.argv[0] sys.exit(0) target = sys.argv[1].lower() if not target.startswith('http'): target = 'http://%s' % target host = urlparse.urlparse(target, 'http').hostname param_count = '-c4 ' if platform.system() != 'Windows' else '' os.system('ping %s%s' % (param_count, host)) |
chmod +x /usr/bin/eping
|
1 2 3 4 5 6 7 8 9 10 |
root@srvScanner-test-dev001:~# eping http://www.lijiejie.com/requests-exceptions-invalidschema-missing-dependencies-for-socks-support/ PING www.lijiejie.com (47.88.12.133) 56(84) bytes of data. 64 bytes from 47.88.12.133: icmp_seq=1 ttl=52 time=138 ms 64 bytes from 47.88.12.133: icmp_seq=2 ttl=52 time=138 ms 64 bytes from 47.88.12.133: icmp_seq=3 ttl=52 time=138 ms 64 bytes from 47.88.12.133: icmp_seq=4 ttl=52 time=138 ms --- www.lijiejie.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 138.204/138.387/138.683/0.317 ms |
Windows下使用pyinstaller生成一个exe文件，copy到C:\Windows\System32
pyinstaller –onefile eping.py
Windows 64 bit executable http://www.lijiejie.com/wp-content/uploads/2017/01/eping.zip
近期评论