{"componentChunkName":"component---src-templates-tag-js","path":"/tags/coding/","result":{"data":{"site":{"siteMetadata":{"title":"LoginRadius Blog"}},"allMarkdownRemark":{"totalCount":2,"edges":[{"node":{"fields":{"slug":"/engineering/learning-how-to-code/"},"html":"

When you work for a tech company in an office capacity, it feels like everyone around you is speaking another language. Which they are, most of their work exists in coding language. To feel more relevant, I signed up on codecademy.com and started working on their beginner courses (for free! you should try it!).

\n

Here are some things I have learned about coding:

\n
    \n
  1. There are many different ways to code. Not every website or app is made using the same terms or patterns, programmers and developers have options for how they want to format their content, and how they want to communicate with their computer.
    2. \n
  2. Coding is more about seeing what you want in your head than about math. Deciding exactly what you want and communicating it is the big struggle. Having your website/app be capable of computing things is optional, but the layout is mandatory. (Not to mention the most noticeable aspect of your site).
    3. \n
  3. Code is like a language. The words and punctuation must be learned for each coding language. Though they can be linked to English words (like “p” for a new paragraph), you have to be able to pull them up quick in your mind. As with any language, repetition is key to recognition.
    4. \n
  4. You don’t have to start from scratch (but you can). There are frameworks available, like Bootstrap, which have some existing templates you can draw from. This helps you when you are constructing your site, so that you can set up your layout quicker, and have more time to work on your content.
    5. \n
  5. \n

    There are some areas of knowledge that you don’t need to memorize, but having an understanding of what they do will be helpful in coding:

    \n
      \n
    1. The hexi-decimal system (and how it relates to color)
      2. \n
    2. Binary code
      3. \n
    3. ASCII
      4. \n
    \n
    6. \n
  6. Once you learn some of the basics, websites you see every day will start to seem simpler. You will be able to pick out some of their elements, and if you look at their bare code, you will recognize how they come together.
    7. \n
  7. The world of coders is large and easy to access. If you have questions, there is someone online who is willing and eager to answer. This is not a skill that anyone was born with, and due to new languages etc, everyone is still learning.
    8. \n
\n

I hope that this list has contributed to your knowledge, and would encourage you to check online for available resources to expand (or begin) your abilities in coding.

\n","frontmatter":{"date":"December 29, 2015","updated_date":null,"title":"Learning How to Code","tags":["Learning","Coding","Learning resources"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1,"src":"/static/26a66e05ab78493dc6d84d3afe0d8a82/630fb/begin-code-300x300.png","srcSet":"/static/26a66e05ab78493dc6d84d3afe0d8a82/69585/begin-code-300x300.png 200w,\n/static/26a66e05ab78493dc6d84d3afe0d8a82/630fb/begin-code-300x300.png 300w","sizes":"(max-width: 300px) 100vw, 300px"}}},"author":{"id":"Carling","github":null,"avatar":null}}}},{"node":{"fields":{"slug":"/engineering/write-a-highly-efficient-python-web-crawler/"},"html":"

As my previous blog, I use the python web Crawler library to help crawl the static website. For the Scrapy, there can be customize download middle ware, which can deal with static content in the website like JavaScript.

\n

However, the Scrapy already helps us with much of the underlying implementation, for example, it uses it own dispatcher and it has pipeline for dealing the parsing word after download.  One drawback for using such library is hard to deal with some strange bugs occurring because they run the paralleled jobs.

\n

For this tutorial, I want to show the structure of a simple and efficient web crawler.

\n

First of all, we need a scheduler, who can paralleled the job. Because the most of the time is on the requesting.  I use the  gevent to schedule the jobs. Gevent uses the libevent as its underlying library, which combines the multithreading and event-based techniques to parallel the job.

\n

There is the sample code:

\n
import gevent\nfrom gevent import Greenlet\nfrom gevent import monkey\nfrom selenium import webdriver\nmonkey.patch_socket()\nclass WebCrawler:\n    def __init__(self,urls=[],num_worker = 1):\n        self.url_queue = Queue()\n        self.num_worker = num_worker\n    def worker(self,pid):\n        driver = self.initializeAnImegaDisabledDriver()  #initilize the webdirver\n#TODO catch the exception\n        while not self.url_queue.empty():\n            url = self.url_queue.get()\n            self.driver.get(url)\n            elem = self.driver.find_elements_by_xpath("//script | //iframe | //img") # get such element from webpage\n    def run(self):\n        jobs = [gevent.spawn(self.worker,i) for i in xrange(self.num_worker)]
\n

The next part is the headless browser part. I use the phantomjs with --webdriver=4444 --disk-cache=true --ignore-ssl-errors=true --load-images=false --max-disk-cache-size=100000. You can get the detailed option from their documents.

\n

Phantomjs uses selenium webdriver as front-end to handle the request. However phantomjs is using the webkit and QT as its underlying browser and controller. It has memory leak bugs therefore the phantomjs will consume ton of memory and it only can use one core of your CPU but you can deploy many instances of the phantomjs on different ports. I wrote a daemon process to monitor the memory and its situation but later I realize I can use Perl script to get the status of process and when it exceeds the limits like 1G memory and send kill signal to the process.

\n

To speed up the crawler, I choose to use static browser to verify the website first because the website is bad written, there might be deadlock occurring so just skip them.

\n","frontmatter":{"date":"July 14, 2015","updated_date":null,"title":"Write a highly efficient python Web Crawler","tags":["Python","Coding"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1,"src":"/static/87be28baa66b2bfdbec1ee9478ca3d79/7d145/python-web-crawler.png","srcSet":"/static/87be28baa66b2bfdbec1ee9478ca3d79/69585/python-web-crawler.png 200w,\n/static/87be28baa66b2bfdbec1ee9478ca3d79/497c6/python-web-crawler.png 400w,\n/static/87be28baa66b2bfdbec1ee9478ca3d79/7d145/python-web-crawler.png 610w","sizes":"(max-width: 610px) 100vw, 610px"}}},"author":{"id":"Mark Duan","github":null,"avatar":null}}}}]}},"pageContext":{"tag":"Coding"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}