• Functional Ruby

    jaturken        
    0 Likes0 Commentsruby

    def push_page_urls_to_queue(url, doc)
      doc.xpath('//a/@href').
        map{ |link| link.text.strip }.
          select{ |u| belongs_to_domain?(u, url) }.
            map { |u| full_url(u, url) }.
              select{ |u| !already_crawled?(u) }.
                map { |u| queue << { url: u, priority: url_priority(u) } }
    end

Comments (0)