mechanize examples

#00 Initialization

require 'rubygems'

require 'mechanize'

agent = WWW::Mechanize.new

agent.set_proxy('localhost', '8000')

agent.user_agent = 'Individueller User-Agent'

agent.user_agent_alias = 'Linux Mozilla'

agent.open_timeout = 3

agent.read_timeout = 4

agent.keep_alive = false



agent.max_history = 0 # reduce memory if you make lots of requests

#01 manual get requests

url = 'http://apoc.sixserv.org/requestinfo/'

page = agent.get url

# or ...

page = agent.get(url, {"name" => "value", "key" => "val"})

#02 manual post submits

url = 'http://apoc.sixserv.org/requestinfo/'

page = agent.post(url, {"name" => "value", "key" => "val"})

#03 form post submits

page = agent.get 'https://twitter.com/login'

login_form = page.form_with(:action => 'https://twitter.com/sessions')

login_form['session[username_or_email]'] = '[Username]'

login_form['session[password]'] = '[Password]'

page = agent.submit login_form

#04 link and history navigation

page = agent.get 'http://www.heise.de/'

page = agent.click(page.link_with(:text => /Telepolis/))

page = agent.click(page.link_with(:href => /artikel/))

agent.back

agent.back

puts page.body

#05 exceptions

begin

  page = agent.get 'http://apoc.sixserv.org/diese/seite/gibt/es/nicht/'

rescue WWW::Mechanize::ResponseCodeError

  puts "ResponseCodeError - Code: #{$!}"

end

#06 referer

page = agent.get(:url => 'http://apoc.sixserv.org/requestinfo/',

:referer => 'http://google.com/this/is/a/custom/referer')

puts page.body

#07 request header manipulation

agent.pre_connect_hooks << lambda do |params|

  params[:request]['X-Requested-With'] = 'XMLHttpRequest'

end

#08 response header

page = agent.head 'http://sixserv.org'

server_version = page.header['server']

puts "Server: #{server_version}"

if page.header.key? 'x-powered-by'

  php_version = page.header['x-powered-by']

  puts "X-Powered-By: #{php_version}"

end

# redirection urls:

agent.redirect_ok = false

page = agent.get 'http://www.sixserv.org/'

puts page.header['location']

#09 content parsing

# X Path / CSS-Selector:

page = agent.get 'http://xkcd.com/'

img = page.search '/html/body/div/div[2]/div/div[2]/div/div/img'

puts img

# Regular Expression:

page = agent.get 'http://example.com/'

page.body.match /< h3>([^<]+)< \/h3>/

puts "Heading 3: #{$1}"

#10 "with" method examples

# *_with: form, link, base, frame or iframe



# get the first link including "foo" inside url:

page.link_with(:href => /foo/)



# all links with text 'more'

page.links_with(:text => 'more')



# get the form with the name 'foo'

page.form_with('foo') # or form_with(:name => 'foo')

resque-schedular undefined method enqueue_at

initializers\resque.rb must reference resque_schedule.

require 'resque_scheduler' 

resque task must be started

COUNT=5 QUEUE=* rake resque:workers

resque-schedule task must be started

rake resque:scheduler

to monitor resque-schedule, resque-web must be started with the config file of resque as parameter. This one must not reference anything from rails directly as resque-web is a sinatra app and it won't be able to load it properly.

resque-web ~/pathToYourApp/config/initializers/resque.rb 

http://stackoverflow.com/questions/7048020/rails-3-0-9-resque-scheduler-and-delayed-job-error-undefined-method-enqueue-at

Ruby - return an array in random order

array.shuffle

Setting User Agent in Mechanize

agent = Mechanize.new
agent.user_agent = 'Custom agent'

rspec documentation

http://rspec.info/documentation/

Resque Cheatsheet

Status

Resque.info
Resque.queues
Resque.redis
Resque.size(queue_name)

# check out what's coming next in the queue
#    Resque.peek(archive_queue)
#    Resque.peek(archive_queue, 1, 5)
#    Resque.peek(archive_queue, 59, 30)
Resque.peek(queue_name, start=1, count=1)

Workers

Resque.workers
Resque.working
Resque.remove_worker(worker_id) # find worker_id from one of the above methods

Queue Management

# For testing a worker, I usually call the 'perform' method directly.
#    Resque.enqueue(ArchiveWorker)
#    Resque.enqueue(ArchiveWorker, 'matching', 'arguments')
Resque.enqueue(klass, *args)
Resque.dequeue(klass, *args)
Resque.remove_queue(queue_name)

Callbacks

# Each of these can either take a block, or be assigned to with a Proc
Resque.before_first_fork(&blk)
Resque.before_fork(&blk)
Resque.after_fork(&blk)

Problems

Redis connects to wrong host - Redis connects to localhost:6379 by default. Customize this by doing the following:

Resque.redis = 'hostname:port:db'  # all 3 values are optional

Workers die stop after first batch completes - This is caused by the workers losing their connection to MySQL. See this gist for a fix and an explanation. Alternatively, you can add this line at the beginning of your 'perform' method:

ActiveRecord::Base.reconnect!

http://whatcodecraves.com/articles/2010/06/22/resque_cheatsheet/index.html

Rake Tasks with Parallel Prerequisites

Tasks with Parallel Prerequisites

Rake allows parallel execution of prerequisites using the following syntax:

multitask :copy_files => [:copy_src, :copy_doc, :copy_bin] do
    puts "All Copies Complete"
  end

In this example, copy_files is a normal rake task. Its actions are executed whereever all of its prerequisites are done. The big difference is that the prerequisites (copy_src,copy_bin and copy_doc) are executed in parallel. Each of the prerequisites are run in their own Ruby thread, possibly allowing faster overall runtime.

http://rake.rubyforge.org/files/doc/rakefile_rdoc.html