Skip to content

Instantly share code, notes, and snippets.

@breckenedge

breckenedge/download_urls_from_osti

Created March 29, 2011 15:19
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save breckenedge/892539 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save breckenedge/892539 to your computer and use it in GitHub Desktop.
Download ZIP
used to download pdfs from the OSTI.GOV information bridge
Raw
download_urls_from_osti
require 'rubygems'
require 'mechanize'
require 'progressbar'
docs = File.readlines('URLS.txt')
pb = ProgressBar.new('downloading', docs.length)
docs.each do |line|
url = line.strip
if !File.exist?(File.basename(url))
agent = Mechanize.new
agent.get(url) # get a session cookie
f = File.new(File.basename(url),'wb')
f << agent.get(url).body # download the file
sleep rand 5
f.close
end
pb.inc
end
@breckenedge
Copy link
Author

breckenedge commented Mar 29, 2011

requires an adjacent file named 'URLS.txt' containing lines of addresses like:

http://www.osti.gov/bridge/servlets/purl/972223-KyjlCP/972223.pdf
http://www.osti.gov/bridge/servlets/purl/5486534-BMf4MW/5486534.pdf
http://www.osti.gov/bridge/servlets/purl/6489792-nii8Km/6489792.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment