Comments on: Playing with pipes

By: asa letourneau

asa letourneau — Mon, 01 Mar 2010 10:34:19 +0000

Amanda,

If you are out there and you get this…knew nothing about screen scraping a few days ago. Came across this http://newprosoft.com/web-content-extractor.htm Not bad for someone like me who has ZERO scripting/coding background!

By: asa letourneau

asa letourneau — Sun, 17 Jan 2010 09:08:11 +0000

Hi Tim,

Saved a pipe in which I added the PROV flickr stream to the rss feeds. The title of the pipe is the same as yours. Not sure what the pipe file name is…couldn’t see any ‘save as’ option. When I run the pipe the prov stream shows up fine. When I embed the badge code on the provcommunity ning…the PROV images don’t show up…even the title of the pipe isn’t the same…will do it again now so you can see what I mean. Not sure what I’m doing wrong? Cheers, Asa.

By: tim

tim — Sun, 17 Jan 2010 06:29:17 +0000

Asa – Yep, that’s basically it. Let me know if you strike any probs, and yes of course you’re welcome to use, embed, tinker, whatever – that’s why I did it.

Actually, I need to do some tinkering myself to fix the NAA Find of the Month link. Also, annoyingly, the NAA has started putting non-collection photos in its photostream. Bah.

By: asa letourneau

asa letourneau — Sun, 17 Jan 2010 03:41:21 +0000

Hi Tim,

Just had a go at adding your pipes rss badge to the provcommunity website and it worked beautifully. Have taken it off because I wanted to ask you if this is okay to do?…and because I would love to tinker with the script, save it then embed it again…i have no programming skills but think i might be able to do it. Do I just follow the link to the pipes url you give, click on clone and then tinker, save and embed. Sounds too easy?

cheers
asa

By: Basil Dewhurst

Basil Dewhurst — Mon, 21 Sep 2009 11:41:02 +0000

Amanda – Tim is right screenscraping is frustrating and painful. I liken screenscaping html and extracting structured data to trying to turn a hamburger into a cow ! For a couple of data sources that contribute to the National Library of Australia’s People Australia program I’ve used two linux apps, wget and tidy to get html documents and turn them into well formed ‘xhtml-like’ documents. I’ve then created (often complex) XSL Transformations to extract the data and output EAC records in XML format. I’ve then used php to apply the transformations to thousands of files which can then be harvested. Painful, worth it but … wouldn’t it be nice if people used standard record formats and standard protocols for exchanging information ??? If you’d like further info please let me know: bdewhurs at nla dot gov dot au

By: tim

tim — Fri, 11 Sep 2009 00:02:24 +0000

Amanda - Screen scraping is the often frustrating process of trying to extract structured data from a web page. In this case, Yahoo Pipes returns a web page as a long string of text, which you can then cut up into useful pieces using regular expressions. In other cases you might be working with the page's DOM and using XPath expressions to find the elements you want -- or a combination of XPaths and regexp. Here's an example where I used PHP and XPath to build an RSS feed. Here's another using Python. The wonderful Programming Historian site has lots of useful information for any aspiring screen scraper. Of course, screen scraping is also what powers many Zotero translators and there's plenty of useful info in this tutorial.

By: Amanda French

Amanda French — Thu, 10 Sep 2009 14:47:30 +0000

I’m curious: how does one “screen scrape”? I always hear about it but never know how it’s done. Nice work, btw!