Mac OS: cURL your way to freedom

 Why cURL?

When we view a website (like this one), your computer’s browser is converting thousands upon thousands of lines of code to display all the images and content you see. Thankfully your browser does all this processing in the background while you wait for a website to load. With modern computers and fiberoptic cable, this is pretty fast. If we didn’t need to convert this code into a visual form for our eyes, just image how much faster a website could load! From there, if we could tell our computer to look at only the data we want, or to perform some automatic task, just think of all the time you could save!

What can it do?

The cURL command, to put it simply, connects your computer to other computers and accesses data using many of the standard file transfer protocols, such as HTTP and FTP. This may not seem like much at first, but apply a few additional commands in conjunction to it to it and you are on your way to taking over the internet.

Continue reading to see how you can use this tool to perform fast multi-site data processing to gather information faster and better.

Simple Example

Let’s use the cURL command to grab an image from the internet. We’re going to add the -O option to it, which takes a file and saves it as its original name:

curl -O http://www.redditstatic.com/reddit_gold-70.png

Notice that this saves the image to your current directory in the terminal. It also provides with a useful status bar in case your file is large. If you want the file saved in a different folder, you have to first make that location your current directory in terminal.

Real Life Problem Solving

You want to check your local weather, but you aren’t exactly sure where you are. Either you are on a bus, or train, or just can’t remember the name of the city you are visiting. By using cURL we can solve this problem.

Get your Longitude and Latitude

To find our current longitude and latitude coordinates we can use the website watsmylatlng.com and applying the cURL command to get the data. I’ve added the –silent option to the command below, which suppresses any download dialog from cURL, and the -L option which some websites require:

curl --silent -L http://whatsmylatlng.com

This command as-is downloads the full website and all of its code. Looking inside this data we can find our computer’s current estimated latitude and longitude. That’s great and all, but it would be more useful if we could cut out only the data we want. Well, let’s do that. For this task we use the commands grep and cut. Grep is used to find, and cut… well, cuts.

curl --silent -L http://whatsmylatlng.com | grep 'class="success"' | cut -d "(" -f  2 | cut -d ")" -f 1

The grep and cut commands basically find the area we want to keep, and cuts it from the rest of the HTML. Much easier to read!

The Current Weather

The basic cURL code to get a weather report from the website wunderground.com would look like this:

curl --silent http://www.wunderground.com/cgi-bin/findweather/getForecast?query=35.445,139.650 | grep facebook_share_forecast | cut -d "'" -f 6-8 | sed "s/', '/ /" | sed "s/ °//"

In this URL, the query line is for latitude and longitude inputs. The coordinates in this example are for Yokohama, Japan (a favorite place of mine).

In the previous example I already got this information, so how do we put them together? For this we’ll need to place the first cURL’s results into a variable that can be used in the next cURL.

Get Your Live Weather, Anywhere

This is our final code to get today’s weather:

LATLONG=$(curl --silent -L http://whatsmylatlng.com | grep 'class="success"' | cut -d "(" -f  2 | cut -d ")" -f 1 | sed "s/, /,/") && curl --silent "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=${LATLONG}" | grep facebook_share_forecast | cut -d "'" -f 6-8 | sed "s/', '/ /" | sed "s/ °//"

Don’t freak out over all of that code. It might seem complicated, but it is quite easy to understand.

The first part:

LATLONG=$(~~~)

This defines our variable LATLONG, our latitude and longitude, as the result of our first cURL command.

The next part puts this variable inside the second cURL command:

curl --silent "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=${LATLONG}"

The resulting data is the HTML of our current location’s weather report.

From here we can add some extra commands to “cut” our data out from the resulting HTML. Pretty cool, right?

Bonus : Throw it in Applescript

If you are on a Mac, you can throw all this together in an Applescript app that will speak your current weather. Just double click your saved app to have it run, or set it up to say the weather every few hours by using a scheduler application like Cronnix:

set result to do shell script ¬
"LATLONG=$(curl --silent -L http://whatsmylatlng.com | grep 'class="success"' | cut -d "(" -f  2 | cut -d ")" -f 1 | sed "s/, /,/") && curl --silent "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=${LATLONG}" | grep facebook_share_forecast | cut -d "'" -f 6-8 | sed "s/', '/ /" | sed "s/ °//""
say result

More Complex: Complete weather report

Finally, we can apply all of this, with a little extra, to get our entire weather report for the week, automatically updated for where we are currently accessing the internet:

LATLONG=$(curl --silent -L http://whatsmylatlng.com | grep 'class="success"' | cut -d "(" -f  2 | cut -d ")" -f 1 | sed "s/, /,/") && curl --silent "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=${LATLONG}" | grep facebook_share_forecast | cut -d "'" -f 6-8 | sed "s/', '/ /" | sed "s/ &deg;//" | awk '{ print "Nown" $2 "n" $1 "n" }' && curl --silent "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=${LATLONG}" | grep -A 8 foreGlance | sed "s/<div>/ /g" | sed "s/ &deg;//" | sed "s/<[^<>]*>//g" | sed "s/ | /C | /" | awk '{gsub("t","",$0); print;}' | sed "/^$/d" | awk '{print;} NR % 3 == 0 { print ""; }' | sed -e "s/^[ ]*//" -e "s/*^[ ]//"

And just for fun, we can put this in a simple applescript that we can have run when we start our computer to let us know the weather report before we start our day. You can download the sample applescript app here.

cURL in Review

The uses for cURL are varied and many, from data mining to sending email – a very flexible command to use and understand. However it is not a magic command and sometimes it is better to use other commands, such as Wget or rsync, but even with its shortcoming cURL is a versatile tool that is fun to use to solve a variety of IT problems.

One thought on “Mac OS: cURL your way to freedom”

Leave a Reply

Your email address will not be published. Required fields are marked *