Help with Apple Script for Yahoo weather and Geektool

Discussion in 'Mac Programming' started by gr4z, Jun 20, 2016.

  1. gr4z macrumors 6502

    Joined:
    Aug 7, 2010
    Location:
    England
    #1
    Hi

    Now that the Yahoo APIs have changed myt original script fails to work.

    I have the following XML that I would like to use with Geektool, but am having issues :

    <yweather:location xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" city="Peterborough" country="United Kingdom" region=" England"/>
    <yweather:wind xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" chill="64" direction="220" speed="11"/>
    <yweather:atmosphere xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" humidity="91" pressure="1009.0" rising="0" visibility="13.9"/>
    <yweather:astronomy xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" sunrise="4:37 am" sunset="9:28 pm"/>
    <yweather:condition xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="11" date="Mon, 20 Jun 2016 01:00 PM BST" temp="64" text="Showers"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="12" date="20 Jun 2016" day="Mon" high="65" low="58" text="Rain"/>


    I dont really know Apple Script very well but my old script uses the following code to extract the data for the forecast:

    set forecast1Conditions to (do shell script "awk '/yweather:forecast/{print $0}' " & weatherFile & "| awk 'NR==1{print;exit}' | sed -e 's/.text=//;s/code=.//' | sed 's/\"//g'") as string
    set forecast1Conditions to (my trim_string(forecast1Conditions, white_space, "both"))
    do shell script ("echo " & forecast1Conditions)


    This bombs out now with the newly formatted XML. Can anyone explain what am I doing wrong in trying to select the text="Rain" from the final yweather:forecast ?? I have plenty of other lines selecting all sorts of stuff so if someone can help with this I should be able to work out what I am doing wrong. I dont really understand AWK and SED.

    Thanks

    Apologies posted this in the UI customisations forum first...
     
  2. kryten2 macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #2
    You can start by breaking that whole do shell script line into pieces and see what each piece produces as output. There are a lot of online tutorials to be found about awk and sed. Perhaps it's time to check those out or look into an XML parser.
     
  3. gr4z thread starter macrumors 6502

    Joined:
    Aug 7, 2010
    Location:
    England
    #3
    OK I am now using the following XML code from a file called weatherfile

    <yweather:condition xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="Mon, 20 Jun 2016 08:00 PM BST" temp="18" text="Partly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="11" date="20 Jun 2016" day="Mon" high="18" low="13" text="Showers"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="28" date="21 Jun 2016" day="Tue" high="20" low="12" text="Mostly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="28" date="22 Jun 2016" day="Wed" high="20" low="12" text="Mostly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="12" date="23 Jun 2016" day="Thu" high="20" low="14" text="Rain"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="24 Jun 2016" day="Fri" high="19" low="13" text="Partly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="25 Jun 2016" day="Sat" high="17" low="12" text="Partly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="26 Jun 2016" day="Sun" high="18" low="12" text="Partly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="27 Jun 2016" day="Mon" high="18" low="13" text="Partly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="28" date="28 Jun 2016" day="Tue" high="18" low="12" text="Mostly Cloudy"/>
    <yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="28" date="29 Jun 2016" day="Wed" high="18" low="12" text="Mostly Cloudy"/>


    However, trying to extract data from the first line using

    do shell script "grep -E 'yweather:condition' " & weatherFile & "| sed -e 's/.*text=//;s/\\/>.*//' | sed 's/\"//g'"

    Results in the text from the last line, 'mostly cloudy'? It should be from the line yweather:condition and the text should be 'Partly Cloudy'.

    I don't understand why. Can anyone help?
     
  4. jasnw macrumors 6502a

    jasnw

    Joined:
    Nov 15, 2013
    Location:
    Seattle Area (NOT! Microsoft)
    #4
    Try using:

    do shell script "grep -E 'yweather:condition' " & weatherFile & "| sed -e 's/.*text=//;s/\/>.*//' | sed 's/\"//g'"

    You have too many backslash characters in the second sed edit command.
     
  5. gr4z thread starter macrumors 6502

    Joined:
    Aug 7, 2010
    Location:
    England
    #5
    Using

    do shell script "grep -E 'yweather:condition' " & weatherFile & "| sed -e 's/.*text=//;s/\/>.*//' | sed 's/\"//g'"

    And the script moans saying expected """ but found unknown token where the \ was removed. Am confused now.
    --- Post Merged, Jun 21, 2016 ---
    OK looking into this further I have found the input XML weatherfile, does not have any carriage returns at the end of each line. So it looks like

    <yweather:condition xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="30" date="Mon, 20 Jun 2016 08:00 PM BST" temp="18" text="Partly Cloudy"/><yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="11" date="20 Jun 2016" day="Mon" high="18" low="13" text="Showers"/><yweather:forecast xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" code="28" date="21 Jun 2016" day="Tue" high="20" low="12" text="Mostly Cloudy"/>

    If I manually edit the CRs in my scripts works. A quick google on how to insert a carriage return after /> and before <yweather throws up some really confusing SED scripts. Anyone have a one liner than can do this?
     
  6. jasnw macrumors 6502a

    jasnw

    Joined:
    Nov 15, 2013
    Location:
    Seattle Area (NOT! Microsoft)
    #6
  7. gr4z thread starter macrumors 6502

    Joined:
    Aug 7, 2010
    Location:
    England
    #7
    Mmm I want to change

    /><yweather

    To

    />
    <yweather


    Not sure this link helps much to be honest as its not a straight forward character swap. I need to insert a carriage return.
     
  8. cqexbesd macrumors regular

    Joined:
    Jun 4, 2009
    #8
    Parsing XML with regular expressions is horrible. To be fair though, XML is horrible in of itself.

    Generally speaking you are best off using XPath for these kind of things. Needless to say it is a joy but better than REs for anything but the most simple extraction.

    There are lots of command line xpath tools available. For the purpose of this example I will use /usr/bin/xpath because I think it comes preinstalled with OSX - at least I seem to have it. If you don't have it you can get it via e.g. macports when you install Perl. Alternatively see if you have xmllint with has an xpath query feature.

    The query
    Code:
    /yweather:condition/@text
    
    Selects the text attribute of each yweather:condition element. Wrap that in the string function to get just the value (and not the attribute name) and we have:
    Code:
    string(/yweather:condition/@text)
    
    Now of course there are some complications. The data given above is not really one XML document (which would have an opening and closing tag surrounding everything) but more like standalone documents (all consisting of just one tag) on each line. Maybe you have extracted this data from something larger? If so then stop and you can skip this step. If not then we have to make sure we feed each XML document to xpath one at a time. If you only ever need the first line then just use "head -1" but assuming you don't know which line you want then we can use a loop:

    Code:
    while read -r xml ; do echo "${xml}" | xpath "string(/yweather:condition/@text)" 2>/dev/null ; done < weatherFile
    
    The redirection to /dev/null hides some meta output form the tool. It probably wouldn't be necessary if you used a different tool. One other downside of this tool is it will print out new lines even if it found nothing. If you want to suppress those then a quick fix is to use tr. This leaves you with:

    Code:
    while read -r xml ; do echo "${xml}" | xpath "string(/yweather:condition/@text)" 2>/dev/null | tr -d '\n' ; done < weatherFile
    
    Then you just need to call that from your AppleScript.
     
  9. jasnw macrumors 6502a

    jasnw

    Joined:
    Nov 15, 2013
    Location:
    Seattle Area (NOT! Microsoft)
    #9
    It's not entirely clear to me exactly what's in this file you're trying to process, but if it is indeed all on one line with no hidden characters between the /> and < strings, you can use something like the following to insert newline characters where wanted:

    Code:
    sed -e 's/></>[</g FILENAME | tr '[' '\n' | REST OF CODE HERE
    
    The sed command inserts a dummy character between the characters that end one line and start the next, and the tr command (which can only deal with single character changes) changes that to a newline character. The dummy character needs to be some character that you don't expect to ever see in the file you're parsing. This is a kludge, but it should work if the file contents are as you've stated and if you can find a workable dummy character. You should be able to do this with just a sed command, but that doesn't appear to work as expected in the OS X sed implementation. A master at designing the sort of regular expressions (regex) used by sed could probably come up with some better approach (or maybe using awk), but more information would be needed about the expected contents of the files you're processing to come up with a bulletproof approach.

    If you are truly dealing with a correctly formated XML file, which is not at all clear to me you are, using an XML parser is a cleaner way to do this, although it will take some spin-up on your part if you want to understand what it's doing.
     
  10. kryten2 macrumors 6502a

    Joined:
    Mar 17, 2012
    Location:
    Belgium
    #10
    Don't know what kind of file the OP is using but after poking around a bit at yahoo this is what I got.

    Code:
    <?xml version="1.0" encoding="UTF-8"?>
    <query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng"
        yahoo:count="1" yahoo:created="2016-06-20T21:25:55Z" yahoo:lang="en-US">
        <results>
            <channel>
                <yweather:units
                    xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                    distance="mi" pressure="in" speed="mph" temperature="F"/>
                <title>Yahoo! Weather - St Nicolas, Oost-Vlaanderen, BE</title>
                <link>http://us.rd.yahoo.com/dailynews/rss/weather/Country__Country/*https://weather.yahoo.com/country/state/city-977271/</link>
                <description>Yahoo! Weather for St Nicolas, Oost-Vlaanderen, BE</description>
                <language>en-us</language>
                <lastBuildDate>Mon, 20 Jun 2016 11:25 PM CEST</lastBuildDate>
                <ttl>60</ttl>
                <yweather:location
                    xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                    city="St Nicolas" country="Belgium" region=" Oost-Vlaanderen"/>
                <yweather:wind
                    xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                    chill="59" direction="230" speed="14"/>
                <yweather:atmosphere
                    xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                    humidity="90" pressure="1014.0" rising="0" visibility="16.1"/>
                <yweather:astronomy
                    xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                    sunrise="5:28 am" sunset="10:3 pm"/>
                <image>
                    <title>Yahoo! Weather</title>
                    <width>142</width>
                    <height>18</height>
                    <link>http://weather.yahoo.com</link>
                    <url>http://l.yimg.com/a/i/brand/purplelogo//uh/us/news-wea.gif</url>
                </image>
                <item>
                    <title>Conditions for St Nicolas, Oost-Vlaanderen, BE at 10:00 PM CEST</title>
                    <geo:lat xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">51.164459</geo:lat>
                    <geo:long xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">4.1415</geo:long>
                    <link>http://us.rd.yahoo.com/dailynews/rss/weather/Country__Country/*https://weather.yahoo.com/country/state/city-977271/</link>
                    <pubDate>Mon, 20 Jun 2016 10:00 PM CEST</pubDate>
                    <yweather:condition
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="26" date="Mon, 20 Jun 2016 10:00 PM CEST"
                        temp="60" text="Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="11" date="20 Jun 2016" day="Mon" high="60"
                        low="57" text="Showers"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="28" date="21 Jun 2016" day="Tue" high="67"
                        low="59" text="Mostly Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="30" date="22 Jun 2016" day="Wed" high="75"
                        low="61" text="Partly Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="4" date="23 Jun 2016" day="Thu" high="78"
                        low="65" text="Thunderstorms"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="4" date="24 Jun 2016" day="Fri" high="72"
                        low="65" text="Thunderstorms"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="30" date="25 Jun 2016" day="Sat" high="70"
                        low="59" text="Partly Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="30" date="26 Jun 2016" day="Sun" high="67"
                        low="58" text="Partly Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="30" date="27 Jun 2016" day="Mon" high="67"
                        low="58" text="Partly Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="30" date="28 Jun 2016" day="Tue" high="66"
                        low="58" text="Partly Cloudy"/>
                    <yweather:forecast
                        xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0"
                        code="30" date="29 Jun 2016" day="Wed" high="69"
                        low="58" text="Partly Cloudy"/>
                    <description>&lt;![CDATA[&lt;img src="http://l.yimg.com/a/i/us/we/52/26.gif"/&gt;
    &lt;BR /&gt;
    &lt;b&gt;Current Conditions:&lt;/b&gt;
    &lt;BR /&gt;Cloudy
    &lt;BR /&gt;
    &lt;BR /&gt;
    &lt;b&gt;Forecast:&lt;/b&gt;
    &lt;BR /&gt; Mon - Showers. High: 60Low: 57
    &lt;BR /&gt; Tue - Mostly Cloudy. High: 67Low: 59
    &lt;BR /&gt; Wed - Partly Cloudy. High: 75Low: 61
    &lt;BR /&gt; Thu - Thunderstorms. High: 78Low: 65
    &lt;BR /&gt; Fri - Thunderstorms. High: 72Low: 65
    &lt;BR /&gt;
    &lt;BR /&gt;
    &lt;a href="http://us.rd.yahoo.com/dailynews/rss/weather/Country__Country/*https://weather.yahoo.com/country/state/city-977271/"&gt;Full Forecast at Yahoo! Weather&lt;/a&gt;
    &lt;BR /&gt;
    &lt;BR /&gt;
    (provided by &lt;a href="http://www.weather.com" &gt;The Weather Channel&lt;/a&gt;)
    &lt;BR /&gt;
    ]]&gt;</description>
                    <guid isPermaLink="false"/>
                </item>
            </channel>
        </results>
    </query>
    
    
     
  11. jasnw macrumors 6502a

    jasnw

    Joined:
    Nov 15, 2013
    Location:
    Seattle Area (NOT! Microsoft)
    #11
    So, looks like bog-standard XML with line-feeds and everything. To OP: have you been posting files as you've received them (such as what kryten2 just posted) or after the file has been passed through some earlier processing? If your initial file is well-formatted XML, then a formal XML parser is a much safer way to go than a roll-your-own parser.
     
  12. gr4z thread starter macrumors 6502

    Joined:
    Aug 7, 2010
    Location:
    England
    #12
    Thanks for all your great help here guys. The XML is stored using the following command

    set weatherCurl to (do shell script "curl --silent 'https://query.yahooapis.com/v1/publ...ecast WHERE woeid= 22961 and u="c"&format=xml' | grep 'yweather:' >" & weatherFile2)

    Obviously weatherCurl and weatherFile2 are defined elsewhere. weatherFile2 is a XML file but there is not any carriage returns anywhere. Not sure why.

    I am not much of a coder I must admit and have used the following crap code to force a carriage return

    do shell script "tr '<' '
    ' < " & weatherFile2 & "> " & weatherFile

    This seems to work my original Yahoo script and now I get the attached on my desktop. So its working albeit with my crude code.
     

    Attached Files:

  13. jasnw macrumors 6502a

    jasnw

    Joined:
    Nov 15, 2013
    Location:
    Seattle Area (NOT! Microsoft)
    #13
    Wasn't sure that what you did (just brute-force enter a carriage return) would work, and I wasn't sure you wanted to sacrifice the "<" character the way you have done. Give the following a try:

    Code:
    set forecast1Conditions to ( do shell script "curl --silent " & wxURL & " tr '<' '\n' | grep condition | sed -e 's/.*text="//;s/"\/>//' )
    
    where wxURL is the URL for the file that you want to process. The 'grep' section picks out the single line containing 'condition', the first 'sed' edit gets rid of all characters up to and including the " at the start of the conditions text string, and the second edit gets rid of the three characters at the end of the line. Using the \n newline character makes what you're doing a bit more transparent when you look at this a year from now. I've never been happy putting a "naked" return in a script, and that's what the \n metacharacter is designed to avoid. (I'm not an Applescript user so I've tested this only within Terminal. There may be some tweaking needed to get things like ' and " sorted out correctly, particularly in the sed commands section.)
     
  14. cqexbesd macrumors regular

    Joined:
    Jun 4, 2009
    #14
    Code:
    curl --silent 'https://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20weather.forecast%20WHERE%20woeid%3D%2022961%20and%20u%3D%22c%22&format=xml' | xpath "string(/query/results/channel/item/yweather:condition/@text)" 2>/dev/null
    
     
  15. gr4z thread starter macrumors 6502

    Joined:
    Aug 7, 2010
    Location:
    England
    #15
    Thanks for this, what would the code be if I wanted @text, @temp?
     
  16. cqexbesd macrumors regular

    Joined:
    Jun 4, 2009
    #16
    If you mean you want both values then:

    Code:
    "/query/results/channel/item/yweather:condition/@*[name()='text' or name()='temp']"
    
    should work - but you would need to parse the output - it won't be a comma separated list. To get XPath to do the concatenation you would need to use XPath 2 which doesn't seem to be supported by /usr/bin/xpath (at least under 10.9).

    If you only want those 2 values though, you know you can just ask for that from Yahoo by changing the query to be:

    Code:
    https://query.yahooapis.com/v1/public/yql?q=select%20item.condition.temp%2C%20item.condition.text%20%20from%20weather.forecast%20where%20woeid%20%3D%2022961%20and%20u%3D%22c%22&format=xml&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys
    
    and with simpler XML it becomes feasible to use AppleScript built in XML handling. e.g.

    Code:
    set http_response to do shell script "curl -s https://query.yahooapis.com/v1/public/yql?q=select%20item.condition.temp%2C%20item.condition.text%20%20from%20weather.forecast%20where%20woeid%20%3D%2022961%20and%20u%3D%22c%22&format=xml&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys"
    
    tell application "System Events"
        set xml_root to make new XML data with data http_response
        set conditions to the XML element "yweather:condition" of XML element "item" of XML element "channel" of XML element "results" of XML element "query" of xml_root
      
        set temp to the value of the XML attribute "temp" of conditions
        set desc to the value of the XML attribute "text" of conditions
    end tell
    
    error handling left as an exercise for the reader. NB text is called desc because text is reserved.

    I would say though, the more I see AppleScript the more I recommend learning a UNIX based scripting language where these things are easier. Personally I use a lot of Perl but there are many choices, most of which aren't wrong.
     

Share This Page