Log in

No account? Create an account
08 January 2012 @ 12:43 pm
Weather! (and perl programming)  
I'm trying to kick-start a few dormant projects at the moment (why, yes, it's a new year, funny how that happens ;) ). One of these is to actually do something with the 26 & a bit years of data on weather in my parents' back garden (max/min temps, barometric pressure, humidity, rain, wind). There's also probably about 26 years worth of failed attempts to do something with the data ;) Starting with hand drawn monthly graphs back when I was a kid, through to various computer related things.

The last attempt got bogged down on designing a form for inputting the data so that the workflow actually worked for me. Pretty much everything I tried to implement just got in the way, and given the amount of data there is it has to be as easy as possible to do the data entry otherwise it won't get done. This time I'm side-stepping that altogether. Personally I find typing numbers into a spreadsheet both easy & accurate - I can just keep looking at the data and 99% of the time I type the right number without checking keys or screen, and in the vast vast vast majority of times when I make an error I know it as soon as I type it (the muscle movement "feels" wrong for the number I'm reading). And I can just work along one row & back the next (number, right arrow, number, right arrow ... number, down arrow, number, left arrow, number, left arrow ... number, down arrow, number, right arrow etc etc etc) - again a muscle feel thing for the right rhythm. But I could never get a form to quite sit as comfortably under the fingers, and not having the data just there to skim review (at the end of the sheet or when I thought I'd made a mistake) was a difficulty too.

So the brainwave on this reboot of the project was: if it's easiest to just type it into a spreadsheet, why not just type it into a spreadsheet? That's what I'm doing this time - type up a month, save it out as a CSV file with a sensible name (like Nov1985). And the data is there & manually editable etc (unlike the iteration of the project where it was in a SQLite database). I'd rather not just use a spreadsheet programme to do the data manipulation, I've tried that before and while I'm sure it's possible to make the spreadsheet dance & do tricks, I found it required too much manual intervention to get the graphs to look as I wanted them to. So that requires a bit of programming (which is the fun bit of the project :) ).

Yesterday I typed up a few months (Oct 1985 - Feb 1986), and started some work on a perl script to read in the data & draw graphs/generate statistics etc. Right now the script works to read in the data from the files & put it into an internal data structure. I also started to write something to calculate averages for the various parameters for a month or a year, but went round in circles a few times (nested loops, limited scope variables, & when to reset variables to zero are things that tend to break my head unless I pay close attention) & it still doesn't currently work.

The immediately obvious issues that need fixing when I next work on this (noted primarily for my own benefit) are:
  • Strip line end characters (\n\r) from line ends before manipulating data. If I recall right, there's an elegant way to do this I just need to look it up.

  • If there is no value for a parameter make this end up as distinct from when the value is 0, set this up while it's reading the data in.

  • Find the graph drawing modules(? extensions? I forget the name for these) that I had before and set them up again.
Current Mood: busybusy
Current Music: Steven Wilson "Grace for Drowning"
Monumentmarnanel on January 8th, 2012 01:20 pm (UTC)
Line breaks are almost certainly \r\n and not \n\r, but either way, "chomp" will do it for you.
Margaretpling on January 8th, 2012 01:26 pm (UTC)
Ah, yes. Thanks :) I knew there must be a simple way to do it, it being such an obvious thing to do :)
graham boydgrahamb on January 8th, 2012 06:01 pm (UTC)
You say the data needs inputting. It's in a printed/hand written form right now? Reckon it'd OCR? Would certainly save you some pain.
Margaretpling on January 8th, 2012 06:14 pm (UTC)
Hmm, I'm not sure. It might - even tho it's handwritten it's fairly clear (at least the stuff I was looking at yesterday), but I'm not sure how much re-interpretation it might need afterwards (things sometimes corrected by writing over the wrong number so not necessarily clear and other stuff of that nature - this was initially done by an 11yo & a 9yo, so probably gets clearer over time ;) ). I should probably think about it now if I'm ever to do it, coz otherwise any script gets too entrenched in having the data in a particular shape.
graham boydgrahamb on January 8th, 2012 06:20 pm (UTC)
I bought myself a Xerox document scanner recently to scan a lot of dryad_wombat's old University notes. While OCR has moved on a bit in the last 10 years, it struggles a bit with her 'artistic' style of note taking but I'd say scientific data is probably an area where it'd do a bit better because there are fixed fields. Errors are more easily spotted and corrected.

[edited for speeling]

Edited at 2012-01-08 06:21 pm (UTC)
Jonbigme on January 9th, 2012 06:32 pm (UTC)
I don't know about Perl, but in Python the csv module makes handling spreadsheets so easy it gives me a fuzzy glow on the inside. That and you can make pretty graphs with mathplotlib :)