Thursday, February 21, 2013

PowerShell Quirks

This morning I had to figure out how to create 45 insert statements in order to populate a new table in oracle from existing data.
The table is a cross-reference table and only consists of 2 columns that need to be populated by the data . Using an Excel spreadsheet I had created a two column CSV file by hand, which was a long and tedious process. I did not want to duplicate this process and since I had it in a friendly format surely there was an easier way.

After experimenting with the IDE I was using I found that I could not simply import the data because it would either destroy the table and recreate it with only those two columns (destroying the columns that I needed to hold the create date, created by, modified date and modified by values) or it would reject the data because it did not contain enough columns in which case I might as well go back to writing each individual SQL statement by hand.

I thought about writing a macro in Vim to accomplish this which is certainly doable but frankly, it’s been a while since I wrote anything remotely that complex in Vim so I dismissed that idea as too time consuming.

Then I realized that this should be trivial in PowerShell and the solution could be used universally by everyone in the development department in a similar situation.  Indeed, creating the script that would generate the SQL statements was trivial.

An Import-Csv filename created a hash array with each row in my CSV file, a single variable with tokens for the two items I wanted to replace, an Out-File and a foreach loop later and it was done. Coding is awesome!!

What I didn't know was that PowerShell gets quirky with carriage returns and newlines when using Out-File to write to a file. No matter what I tried, I always wound up with a file containing a single line with forty-five SQL statements. After beating my head against it for a while I finally realized I was hurting myself. The simple way to do this (although not elegant) is to simply use the Write-Output string command and pipe that to the Out-File command using the append flag.
The write-output command always adds a windows newline character to the end of the string but since it’s piped to the Out-File it goes to the file rather than the console where it normally goes.

Just something to be aware of if you haven’t played with writing to files in PowerShell. If you’re aware of that issue and have your data in a CSV format, writing this script to create strings that can be tokenized takes about 2 minutes.

I agree with Scott Muc. Many things Microsoft leave a bad taste in my mouth but PowerShell typically isn't one of those.

Wednesday, February 20, 2013


Over the course of time there is an inevitable conversation that arises, usually on a periodic basis, about standardizing on particular development methodologies and frameworks. Just so I'm clear, I am going to refer to everything a developer uses to accomplish his specific task/project is a tool, from the IDE to the management methodology to the frameworks. I'm also speaking in general of companies that do internal development but don't actually create software for retail sales, i.e., they don't make any profit off of development.

I'm going to start with an observation. Everyone that I can ever recall having this conversation with, without exception, have all agreed that one of the issues with development, especially at large companies, is the corporate tendency to try and impose on developers a standardized framework for each problem. Everyone see's this all the time in whatever department they are working in.

Just about everyone could come up with a story of having been in a situation where something was mandated from above that they had to live with. If you were lucky, you got a voice in the decision making process but regardless of if your favorite shiny object came out on top of the decision, you had to live with it.

They will start with statements similar to the ones below.
We need to settle on a single database and make sure that everything uses that particular DB. We need to standardize on SVN/Git/Mercurial/TFS and make sure that everything gets moved into that repository so that we can reap the benefits. We need to pick a single IDE and everyone gets to stick with it. And on and on.

In my personal experience, eventually everyone jumps on board the push for standardization, if for no other reason than they hate the product that seems to be winning. Despite the fact that these same people lament this tendency by business because they really believe that the right tool for the right job is the correct way to code!!
In other words, in almost all cases, your colleagues, peers, managers all truly believe that the task at hand should dictate the tools you use to accomplish that task as efficiently and effectively as possible.

Now, there are very valid reasons for standardizing policies, processes and whatnot out there. I adore standards, truth be told. And if I am the business and my bottom line is profit, there are even more reasons. I can leverage everything from economics of scale to product support by purchasing a single product and sticking with it. This is certainly a very large concern and an effective factor that encourages business to push strongly standardization.

Even developers in the trenches have a tendency to push for standardized frameworks and tools because most have a favorite and learning new ones can be viewed as a painful struggle which wastes their valuable time and is ultimately pointless since they could have accomplished their goal in a fraction of the time with the tools they are comfortable using. Few people willingly stretch far outside their comfort zone, even fewer will do it while under the stress of deadlines and, just a guess here, even fewer managers will want to defend the developers for doing so because that means they're productivity is diminishing because they fighting a learning curve.

Now let's step back into the theoretical world of developing software solutions for a minute. As a business my goal is to provide a software solution that generates the greatest amount of satisfaction and benefit for the consumer as possible so that I can generate as much desire for my product, and hence profit, as possible. As a developer my goal is to create a solution that meets the customers needs, is easy to use and generally exceeds their expectations if at all possible. There are other goals involved for everyone, to be sure, but they aren't my point so I'm being a little vague here.

If the corporate standards dictate that I must use a particular technology tool to accomplish my task and my tasks are widely varied, sooner or later, (usually sooner) I'm going to have a task in which I cannot deliver the best software that I can because I won't be able to use the appropriate tool(s).

Is a SharePoint installation really the best solution for a given task? It makes perfect sense to leverage the corporate SharePoint installation if the task lends itself to that environment but what if it doesn't?
Do I really need a full blown MS SQL Server or Oracle installation for every task that requires a DB back end?  What if it needs to be used by some guy in the field that has no access to an internet connection?
Is the best tool for creating a data access layer really NHibernate when the model is trivially simple? Could't I just use Entity Framework and if the application grows swap out to a more appropriately complex tool like NHibernate later when it's actually needed?

What happened to that lamentation that we all shared that the best tool for the task should be used rather than some standard that was decided upon?

Back to the NHibernate vs EF debate (cause that was what was under discussion by my coworkers earlier today) Why do we have to choose between NHibernate and EF?

In my opinion, there should be room for a choice of the most appropriate tool. Entity Framework are to Lincoln Logs what NHibernate is to Lego's. No one is saying that Lincoln Logs should be abandoned by children for Lego's because Lego's are superior. They both have their place. They are both the right tools provided you're working on the right job.

The biggest argument is that it's inefficient to use multiple tools that perform the same function and I would agree with that up to a point. If the tools do the exact same job the exact same way then maybe you have an argument for standardization assuming standardization actually generates real benefit in the long run.

A common example is the IDE. A standardized IDE has been something I've seen pushed over a large number of businesses. My question is, why? What do we gain? Typically the answer is, IDE specific issues will arise. If Bud checks in his development work and he's written it using VS2012 and Bob checks it out and opens the files in say Eclipse, the project files that are being used VS will make no sense and Bob can't compile the projects. Or there are compatibility issues between versions of IDE XYZ so we all need to be using not only the same IDE but the same version.

My answer is the same; why? It takes some effort to ensure that everyone is using the same build process but if everyone is truly using the same standard build process, why can't Bob compile Buds project? Other than convenience for offline work, why are the project files even in the repository? Chef, Maven, TFS, etc, don't care about your project files (or at least they shouldn't IMO). They exist to provide an independent and impartial build environment open to whoever is allowed to request a build.

Another example, most of our developers are familiar with Ruby, we don't want to learn C# or Scala, the paradigms are too different, it will require lots of extra effort to train them, it will be more expensive and time consuming to build the software if we do it that way.

The above answers are correct but most people ignore what they will have gained by it. Exposure to new technologies, techniques, frameworks - the things that are the meat and potatoes of your daily work, is one of the primary ways we grow as developers! Good developers will get better, bad developers will get good and everyone's selection of appropriate tools will have increased. This is never a bad thing!
There are valid reasons that this approach won't work on every project, obviously, but the benefits of learning new things in a group environment are well documented. Here is the first thing I came across when I Googled, 'learning in a group environment'. Cooperative and Collaborative Learning

If it works for kids in a classroom, why wouldn't it work for adults? Isn't that one of the lessons that agile programming and methodologies tell us over and over? Group collaborative learning is good? Surely you can find at least one project a year that is small enough to benefit from this approach. The more often you do it, the easier it will become to do it again, maybe on larger projects. And each time, the tools that the development teams are familiar with grow.

Just my two cents.

Tuesday, February 5, 2013

Adventures in .NET

So today I am going to try and create a proof of concept application using VS2012 and a blank Lightswitch project. I'm interested to see how that goes. coming from the Java OSS side of life this will be my first attempt to build anything with the Lightswitch framework, .NET4.5 and my first application (as opposed to a simple service) with any of the above.

Let's see how it goes.

First hurdle, I have install the Oracle Developers Toolkit in order to get any kind of usefulness out of any of the frameworks. This already involves two things for me run the range of mildly annoying to rage induced blindness; installers and oracle drivers.

Having already gotten off to a stellar start and having plenty of time before the installer finishes, let me mention a couple of things that irritate me about both of those.

1) I know that drivers and frameworks and all that jazz for databases grow over time and can take up a lot of space but, come on, folks, why is my client side only install taking up more space than the actual frameworks that they are supporting. Really? REALLY??!?!?
Since when did a driver, even a development interface, become an instrument of the almighty advertisement? It really offends me that when all I want, care about or will ever use is a simple driver that *might* take up a few megabytes of disk space comes so bloated with crap that companies are desperate to get me to use that when the install is finished I have lost tens of gigabytes of disk space! Or maybe I'm narcissistic, maybe they don't want me to use any of that crap at all, maybe they just want to use me to pad the numbers they hand to declare how popular their widget(s) are by counting me in because I installed it. OK, now that I've contemplated that, I'm still offended! The end result is the same. I'm losing time, space and hence money because they want to saddle me with everything and the kitchen sink when all I wanted was a damned driver.

2) Why in the hell do they give me the option to choose my installation directory and then ignore my request? Seriously, (almost) every installer asks me where I would like to install my software to but once we get out of the tiny stuff and into things like Java, Oracle clients, and my all time favorite VS2012, the installer treats it as a strong suggestion at best and outright ignores it at worst. When I install VS2012 and it asks me where I want it to go and I say someplace on the F drive (my data drive) it will still install hundreds of gigabytes on the C drive, far more than what it installs on the F drive.
I have a small boot drive, not by choice, but because that's what was passed out to me when I first came on board here. When I got around to installing VS on my machine in spring of last year, it would't fit. I didn't have enough gigabytes on my boot drive to install even the bare minimum. Really? My IDE HAS to have access to the C: drive in order to function?? And if thats not the case, why are you ignoring the install directory preferences that you just solicited from your user?? Oracle does the same thing although not to nearly that extreme.

OK, looks like finally, close to an hour later, I am getting close to actually finishing the oracle install. I can't wait to see the next hurdle.

Friday, January 11, 2013

Must Remember to Write, Even If I Don't Think It's Worth It

Since I haven't post an entry in a bit I thought I would throw something up.

My first Coursera class of the year starts next week and I'm looking forward to it. It should give me a change to reinforce the budding FP skills that the Scala class started me on since the class will be a comparative look at programming languages with the languages in use being ML, Racket and Ruby. Should be an interesting.

I am still loving the new RC helicopter hobby. In an unexpected twist I think I get as much pleasure out of repairing the little sucker when I crash it as I do actually flying it. Definitely not what I was expecting on several levels. I am guessing that this will balance out once my flying skills start to catch up with my repair skills. Although, to be fair, I have only had to make three repairs. On the other hand, I have screwed around and made two sets of training gear, taken a stab at leveling the swashplate by eyeball (I really need a swashplate level to do it properly) and I'm about to try my luck at setting up the blade tracking and blade balancing.
Which reminds me, I should go by HobbyTown and pick up some tracking tape and a blade balance. One of the guys that got me started on this new hobby is convinced I'm going to wind up getting a quad flyer at some point in the sooner-rather-than-later future. I have to admit, those things are *FUN*. Although he says he finds it more difficult to handle the quad, I personally think its a dream to fly. Goofy easy to control compared to my mCPx heli and almost as maneuverable. It also has a much larger battery for longer flight times and is still capable of doing loops (I haven't checked but I think it should be able to do rolls as well).

I may pick one up just because some of the motor skills should be transferable to my heli and it will give me confidence when I get discouraged at my progress on the mCPx that if I can do it on the quad, I can do it on the heli. As a bonus, I would be pretty confident flying it around the house and torturing the animals with it!

OK, I do feel better about posting since it's part of my plan to improve fundamental skills and obviously, I do need it.

Wednesday, January 2, 2013

The New Year Has Begun

It's clearly been a while since I put up a post. This has been due as much to the holidays as much as it has been caused by distractions, lack of laziness and procrastination.

I've was busy with a production deployment in early December and have been on vacation enjoying the holidays with family and friends after that. Today is my first day back at the office in a couple of weeks and while it's good to be back, I'm still sorta glad it's a short week. It was hard getting up this morning and making the ride in with the temp at 37 degrees and my wife still warm in bed.

It was nice to see a couple of good clean blog entries by Uncle Bob giving a great introduction into Functional Programming and why it's supposed to be, "The Next Big Thing". I personally thing it's past that point already, it's just that the majority of people don't necessarily realize it is or why everyone is saying that. The relevant posts are FP Basics E1 and FP Basics E2.

Also on my radar this month will be another Coursera class. It doesn't look like it will be particularly challenging but it's an introduction to a range of different programming languages functional as well as OO, most of which I have spent little, if any, time doing anything more significant than the old standby, "Hello World!" I look forward to learning how to make a program say that in a few more languages and with a little luck and some work, I can get a solid handle on some new tools that I will be able to use effectively and appropriately.

I've also picked up RC helicopter flying. As in most things I do, I kind of jumped in over my head and while I didn't spend a ton of money, what I did do is overestimate my skills (again) and get a heli that is, in all honesty, not something I should be trying to learn on as a beginner. I cheer myself up with the knowledge that if I can conquer this bird, I will have made a significant achievement (well, to me anyway) and that I actually love just practicing ground exercises, much less the low hover stuff that I occasionally use to remind myself why I don't get the bird in the air until I've mastered other skills first.

I also have at least a couple of interesting tasks at the office to work on in addition to the dreary crap that normally makes up a work day. Neither are really sophisticated but one of them will afford me the opportunity to learn and explore some new tools and frameworks. For a guy who has been a die hard OSS and the POSIX-style OS, this is extremely important now that I have gone to the Dark Side and work as a perm at a Windows based company.
As an aside, I have been very impressed over the last few years with how Microsoft has started opening up their software but also started to adopt and promote things that I have considered industry standard since the early 2000's but MS refused to acknowledge.

I will try to organize my thoughts and present some cogent observations on my introduction into the universe of C# web application frameworks and, if I'm lucky, maybe a little bit about how F# plays within them.